02 · Units

Entropy

H = −Σ p log p. A sure thing is 0. A fair coin is 1. A biased coin is in between. That is the average surprise sitting in a distribution.

In English

Entropy is the average surprise of a random draw. Certain = 0. Fair coin = 1. A 90/10 coin sits in between. Formula: minus the sum of p log p. You do not need to memorize it — feel it with the slider.

Try this: Drag the bias from 50/50 to 99/1. Entropy should fall toward zero.
Keep this: Entropy is average surprise sitting in a distribution.

Lab · bias the coin

P(heads) = 0.5 · H = 1 sh

tails
0.50
heads
0.50

H(p) = −Σ p log₂ p. Maximum at uniform. Zero at a spike. A language with a more peaked next-token distribution has lower entropy — it is more compressible.

English sitting in a tokenizer is not 15 bits per token. A good model’s average −log q is around 1 bit per byte of text, give or take.