03 · Events

Surprise

−log p(event). Rare things are expensive to encode and shocking to a model. The loss on a token is exactly this number.

In English

One event’s surprise is −log p. A token the model thought was 50% costs 1 bit when it arrives. A token it thought was 3% costs about 5. That number on the true token is the training loss.

Try this: Click a common outcome, then a rare one. The rare one should cost more bits.
Keep this: Rare and unexpected is expensive. That bill is the loss.

Lab · pay for the rare token

A tiny next-token distribution. Click an outcome. The bill is −log₂ p.

Pick a token

the
0.40
cat
0.30
sat
0.20
qbit
0.10

Self-information of an event is −log p. Entropy is the average of that. Training sees the realized event and charges the model that amount.

A “confidently wrong” model pays a huge surprise. Calibration is keeping p close to how often things actually happen.