Volume VI · Information
Surprise you can count. Entropy, KL, and the number an LLM actually minimizes.
Information is surprise you can count. A fair bit is one shannon. An LLM is a machine that tries to need fewer of them on the next token.
In English
Information is surprise you can measure. A fair coin-flip is one bit. A language model is a machine that tries to be less surprised by the next token. This volume is that number.
- Try this
- Start with Bit, then Entropy. Cross-entropy is the one that trains LLMs.
- Keep this
- Bits measure surprise. Models try to need fewer of them.
The stack · click a layer
What this is
Coins, surprise, KL, and the exact number an LLM minimizes. Compression as the theorem behind “the loss went down.”
What this is not
Not a coding-theory textbook. Shannon bits, not nats, so the numbers match the “one bit” slogan. The math is the real −p log p.
- 01 · UnitsBitFlip a fair coin. That was one bit.One fair yes/no = one bit. Bias shrinks it.
- 02 · UnitsEntropyBias the coin. Entropy falls.Entropy is average surprise sitting in a distribution.
- 03 · EventsSurpriseClick a rare outcome. Pay more bits.Rare and unexpected is expensive. That bill is the loss.
- 04 · EventsDivergenceTwo coins. Extra bits both ways.KL is extra bits when you used the wrong code. Not a distance.
- 05 · The lossCross-entropyPick the true token. Read −log q.LLM loss = −log q(true token). Softmax + that number.
- 06 · The lossCompressionA tiny codebook versus the entropy bound.Better model ≈ better compression. Training shortens the archive.