Volume VI · Information資訊
Surprise you can count. Entropy, KL, and the number an LLM actually minimizes.
Information is surprise you can count. A fair bit is one shannon. An LLM is a machine that tries to need fewer of them on the next token.
In English
Information is surprise you can measure. A fair coin-flip is one bit. A language model is a machine that tries to be less surprised by the next token. This volume is that number.
- Try this
- Start with Bit, then Entropy. Cross-entropy is the one that trains LLMs.
- Keep this
- Bits measure surprise. Models try to need fewer of them.
The stack · click a layer
What this is
Coins, surprise, KL, and the exact number an LLM minimizes. Compression as the theorem behind “the loss went down.”
What this is not
Not a coding-theory textbook. Shannon bits, not nats, so the numbers match the “one bit” slogan. The math is the real −p log p.
- 01 · UnitsBit位元Flip a fair coin. That was one bit.One fair yes/no = one bit. Bias shrinks it.
- 02 · UnitsEntropy熵Bias the coin. Entropy falls.Entropy is average surprise sitting in a distribution.
- 03 · EventsSurprise驚訝度Click a rare outcome. Pay more bits.Rare and unexpected is expensive. That bill is the loss.
- 04 · EventsDivergence散度Two coins. Extra bits both ways.KL is extra bits when you used the wrong code. Not a distance.
- 05 · The lossCross-entropy交叉熵Pick the true token. Read −log q.LLM loss = $-\log q(\text{true token})$. Softmax + that number.
- 06 · The lossCompression壓縮A tiny codebook versus the entropy bound.Better model ≈ better compression. Training shortens the archive.