06 · The loss

Compression

A good model is a good compressor. Tokens that the model assigns 50% cost 1 bit; tokens it assigns 3% cost ~5. Training is shortening the archive.

In English

A model that is less surprised is a better compressor: it spends fewer bits on the next token. Training is shrinking the archive of the internet, in disguise. A tiny codebook that ignores probabilities will lose to entropy.

Try this: Compare a dumb codebook to the entropy bound. The bound is the floor. A good model tries to meet it.
Keep this: Better model ≈ better compression. Training shortens the archive.

Lab · a tiny codebook

Four symbols. A naive code uses 2 bits each. A prefix code that matches frequency uses fewer on average. Entropy is the floor you cannot beat.

e · p=0.40 · 1 b · ideal 1.32
t · p=0.310 · 2 b · ideal 1.74
a · p=0.2110 · 3 b · ideal 2.32
q · p=0.1111 · 3 b · ideal 3.32

e
0.40
t
0.30
a
0.20
q
0.10

Naive: 2.00 b
This code: 1.9 b
Entropy: 1.85 b

Shannon: you cannot beat entropy on average with a uniquely decodable code. Huffman gets close. A trained model gets closer on the distribution of text.

“Compression is intelligence” is oversold as philosophy and undersold as an engineering fact: if the loss went down, the archive got shorter.