04 · Events

Divergence

KL(p‖q) is how many extra bits you spend encoding p with a code built for q. It is not a distance: KL(p‖q) ≠ KL(q‖p).

In English

You built a code for coin Q, but the world is coin P. KL(P‖Q) is the extra bits you waste. It is not a symmetric distance: mixing up which is which changes the number. Models care because they are Q trying to match P.

Try this: Set two different biases. Read KL both ways. They should not match.
Keep this: KL is extra bits when you used the wrong code. Not a distance.

Lab · not a distance

p(heads) = 0.7

q(heads) = 0.3

p tails
0.30
p heads
0.70

q tails
0.70
q heads
0.30

KL(p‖q): 0.49 sh
KL(q‖p): 0.49 sh

KL(p‖q) = Σ p log(p/q) = H(p,q) − H(p). Extra bits. Asymmetric, zero iff p=q, and the thing that makes MLE on a model match “project onto the model family.”

Mode-covering versus mode-seeking lives here: KL(p‖q) vs KL(q‖p). Generative models pick a side and then suffer the other.