04 · Events

Divergence散度

$KL (p ∥ q)$ is how many extra bits you spend encoding $p$ with a code built for $q$ . It is not a distance: $KL (p ∥ q) \neq = KL (q ∥ p)$ .

In English

You built a code for coin Q, but the world is coin P. KL(P‖Q) is the extra bits you waste. It is not a symmetric distance: mixing up which is which changes the number. Models care because they are Q trying to match P.

Try this: Set two different biases. Read KL both ways. They should not match.
Keep this: KL is extra bits when you used the wrong code. Not a distance.

Lab · not a distance

KL (p ∥ q) = i \sum p_{i} lo g \frac{p _{i}}{q _{i}}

p(heads) = 0.7

q(heads) = 0.3

p tails
0.30
p heads
0.70

q tails
0.70
q heads
0.30

KL(p‖q): 0.49 sh
KL(q‖p): 0.49 sh

KL(p‖q) = Σ p log(p/q) = H(p,q) − H(p). Extra bits. Asymmetric, zero iff p=q, and the thing that makes MLE on a model match “project onto the model family.”

Mode-covering versus mode-seeking lives here: KL(p‖q) vs KL(q‖p). Generative models pick a side and then suffer the other.