05 · The loss

Cross-entropy

Train a model and you minimize −log q(true token). That is cross-entropy against a one-hot. Softmax plus this number is the LLM objective.

In English

Cross-entropy is “how many bits to name the true next token using the model’s probabilities.” Softmax turns raw scores into those probabilities. Training an LLM is: make this number smaller, on lots of text.

Try this: Pick the true token. Move the model’s guess. −log q should drop as q on the true token rises.
Keep this: LLM loss = −log q(true token). Softmax + that number.

Lab · the LLM objective

Four logits, softmax, one true token. Loss = −log q(true). That is every gradient step inside Volume I.

True token

logit the = 1.2

logit cat = 0.4

logit sat = 0.1

logit qbit = -0.6

−log q(the) = 0.96 bits

the
0.51
cat
0.23
sat
0.17
qbit
0.08

Cross-entropy H(p,q) = −Σ p log q. For a one-hot p it is just −log q_true. Softmax turns logits into q. That pair is the LLM loss.

Temperature, top-k, and nucleus sampling are games you play with q after training. They do not change the weights. The loss did.