05 · Capacity
Overfit
A high-degree polynomial can hit every training point and miss the function. Memorization is not generalization. Scale plus data plus a little noise is why LLMs do not just memorize.
In English
A curve with enough wiggles can pass through every training point and still be a bad map of the world — it memorized the noise. That is overfit. Big models avoid it with lots of data, some randomness, and not training forever.
- Try this
- Raise the polynomial degree. Training error falls. The true curve does not match. That gap is the warning.
- Keep this
- Fitting the points is not the same as fitting the function.
Lab · fit the noise
Degree 1
Train MSE 0.075 · holdout MSE 0.134 · filled = train, hollow = holdout
Training error going to zero is not the goal. Holdout error is. Extra degree (extra parameters) can buy the first and spend the second.
LLMs look overparameterized and still generalize because the data is huge, the noise is structured, and the optimizer does not find the interpolating monster.