06 · Capacity
Landscape
A bowl is easy. A saddle has a downhill and an uphill at once. Real loss surfaces are high-dimensional saddles; SGD wanders them anyway.
In English
A bowl has one bottom. A saddle is a pass in the mountains: downhill one way, uphill the other. High-dimensional loss looks like a lot of saddles. We still walk it with noisy little steps, and it works better than it has a right to.
- Try this
- Click a bowl, then a saddle. Step. Notice “downhill” depends on which way you face.
- Keep this
- Real loss is not a bowl. SGD wanders saddles anyway.
Lab · a well and a saddle
L = 1.4 · ∇ = [2.4, -0.4]
Click the map to place the point
A well has one way down. A saddle has a way down and a way up. Almost every critical point of a large net is a saddle, which is good news for SGD.
You do not need to find the global minimum. You need a point whose next-token surprise is small on text you have not seen.