03 · Descent

The step

w ← w − η ∇L. Too large an η and you jump the bowl. Too small and you die of boredom. That is the whole of gradient descent.

In English

New weight = old weight minus a little of the slope. That “little” is the learning rate η. Too big: you leap over the valley. Too small: you crawl. That update is gradient descent.

Try this: Set a tiny η and step a few times. Then a huge η and watch it bounce. Then find one that settles.
Keep this: w ← w − η × slope. η is a taste, not a law.

Lab · one update

η = 0.25

w = -0.6 · L = 4.84 · ∇ = -4.4

The learning rate is not a deep fact. It is the gain on a feedback loop. Too hot, oscillation. Too cold, a crawl. Schedules exist because the bowl changes shape.

Adam keeps a running guess of how noisy and how curved each coordinate is, then fakes a better η. The step is still −gradient.