02 · The slope
Gradient
The gradient is the direction of steepest increase. Learning walks the other way. In one dimension it is just the slope.
In English
If loss is a hill, the gradient is the steepest uphill direction. Learning walks the other way. In one dimension it is just “is the slope positive or negative?”
- Try this
- Slide the weight. Read the slope. When the slope is zero you are at a flat — maybe the bottom, maybe a saddle.
- Keep this
- Gradient = steepest increase. We step the opposite way.
Lab · the slope
Weight w = -0.4 · slope -4
Slope points uphill to the left — walk right.
dL/dw is the slope. The update will subtract it, so the sign tells you which way to walk. Magnitude tells you how steep, not how far.
In many dimensions the gradient is a vector of those slopes. You cannot plot it. You can still subtract it.