02 · The model
Attention
Self-attention is how a token looks at the others and decides what matters. That is the whole trick inside a transformer.
In English
When the model reads “the bank of the river,” the word bank needs to look at river, not at money. Attention is that looking: each token asks the others “how much do you matter to me?” and mixes their answers.
- Try this
- Click different tokens. The lines are who that token is looking at. Then move the layer slider — early layers look nearby, later ones look for meaning.
- Keep this
- Attention is weighted looking. A transformer is that, stacked.
Lab · self-attention
Click a token — it becomes the query
- raised39.0%
- rates39.0%
- river9.7%
- .2.8%
Each token builds a query vector and every token offers a key. The dot products become weights. The weighted values are what the token “sees.” Multi-head attention is several of these in parallel. Stack enough layers and you get a transformer. The keys and values computed here are exactly what the next lab refuses to recompute.
Early layers are local and syntactic. Later layers bind meaning and reference. The picture below is a cartoon of that shift — useful as a map, not as a measurement of any particular checkpoint.