03 · The model

Cache

Attention needs every previous key. Keep those keys and decode is a line, not a square. The next request reuses the longest identical prefix — similar does not count.

In English

Each new token has to look at every token before it. Doing that from scratch every time is slow. A KV cache saves the old “looks.” The first pass over the prompt is the expensive wait; each token after that is cheap. A second chat reuses the saved prefix only if it is identical, not merely similar.

Try this: Hit Generate with KV cache on and note “Paid so far.” Reset, switch to Recompute, Generate again. Then run a second request with one letter changed in the system prompt.
Keep this: Prefill is time-to-first-token. Decode is tokens-per-second. Similar prefixes do not count.

Lab · prefill once, then decode

Attention at step t needs every previous key. Save those keys and each new token is a line, not a square. Throw them away and you pay the square every time.

Prefix length

cache cold

This step: —
Paid so far: 0
If every step recomputed: 256

Lab · the next request

Request 1 already paid 196 for 14 tokens. A second request reuses the longest identical prefix — byte for byte, not “pretty close.”

1 · You are a precise assistant. Answer briefly User: 2+2?

2 · You are a precise assistant. Answer briefly User: 3+3?

Shared prefix 10 / 14 · hit through “:”

Cold prefill: 196
With prefix cache: 96

Prefill is the quadratic: every new token in the prompt attends to every previous one. The KV cache stores those keys and values. Decode then pays a line — the new query against the cache — and appends one more slot. That is why time-to-first-token and tokens-per-second are different numbers.

Prefix caching is the same object across requests. The match is exact, from the left. Change a system-prompt token and everything after it is cold. Compaction (harness chapter) is what you do when the cache no longer fits the window. A semantic cache of whole answers is a different product — lookup, not attention.