01 · The model

Tokens

Models do not read words. They read tokens — pieces of text from a fixed vocabulary — and emit a probability distribution over the next one.

In English

The model does not see letters or words. It sees tokens — chunks from a fixed list, like a very large alphabet. It then scores every chunk in that list as “what might come next” and picks one.

Try this: Type a short sentence. Watch it split. Then drag temperature: low is picky, high is chaotic.
Keep this: Text in → tokens → a bet on the next token. That is generation.

Lab · tokenize and sample

Prompt

5 tokens·pedagogical BPE, not tiktoken

Next-token distribution

Temperature0.70

·Paris
99.8%
·the
0.1%
·Lyon
0.1%
·a
0.0%
·located
0.0%
·not
0.0%
·Berlin
0.0%
·London
0.0%

Training squeezes a corpus into a vocabulary of tens or hundreds of thousands of pieces. “hello” might be one token. “antidisestablishmentarianism” will not. Spaces usually belong to the following word — that leading dot in the chips is a space.

The forward pass is a function: a list of token ids in, a vector of logits out, one number per vocabulary entry. Softmax turns logits into a distribution. Sampling picks. That is generation. There is no separate “thinking” stage inside the weights.