How a transformer
actually thinks.
Eight stages. The whole loop, one screen at a time.
Text becomes tokens.
The model only knows pieces it has seen during training. Unusual or long words break into common parts — so the same fixed vocabulary handles everything, from “barrister” to a word the model has never seen.
Each token is a point in space.
Every token looks at every previous token.
It does this many times in parallel.
Then each token is processed independently.
Where most of the model’s knowledge is stored.
All of that, stacked 80 times.
Every word looks at the other words and decides which ones matter to it. Attention is how the model works out that it refers to cat:
“The cat sat because it was tired.”
Once attention has gathered the right context, the layer does a bit of private thinking — refining what each word now means in the light of the others.
Stage 05 was a look inside this step. It’s where most of the model’s knowledge is stored.
Doing this many times lets understanding build up gradually. Early layers catch simple things — grammar, which word goes with which. Later layers handle abstract meaning — tone, intent, reasoning. Less like 80 separate machines, more like 80 rounds of revision on the same draft.
80 layers is roughly the depth of a model like Llama 70B. Different models go deeper or shallower — the idea is what scales, not the figure.
Output is a probability over every word.
Pick one. Append. Repeat.
That is the whole machine.
No memory between steps. No reasoning beyond the loop. Just attention, projection, and a probability over the next token — at remarkable scale. Next in the series: what the machine can actually see.
Next: The Context Window →