A transformer reads a sequence of tokens and predicts the next one. It does that by letting every token look at the other tokens (attention), thinking about what it saw (feed-forward), and repeating that many times. Below is the whole loop, then each piece, with things you can click.

Text"The cat sat" Tokens →vectors + position Transformer block × N (e.g. 32–100+) Self-attentiontokens share information Feed-forward (MLP)each token thinks alone Scoresover vocabulary Next token"on" append the new token and run again → that's how text gets generated

1Text becomes vectors

Text is chopped into tokens (words or word pieces). Each token is looked up in a table to get an embedding: a list of numbers (thousands of them in real models) that encodes its meaning. Similar words get similar vectors. A position signal is mixed in, because attention by itself doesn't know word order.

positive valuenegative valueeach strip = one token's vector (8 of its numbers shown)

2Self-attention: every token looks back at the others

This is the core idea. To understand "it" in the sentence below, the model needs to know what "it" refers to. Attention lets "it" pull information from "animal". Click any word to see where it looks. Shading = how much attention it pays to each earlier word.

How it decides who to look at. Each token's vector is turned into three smaller vectors by three learned matrices:

Query"What am I looking for?"
Key"What do I contain?"
Value"What I'll hand over if picked"

A token's Query is compared (dot product) with every Key. Big match = high score. Scores go through softmax so they become weights that add up to 1. The output is the weighted average of the Values.

Attention(Q,K,V) = softmax( Q·Kᵀ / √d ) · V

Try it: rotate the Query of "it" and watch the weights move. When the query points toward a key, that word wins.

3Many heads, all at once

A layer runs many attention heads in parallel (e.g. 32). Each learns a different kind of relationship: one tracks the previous word, one links pronouns to nouns, one links verbs to subjects. Their outputs are combined. The grid shows the full table for the head selected above: rows = the token doing the looking, columns = the token being looked at.

The upper-right triangle is empty on purpose: a token may only look at earlier tokens (the "causal mask"), so the model can't peek at the answer while learning to predict it.

4The block: attention, then think, then repeat

x Normalize Attention + Normalize Feed-forward + x′ residual: add the input back on residual again x′ has the same shape as x, so blocks stack like Lego. Each pass refines every token's vector.

Feed-forward = a small neural net applied to each token separately. Attention moves information between tokens; the MLP processes it within a token (much of the model's stored knowledge lives here). Residuals mean each block only adds a correction, which keeps very deep stacks trainable.

5Pick the next token

After the last block, the final token's vector is compared against every word in the vocabulary to get a score (a "logit") for each. Softmax turns scores into probabilities. Temperature controls how adventurous the pick is.

"The animal didn't cross the street because it was ___"

6Repeat, one token at a time

Generation is just this loop: run the whole stack, pick a token, append it, run again. Click to step.

✓The whole thing in one breath

Tokens → vectors (+position) → [attention lets tokens share context → MLP processes each token → add back residual] × N → scores over vocab → softmax → pick a token → append → repeat. Training just means: show it trillions of tokens of text, measure how wrong its next-token guesses were, and nudge every matrix (embeddings, Q/K/V, MLPs) to be a little less wrong.