A real character-level GPT is training in this page right now: embeddings → attention → MLP → softmax, trained by backprop + Adam on next-character cross-entropy. Nothing is precomputed.

step 0 loss – val – ≈ – choices 0 steps/s
layers
heads
width d
context
learning rate temperature

What it writes now

brighter = more confident · hover a character to see the choice it made
Hover any generated character to see the probability distribution the model sampled it from.

Next-character loss

cross-entropy in nats · lower = better predictions

Dashed lines are what simpler models can do on this text: guessing uniformly, using letter frequencies (unigram), and looking back one character (bigram). Dropping below the bigram line means it is using longer context, which is only possible through attention.

Same seed, over time

identical random draws each time, so every change comes from the weights

Attention

Row = the character doing the looking (query); columns = what it can look at (keys). Each row sums to 1. The upper-right is masked: no peeking at the future.

Embedding space

top-2 PCA of token vectors
vowelsconsonantsspace · punctuationdigits · symbols

Starts as random noise. Characters used in similar places get similar vectors, so groups form with no labels. This is the distributional hypothesis at the character level.

Things worth seeing

  1. Layers 1 vs 3 on Code: deeper models learn the colon → newline → 4-space indent pattern sooner.
  2. Context 8 → 32: more context gives attention more to work with and lowers the loss.
  3. Learning rate at max: the loss spikes and thrashes. Too low and it crawls.
  4. Poem: the corpus is tiny, so train loss heads to ~0 while val loss rises. That is memorization (overfitting), on screen.
  5. Temperature 0.1 vs 1.5: same model, different decoding. Low repeats its favorite phrase, high produces noise.