75 years in one strip (the segments aren't to scale; things sped up a lot)

Every era adds one idea, and you've now seen each of them in this thread: counting → embeddings → attention → the transformer + pretraining → scaling laws → SFT/RLHF → RL for reasoning → agents.

Model size, where it was published (log scale, billions of parameters)

● = total parameters. For mixture-of-experts models, the line down to ○ shows how many are active per token. Closed frontier models (GPT-4 and later, Claude, Gemini) don't publish sizes. Growth didn't stop in 2023; it just stopped being published. Since Chinchilla (2022), training tokens grew faster than parameters: GPT-3 used 0.3T tokens, Llama 2 2T, Llama 3 15T, and Qwen3 about 36T.

The current frontier: what's known vs. suspected

Closed labs don't publish recipes. confirmed = officially stated or visible in open models. reported = credible estimates or leaks. unknown = genuinely not public.

Still a decoder-only transformer at its core: the same blocks as the diagram earlier. confirmed
Mixture of experts: many MLPs per block, each token sent to a few. Standard in top open models (DeepSeek-V3, Qwen3, Llama 4, Kimi K2). confirmed (open) Believed true of the closed frontier. reported
Scale: likely trillions of total parameters and tens of trillions of training tokens. GPT-4 was reported at about 1.8T MoE, never confirmed. reported
Heavy RL after pretraining: verifiable rewards (math, code, tests) plus agent environments, now a large share of total compute. confirmed in broad strokes
Reasoning at answer time: models "think" before answering, and more thinking gives better answers. confirmed
Long context (200k to 1M+ tokens) and native multimodality (images, audio). confirmed
Synthetic data + distillation: big models generate training data and teach smaller, cheaper models. confirmed
Exact sizes, data mixes, RL recipes, reward models. unknown
Open debates: how far RL extends past the pretraining data, whether the "data wall" binds, and whether compute and power become the main limit. open question