The RL loop (GRPO style), on one prompt

1 · Roll outsample G full answers to the same prompt. Each is one "rollout".
2 · Rewarda checker grades each answer: ✓ = 1, ✗ = 0.
3 · Advantagecompare each answer to the group: (r − mean) / std.
4 · Updateraise the probability of every token in the + answers, lower it for the − answers.
Prompt:
Rollouts per prompt G:

πThe model's current habits (how likely it is to answer each way; the line marks the value before the last update)

↻This step's rollouts (token tint = the push it receives: green up, red down)

Press ▶ to sample.

%Accuracy on the first try over training

The line is the model's real probability of answering correctly. The dots are the share of each group that got it right. A real run does this for millions of prompts at once, and every answer is thousands of tokens.