Translate "je suis étudiant" → "I am a student"

Two RNNs. The encoder reads the French. The decoder writes English one word at a time, feeding each word it just wrote back in as its next input.

The math, both versions

The attention line is Q·K → softmax → weighted sum of V. The query is the decoder state st, and the keys and values are the encoder states hj. The 2017 transformer kept this line, applied it within a single sentence as well ("self-attention"), and removed the RNNs entirely.

Train a real BPE tokenizer, one merge at a time

Start with single characters. At each step, count every adjacent pair of symbols, weighted by how often each word appears. Merge the most frequent pair into one new symbol everywhere, then repeat. Edit the training text to try your own.

Words, currently split into symbols
Pair counts (the top one gets merged next)
Merge rules learned, in order
Vocabulary = single characters + one new token per merge

Now tokenize a word the tokenizer never saw

To tokenize: split into characters, then apply the learned merges in the same order they were learned.

That's the whole algorithm. Real tokenizers run it on terabytes of text until there are about 100k tokens. They work on raw bytes, so any string can be encoded, and they treat a leading space as part of the token (" the" and "the" are different tokens). Frequent words become one token, and rare ones get spelled out in reusable pieces.