"The Eiffel Tower is located in the city of ___" ① vector for "of" attention already copied in "Eiffel Tower" + "city of" ② W_up rows = "keys" each neuron detects a pattern ③ W_down cols = "values" what each neuron writes back ④ output only here is it a probability The "fact" = this key→value wiring, stored as plain numbers in the weights. Nothing in ②–③ is a probability. Probabilities are created at step ④, on the fly, for each prompt.
neuron activation (after the nonlinearity)direction written back into the vectornext-word probability
Illustrative numbers, simplified. In a real model a fact is spread across many neurons and several middle layers, and each neuron is involved in many unrelated facts.