RNN, LSTM, GRU, CNN

One cell, reused at every time step

The same W_h, W_x and W_y are used in every box. Only h, the state, changes from step to step. It's the network's entire memory of everything it has read so far.

Experiment: how long does the first input survive?

A 1-number RNN, . Two runs are identical except for the first input (x₁ = +1 vs −1), followed by 29 steps of the same random noise. If the gap between the two runs closes, the network has forgotten x₁.

recurrent weight w activation φ

state h_t for both runs

gradient size |∂h_t/∂h₁| (log scale)

This product is the whole story. Multiply many numbers below 1 and you get a gradient that vanishes: the model can't learn that x₁ mattered 30 steps later. Above 1 (try "linear" with w > 1) it explodes. Tanh also saturates: when |h| is near 1, φ′ ≈ 0, which kills the gradient even when w is large.

The fix: a separate memory lane, the cell state c

σ squashes to 0–1, so each gate is a learned dial (0 = closed, 1 = open), set separately for each memory slot at each step.

Same experiment: write x₁ at step 1, then keep reading noise

forget gate f input gate i (later steps)

cell state c_t for both runs

|∂c_t/∂c₁|: LSTM (solid) vs plain tanh RNN, w = 1 (dashed)

c is updated by adding, not by being squashed through a nonlinearity again. When f ≈ 1, memory and gradient both pass through almost untouched. That's the same trick as the residual connection (x + f(x)) in transformers. LSTMs are usually initialized with b_f ≈ 1, so they start out remembering.

GRU: a simplified LSTM with 2 gates and no separate cell

The last line is linear interpolation: h_t lies on the line between the old state and the candidate. (1 − z) acts as the LSTM's forget gate and z as its input gate, linked so they always add up to 1.

Drag the gates: a 2-number hidden state you can see

update z reset r

z = 0: copy the old state exactly. Perfect memory, and the gradient passes through unchanged.
z = 1: replace it with the candidate.
r = 0: the candidate ignores the old state, so it starts fresh from x alone.

Over t steps with a constant z, the old memory shrinks like (1 − z)^t. It's the same "keep-fraction product" as the LSTM's ∏f. GRUs have fewer parameters and often perform about the same as LSTMs.

Convolution: one small filter, slid across every position

filter K

input x (8×8)

∗

filter K (3×3, learned)

output y (6×6): hover/tap

The filter is a pattern detector, and each output is a dot product between the filter and one patch (the same operation as Q·K). The same 9 weights are reused at every position, which is weight sharing over space, just as an RNN shares weights over time. A CNN learns many filters per layer. Early layers learn edges, and later layers combine them into shapes and objects.

Stacking layers widens what each unit can "see"

layer type

For text, CNNs slide along the sequence. Their view is a fixed window that grows with depth, which makes them a learned cousin of n-grams. They run fully in parallel, which RNNs can't, but they can't see past their window.

How each architecture gets at context

	What it sees	Memory	Parallel?	Main weakness
n-gram	last n−1 words	count tables	yes	no generalization; fixed window
CNN	window that grows with depth	none beyond the window	yes	can't see beyond its window
RNN	all past, in theory	one fixed vector, overwritten every step	no (sequential)	vanishing/exploding gradients
LSTM / GRU	all past, with gates	gated fixed vector; additive path	no	still sequential; still one bottleneck vector
Transformer	all past, directly	every past token kept (KV cache)	yes (in training)	cost grows with length² (every token vs. every token)

Three ideas that keep coming back

Weight sharing. CNNs reuse a filter across space, RNNs reuse a cell across time, and transformers reuse the same block at every position. Fewer parameters means it generalizes.
An additive path for memory and gradients. LSTM: c = f·c + …; GRU: h = (1−z)·h + …; ResNet and transformer: x + f(x). Every architecture that got deep needed one.
Gating and weighting by learned relevance. LSTM gates decide what to keep, and attention weights decide what to look at. Attention is basically a soft gate over all past positions, computed fresh for every query.