Sequences & memory

A convolution assumes a fixed grid. But a sentence can be three words or three hundred, and the meaning of"it" can depend on something said far earlier. Padding every input to a fixed length and feeding it to a dense net throws away the one structure that matters most in language: order, and the fact that the same word means different things in different positions. We need a network that reads asequence step by step and remembers. The recurrent neural network(RNN) does exactly this: it keeps a hidden state — a running summary of everything seen so far — and updates it at every step from the previous state plus the new input, reusing one set of weights the whole way through.

6.1 Recurrence: a memory that carries — and fades

At each step t the RNN takes the previous hidden state h_t−1 and the new input x_t, mixes them through a weight, and squashes the result with a tanh. The same weights are reused at every step — weight sharing again, but across time instead of space, which is what lets one small network read a sequence of any length.

Here is the first thing to feel before any talk of gradients. Suppose a single important input arrives at one step — a name, a subject, an open bracket — and after that the inputs go quiet. How long does the hidden state keep any trace of it? With no new input the update is just h_t = tanh(w·h_t−1), and near zero that behaves like multiplying by wevery step. If w is well below 1 the memory of that input fades in a handful of steps; as w approaches 1 it lingers for dozens. Drag the recurrence weight below and watch how far a single pulse of input reaches into the future.

That fading is the forward face of the problem. The hidden state is the network's only memory, and a plain recurrence can only hold a memory as long as its own dynamics allow. Push wup to keep memories longer and you are one step from instability; pull it down for stability and the network forgets almost immediately. The backward pass, where we actually train, has exactly the same knife-edge — and it is even less forgiving.

6.2 Backprop through time: the vanishing gradient

To train the RNN we need the gradient of the final loss with respect to the weights, which means tracing the error backward through every step — backpropagation through time. The gradient reaching anearly hidden state h_k has to pass through every step betweenk and the end, and each passage multiplies it by that step's local derivative:

That product is the whole story. Each factor is w times atanh derivative that is never bigger than 1 — and near the origin, where the activations are small, that derivative is essentially 1, so the factor is just w and the product collapses to w^T−k. That is the classic spectral-radius result: if w < 1 the gradient vanishes — early steps get almost no learning signal, so the network literally cannot learn long-range dependencies — and ifw > 1 it explodes. There is only a razor's edge at exactly 1. Drag the recurrent weight and the sequence length below and watch the gradient decay backward through time on a log scale — a straight diagonal is exponential collapse. Then flip the toggle toLSTM to preview the fix.

6.3 The LSTM: a gated highway for memory

The fix, from Hochreiter & Schmidhuber in 1997, is to add a second state that the network can carryalmost unchanged — the cell state c_t. Instead of overwriting memory every step through a tanh, small learned gates (each asigmoid that outputs a value between 0 and 1) decide what to keep, what to add, and what to read out. The forget gate f_t multiplies the old cell state; the input gate i_t scales new candidate content g_t; the output gateo_t exposes a filtered view as the hidden state.

The panel below is one LSTM cell. The cell state runs along the top as a highway; each gate is a valve on it. Set the gates and watch the update c ← f·c + i·g apply, and watch the right-hand plot run the same fixed gates for many steps so you can see where the memory settles. Two settings are worth finding by hand: with the forget gate at 1 and the input gate at 0the old value is copied forward untouched — the memory is simply held, indefinitely. And when a cell is held that way, the backward factor along it is just f ≈ 1 — no tanh derivative crushing it — so the gradient flows almost losslessly. Olah called this the constant error carousel: a lane that carries the signal the whole way back.

This is why the LSTM changed everything from 2014 to 2017 — machine translation, speech recognition, Karpathy's character-level text generation. A well-tuned LSTM (or the lighter GRU, which merges the cell and hidden state into one and uses two gates instead of three) can hold a dependency across a hundred steps that a vanilla RNN loses in ten. In practice you also stack a few layers for depth and often run one LSTM forward and one backward (bidirectional) so each position sees both past and future context.

6.4 Taming the explosion: gradient clipping

Gating fixes vanishing. The exploding side has a blunter, cheaper fix. Pascanu, Mikolov & Bengio pictured the loss surface of a recurrent net as mostly gentle with the occasional cliff — a wall where the gradient suddenly becomes enormous. A normal-sized learning rate times a giant gradient is a giant step, and the parameters are flung far across the landscape, undoing all prior progress in a single update. The fix is gradient clipping: if the gradient's norm exceeds a threshold, rescale it back down to that threshold before stepping. Same direction, sane length. Below, a ball rolls down toward a cliff on the right; run it with clipping off and the huge gradient at the wall launches it far to the left, then toggle clipping on and watch it descend under control.

Clipping is one line of code and it is still standard in every recurrent and transformer training loop today — the gates solved the slow leak, clipping caps the rare blowout. Together they made deep sequence learning trainable at all. But notice what recurrence never escaped: it is sequential. Step t cannot start until step t−1 is done, so a long sequence is a long dependency chain no matter how clever the cell. That single constraint is what attention was invented to break.