Recurrent Neural Networks (RNN, LSTM)

Before the transformer era, sequences — text, speech, time series — were handled by recurrent neural networks (RNNs). Their main idea, “memory in the hidden state”, is still alive: without it you cannot understand either LSTMs on legacy production systems or modern state space models.

Intuition

An ordinary network sees the whole input at once. But text arrives one token at a time, and its length is unknown in advance. An RNN reads a sequence the way a person does — left to right, element by element — and keeps a “summary” of what it has read: a hidden-state vector h. At every step the network mixes the new input with the old summary and writes an updated summary:

Here x_t is the current input (e.g. a one-hot character vector), h_t−1 is memory from the previous step, and the matrices W, U and bias b are the same at every step. The prediction (say, of the next character) is made by another linear layer on top of h_t with softmax.

Important: the size of h is fixed. The entire history, however long, is squeezed into one vector — that is both the strength (constant memory) and the weakness (a bottleneck: details of the distant past get overwritten).

How an RNN is trained: unrolling in time

To apply backpropagation, the RNN is “unrolled” in time: a chain of T steps becomes a deep network of T identical layers with shared weights, and the gradient flows back through it — from the last step to the first. This is called backpropagation through time (BPTT). The gradient w.r.t. U is summed over all steps — because there is only one matrix.

Here lies the main trouble: crossing T steps, the gradient is multiplied T times by the transition Jacobian (essentially by U and tanh derivatives, which are ≤ 1). If the factors average below one — the gradient vanishes, and the network cannot learn dependencies tens of steps long. If above — it explodes; gradient clipping saves you from explosion, and more serious architectural crutches save you from vanishing.

Notice: the “aabb” pattern cannot be predicted from the current character alone — after “a” comes either “a” or “b” depending on what came before. That is why the network needs memory h: it must encode “is this the first ‘a’ or the second”. A plain bigram model would be stuck at 50% forever here.

LSTM and GRU: gated memory

LSTM (long short-term memory) fixes vanishing gradients architecturally. Alongside the hidden state there is a cell state — a “conveyor belt” that information flows through across steps almost unchanged — and three gates, each a small sigmoid layer with values from 0 to 1:

• Forget gate — what to erase from memory (“the sentence ended — forget its subject”);
• Input gate — what from the new input to write into memory;
• Output gate — which part of memory to show outside at this step.

Because memory is updated additively (something added, something erased) rather than through repeated multiplication, the gradient flows along the conveyor for tens and hundreds of steps. GRU (gated recurrent unit) is a simplification with two gates: a bit faster, usually comparable in quality.

Why transformers displaced RNNs — and where RNNs live on

RNNs have an irremovable flaw: step t cannot be computed before step t−1. Training is fundamentally sequential and does not parallelize along the text length — the GPU sits idle. Attention in transformers looks at all positions at once: training parallelizes fully, and a dependency between tokens 1000 steps apart is one operation, not 1000 hops through memory.

But the RNN idea did not die: streaming speech recognition and signal processing on small devices love recurrence for constant memory per step (no growing KV-cache). And state space models (SSM, e.g. Mamba) are essentially modern linear recurrent nets: inference like an RNN (step by step, cheap), training parallel like a transformer.

All interview questions in this section: Neural Networks — interview questions