Activation Functions
An activation is the nonlinearity between layers without which a neural net collapses into a linear model. The choice of activation determines how the gradient flows, so “sigmoid or ReLU and why” is a must-ask interview question.
Intuition
A linear layer can only rotate, stretch and shift space. An activation function adds “kinks” and “bends” — and from those the network builds complex decision boundaries. A second, less obvious role: the activation steers the gradient. On the backward pass the gradient is multiplied by the activation's derivative at every point — if the derivative is near zero, that layer stops learning.
How it works: the activation zoo
Sigmoid — σ(x) = 1 / (1 + e−x). Squashes the input into (0, 1); historically the first “smooth step”. Two problems: first, saturation — for |x| > 4–5 the derivative is nearly zero (its maximum is only 0.25 at zero), so the gradient does not get through that neuron; second, the output is not zero-centered — all outputs are positive, so weight gradients of the next layer share the same sign and descent zigzags.
Tanh — tanh(x), range (−1, 1). Zero-centered (better for optimization), but saturates exactly like sigmoid.
ReLU (rectified linear unit) — max(0, x). Cheap, does not saturate on the right (the derivative there is exactly 1); with it deep nets first started training properly. The downside is dead neurons: if a neuron lands in the negative region on every input, its gradient is identically zero and it never recovers.
Leaky ReLU — max(αx, x) with a small α ≈ 0.01: in the negative region a tiny slope remains, so the neuron does not die for good.
GELU / SiLU — smooth “blurred ReLUs”: GELU(x) = x·Φ(x) (Φ is the normal CDF), SiLU(x) = x·σ(x). These are what modern transformers use (BERT, GPT, LLaMA): smoothness and a small dip below zero give slightly better quality in practice.
Separately, softmax is not an elementwise activation but a normalization of a logit vector into a probability distribution: softmax(z)i = ezi / Σj ezj. Each component depends on all inputs, and the outputs sum to 1. Softmax belongs in the output layer of multiclass classification (and in transformer attention), not in hidden layers.
How to choose in practice
The default rule is simple. Hidden layers: ReLU — cheap and cheerful for MLPs and CNNs; GELU/SiLU — the transformer standard. The output layer is dictated by the task: regression — no activation (linear output), binary classification — sigmoid + binary cross-entropy, multiclass — softmax + cross-entropy. Sigmoid and tanh in the hidden layers of deep nets are almost never seen today — precisely because of saturation: a chain of derivatives, each ≤ 0.25, turns the gradient to dust over a dozen layers (more in the topic on backprop).
Play with the interactive: raise “input scale” and you see how large-magnitude inputs drive sigmoid and tanh into saturation — almost the whole axis turns red. That also explains why inputs and activations are normalized (batchnorm/layernorm): to keep values in the activation's “live” zone. Notice ReLU too: exactly the whole negative half-axis turns red — that is the zone where a neuron risks dying — but on the right the derivative is one at any input scale, and the gradient passes without attenuation.
- “What's wrong with sigmoid in hidden layers?” — saturation (maximum derivative 0.25 → vanishing gradients) and non-centered output (zigzag descent).
- “What is a dead ReLU and how do you fix it?” — a neuron stuck at zero forever; Leaky ReLU/GELU, a careful learning rate, sensible initialization.
- “Why is softmax not an ordinary activation?” — it ties all components of the vector together: output i depends on all logits, and they sum to 1.
- “What is used in transformers?” — GELU/SiLU in the FFN blocks; softmax in the attention mechanism.
All interview questions in this section: Neural Networks — interview questions