Gradient Descent
Gradient descent for interviews: the update step, learning rate, local minima, batch/SGD, and why the same algorithm trains everything from regression to neural nets.
Intuition: descending a mountain in fog
Imagine you are standing on a mountainside in thick fog and want to get down to the valley. There is no map — you can only feel the ground under your feet. A sensible strategy: feel which way the slope drops most steeply, take a step that way, and repeat until you are in the lowland.
In machine learning the “mountain” is the surface of the loss function L(θ): every combination of model parameters θ has its own height — the size of the error. The derivative lets you “feel the slope”: it says how the loss changes when a parameter changes a little. With many parameters there are many such derivatives, and the vector of them is called the gradient ∇L — it points in the direction of steepest ascent. So to go downhill, you walk against it.
How it works
The whole algorithm is one line, repeated in a loop:
Here η (eta) is the learning rate: the multiplier that sets the step length. The minus is there because the gradient looks uphill and we go downhill. Choosing η is the main practical question:
- Too small — the descent is correct but painfully slow: thousands of steps where dozens would have been enough.
- Moderate — fast, stable approach to the minimum.
- Too large — the step overshoots the valley onto the opposite slope, the error starts to grow, and the process diverges: the loss flies off to infinity.
Run three experiments: 1) η ≈ 0.05 — the ball crawls and gets stuck in the nearest valley; 2) η ≈ 0.3–1 — steps jump over shallow pits and find a deeper valley; 3) η > 7 — every step flings the ball higher, the loss grows: that is divergence.
Local minima and saddles
Descent is guaranteed to find a global minimum only for convex functions — like MSE in linear regression. Neural-net loss surfaces are rugged: there are local minima — valleys you cannot leave with a pure gradient step — and saddle points — places where the gradient is nearly zero but it is not a minimum: the function rises in some directions and falls in others. In high dimensions saddles are far more common than local minima, and they are what slow training down. In practice the noise of the stochastic gradient, momentum and the adaptive methods from the topic on optimizers help.
SGD, batch and mini-batch
An honest gradient is computed over the entire dataset (batch gradient descent) — accurate, but expensive: one step needs a pass over millions of examples. The other extreme is stochastic gradient descent (SGD): the gradient is estimated from a single random example. Steps are noisy but very cheap, and the noise is sometimes even useful — it helps escape bad local minima.
The industrial standard is a compromise: mini-batch gradient descent, where the gradient is averaged over a small batch of examples (usually 32–1024). That both reduces estimate noise and maps cleanly onto GPU parallelism. When papers say “SGD”, they almost always mean the mini-batch variant. The engine of deep-network training is the same one: backpropagation only computes the gradient efficiently; the step itself is still the same descent.
- “Why is the step taken against the gradient?” — the gradient points in the direction of steepest increase of the function, and we are minimizing.
- “What happens with a too large/too small learning rate?” — divergence or oscillation / slow convergence and getting stuck. An ideal answer mentions a learning rate schedule: start larger, decrease towards the end.
- “Why is SGD better than full-batch gradient?” — a step is thousands of times cheaper, and the noise helps explore the surface; convergence per step is worse, but per unit of time it is faster.
- “Why can't we just solve ∇L = 0 analytically?” — neural networks have no closed-form solution; the equations are nonlinear in billions of parameters.
All interview questions in this section: Machine Learning Fundamentals — interview questions