Interview Questions: Neural Networks

32 questions on “Neural Networks” — with short answers and a link to an in-depth explanation of each topic.

How to use it: cover the answer, answer out loud within a minute, then compare. All sections are on the interview questions page. Topics marked “in Russian” are not translated yet.

• From a Neuron to a Multilayer Perceptron · 4
• Activation Functions · 4
• Backpropagation · 4
• Optimizers: SGD, Momentum, Adam · 4
• Training in Practice: Batches, Dropout, Batchnorm · 4
• Recognizing Digits from Pixels · 4
• Convolutional Neural Networks (CNN) · 4
• Recurrent Neural Networks (RNN, LSTM) · 4

From a Neuron to a Multilayer Perceptron (in Russian)

• “Why can't a perceptron solve XOR?” — its decision boundary is a hyperplane, and the XOR classes are not linearly separable; you need a hidden layer that reshapes the feature space.
• “What happens if you remove the activations from a 10-layer network?” — a product of linear maps is linear; the network is equivalent to a single linear layer.
• “What does the universal approximation theorem guarantee?” — the existence of an approximation of a continuous function by a wide MLP; it guarantees neither a reasonable network size nor that gradient descent will find those weights.
• “Why the bias?” — without it the decision boundary has to pass through the origin.

Activation Functions (in Russian)

• “What's wrong with sigmoid in hidden layers?” — saturation (maximum derivative 0.25 → vanishing gradients) and non-centered output (zigzag descent).
• “What is a dead ReLU and how do you fix it?” — a neuron stuck at zero forever; Leaky ReLU/GELU, a careful learning rate, sensible initialization.
• “Why is softmax not an ordinary activation?” — it ties all components of the vector together: output i depends on all logits, and they sum to 1.
• “What is used in transformers?” — GELU/SiLU in the FFN blocks; softmax in the attention mechanism.

Backpropagation (in Russian)

• “Explain backprop in one sentence” — recursive application of the chain rule from the loss to the weights, reusing intermediate values; one backward pass costs about as much as two forward passes.
• “How is backprop different from SGD?” — backprop computes the gradients, the optimizer takes the step. These are two different stages of the training loop.
• “Why were deep networks hard to train and what helped?” — vanishing gradients; ReLU, normalization, initialization and residual connections helped.
• “Why zero_grad() in PyTorch?” — gradients accumulate between backward calls by default, so they must be zeroed before a new step.

Optimizers: SGD, Momentum, Adam (in Russian)

• “How is Adam different from SGD with momentum?” — Adam adds a second moment (the mean of squared gradients) and divides the step by its square root: an adaptive per-parameter scale + bias correction.
• “Why bias correction in Adam?” — m and v are initialized to zero and are underestimated in the first steps; the 1/(1−β^t) correction removes the bias.
• “What is the difference between Adam and AdamW?” — weight decay is applied separately from the gradient rather than through an L2 penalty, which gets distorted by the adaptive normalization.
• “Why warmup?” — it protects against blowing up in the first steps, before the moment statistics have accumulated.

Training in Practice: Batches, Dropout, Batchnorm (in Russian)

• “Why can't you initialize weights with zeros?” — symmetry: identical neurons get identical gradients and never become different.
• “What does dropout do at inference?” — nothing: it is turned off, and the scale is compensated by dividing by 1 − p during training.
• “Why is layer norm better than batch norm for transformers?” — it does not depend on batch size or sequence length, has no running statistics and no train/eval differences.
• “The loss became NaN at step 50 — what do you do?” — lower the lr, enable warmup and gradient clipping, check the data for outliers/zeros inside logarithms.

Recognizing Digits from Pixels (in Russian)

• “How many parameters does a linear MNIST classifier have?” — 784×10 weights + 10 biases = 7850. The ability to quickly count parameters is tested all the time.
• “What will we see if we visualize the weights as images?” — blurry digit templates: positive weights where the class's stroke usually goes.
• “Why softmax + cross-entropy and not MSE?” — cross-entropy gives a strong gradient on a confident mistake and works correctly with class probabilities.
• “Why normalize the pixels?” — a uniform input scale speeds up and stabilizes gradient descent; the predictions of an ideal model would not change with scale.

Convolutional Neural Networks (CNN) (in Russian)

• “Count the parameters of a convolutional layer” — (K·K·C_in + 1)·C_out: for example, a 3×3 kernel, 3 input and 16 output channels → (9·3+1)·16 = 448. Compare it with 150M for a fully connected layer — a striking argument.
• “What will a 16×16 output become after a 3×3 convolution without padding?” — 14×14: the formula is (N − K)/stride + 1.
• “Why pooling?” — to shrink the maps (cheaper), widen the receptive field and add robustness to small shifts.
• “How is a CNN better/worse than a ViT?” — CNN: strong inductive bias, good with small data and on weak hardware; ViT: more flexible and scales better on large datasets.

Recurrent Neural Networks (RNN, LSTM) (in Russian)

• “Write the formula for an RNN step” — h_t = tanh(W·x_t + U·h_t−1 + b); stress that W and U are shared across all steps.
• “Why do gradients vanish?” — in BPTT the gradient is multiplied by the transition Jacobian T times; a product of factors < 1 decays exponentially.
• “Why does an LSTM have three gates?” — controlled writing/erasing/reading of memory; the additive cell update gives the gradient a “highway” through time.
• “Why did the transformer win?” — parallel training over the whole length and direct access to distant context instead of T sequential transitions.