ML Engineer Interview Questions

296 questions asked in interviews on machine learning, statistics, neural networks, LLMs, RAG and AI agents — with short answers and a link to the topic. Collected from the “In the interview” boxes of all 74 textbook topics.

How to use it: cover the answer, answer out loud within a minute, then compare. Topics marked “in Russian” are not translated yet.

• Machine Learning Fundamentals · 72 · separate page
• Statistics for ML · 12
• Neural Networks · 32 · separate page
• Language Models: From N-grams to the Transformer · 24 · separate page
• LLMs: Training and Inference · 40 · separate page
• Multimodal Models · 28 · separate page
• Hardware and Deployment · 12
• Agents and Tools · 20 · separate page
• RAG: Retrieval + Generation · 32 · separate page
• Production and Quality · 24 · separate page

Machine Learning Fundamentals

What Is Machine Learning (in Russian)

• “How is ML different from classical programming?” — the rules are not written by hand; they are derived from data by optimizing parameters.
• “Give examples of tasks for each paradigm” — supervised: spam filter, price prediction; unsupervised: customer segmentation, anomaly detection; RL: games, robotics, RLHF.
• “Why have separate validation and test sets?” — hyperparameters are tuned on validation, so it is “worn out” by that tuning; the test set has to stay untouched for the final evaluation.
• “Classification or regression?” — look at the type of output: a discrete class or a continuous number. Trick question: predicting click probability is regression in form, but it solves a classification task.

Types of Learning: Supervised, Unsupervised and Reinforcement (in Russian)

• “How is self-supervised different from unsupervised?” — self-supervised has a target label extracted from the data and an explicit prediction loss; unsupervised has no label, only structure. Tasks like clustering are unsupervised; masking and next-token prediction are self-supervised.
• “Why is an LLM self-supervised?” — the model is trained to predict the next token; the “correct answer” is the text itself, nothing has to be labeled by humans, so the internet becomes free labeled data.
• “What is exploration/exploitation?” — the RL dilemma: explore new actions for a potentially bigger reward, or exploit what has already been found to work.
• “Why transfer learning?” — to reuse learned representations so you can train on small data faster and more accurately than from scratch.

Linear Regression (in Russian)

• “Why MSE and not MAE?” — the square is differentiable and penalizes large errors more; MAE is more robust to outliers.
• “When the normal equation, and when gradient descent?” — the closed form is good with few features; descent scales to large data and streaming.
• “What happens with correlated features?” — the weights are unstable; regularization (L2/ridge) or feature selection helps.
• “How will the model react to an outlier?” — it will shift a lot because of the squared error; check in the interactive, mentally, where a point high above the rest would pull the line.

Gradient Descent (in Russian)

• “Why is the step taken against the gradient?” — the gradient points in the direction of steepest increase of the function, and we are minimizing.
• “What happens with a too large/too small learning rate?” — divergence or oscillation / slow convergence and getting stuck. An ideal answer mentions a learning rate schedule: start larger, decrease towards the end.
• “Why is SGD better than full-batch gradient?” — a step is thousands of times cheaper, and the noise helps explore the surface; convergence per step is worse, but per unit of time it is faster.
• “Why can't we just solve ∇L = 0 analytically?” — neural networks have no closed-form solution; the equations are nonlinear in billions of parameters.

Logistic Regression and Classification (in Russian)

• “Why doesn't linear regression with a threshold work for classification?” — the output is not bounded to [0, 1], MSE wrongly penalizes confident correct answers, and distant points shift the boundary.
• “Why cross-entropy and not MSE?” — with a sigmoid, MSE is non-convex and gives vanishing gradients on confident mistakes; cross-entropy is convex and strongly penalizes confident mistakes.
• “Is the logistic regression boundary linear?” — yes: σ is monotonic, so the condition p = 0.5 is equivalent to w·x + b = 0. Nonlinearity is achieved by adding features (x², products).
• “How are sigmoid and softmax related?” — softmax for two classes reduces to a sigmoid; the sigmoid is a special case.

Bayes' Theorem and Naive Bayes (in Russian)

• “A disease affects 1%, the test is 95% accurate, the test is positive — what is the probability of disease?” — work it out step by step on 10,000 people: 95 true positives versus 495 false positives → 95/590 ≈ 16%. Name the effect: base rate neglect.
• “How is a generative model different from a discriminative one?” — a generative model learns P(x|y)·P(y) and derives the answer via Bayes (NB); a discriminative model learns P(y|x) directly (logistic regression).
• “Why Laplace smoothing?” — without it, a word never seen in a class gives zero probability and zeroes out the whole product.
• “Why compute in logarithms?” — a product of hundreds of small probabilities underflows; a sum of logarithms is numerically stable.

Overfitting and Regularization (in Russian)

• “How do you tell overfitting from underfitting?” — by the pair of train/val errors: a large gap is overfitting, both high is underfitting.
• “How is L1 different from L2?” — L1 zeroes out weights and selects features, L2 shrinks all weights evenly; L2 is smooth and easier to optimize.
• “What do you do if the model overfits?” — in order of increasing cost: regularization/early stopping → simplify the model → collect more data.
• “Why k-fold if you have a train/val split?” — on small data a single split is noisy; averaging over k folds gives a stable estimate and lets you tune hyperparameters fairly.

Bias–Variance: Decomposing the Error (in Russian)

• “What is the bias-variance trade-off?” — expected error = bias² + variance + noise; increasing complexity lowers bias but inflates variance; the optimum is at the bottom of the U-curve.
• “Is overfitting high bias or high variance?” — high variance: the model is perfect on train but jumps around from sample to sample. High bias is underfitting.
• “How do ensembles affect the decomposition?” — bagging/RF average models and reduce variance; boosting builds up accuracy and reduces bias.
• “Can the noise σ² be removed?” — no, it is an irreducible limit caused by randomness in the data itself; error on new data will not go below it.

Quality Metrics (in Russian)

• “Classes 1:100, accuracy 0.99 — is the model good?” — no: the constant “everyone is healthy” gives the same. Look at precision/recall, PR-AUC.
• “What happens to precision and recall when you raise the threshold?” — precision usually goes up, recall goes down: we catch fewer, but with more confidence.
• “How do you explain AUC?” — the probability that a random positive is ranked above a random negative; AUC says nothing about a specific threshold.
• “Which metric would you use for X?” — start with the cost of FP and FN errors in money/risk, then name the metric, not the other way around.

Cross-Validation and Hyperparameter Tuning (in Russian)

• “Why do you need validation if you have a test set?” — the test set is touched once for an honest final estimate; tuning on the test set is leakage and inflates quality. Hyperparameters are chosen on validation/CV.
• “What is nested CV and when is it needed?” — the outer loop fairly evaluates the pipeline, the inner loop tunes hyperparameters; it is needed so that the estimate is not inflated by the very act of picking the best option.
• “Grid search or random search?” — with low dimensionality and discrete values, grid; in a high-dimensional space where only a couple of parameters matter, random search (Bergstra) or Bayesian optimization is more efficient.
• “How do you cross-validate a time series?” — only forward-chaining/walk-forward; random folds leak information from the future.

Decision Trees and Ensembles (in Russian)

• “How is Random Forest different from bagging?” — it adds a random subset of features at each node: trees are more diverse and less correlated.
• “RF or boosting — which when?” — RF: a fast, robust baseline with minimal tuning. Boosting: maximum quality on tabular data, but tune the learning rate, depth and early stopping.
• “Why can't boosting train trees in parallel?” — each tree depends on the errors of the previous ones; the parallelism is inside a single tree.
• “Why does boosting beat neural networks on tables?” — heterogeneous features, categories, missing values, small and medium samples are where trees shine; networks need more data and preprocessing.

SVM and the Kernel Trick (in Russian)

• “What are support vectors?” — points on the edge of the street (or inside it with a soft margin); only they define the boundary, removing the others does not change it.
• “Explain the kernel trick in words” — the data is implicitly mapped into a high-dimensional space where it becomes linearly separable, and the kernel cheaply computes dot products there without building the mapping itself.
• “What do C and gamma do?” — C penalizes margin violations (large C → narrow street, overfitting), gamma sets the radius of a point's influence in RBF (large gamma → islands around points, overfitting).
• “Why isn't SVM used on large data?” — the n×n kernel matrix and O(n²)–O(n³) training.

kNN and Clustering (in Russian)

• “Why is kNN called lazy?” — there is no training phase: the model stores the sample and all the work happens at query time; hence the expensive O(n) prediction.
• “What happens to kNN without feature normalization?” — the distance is dominated by the feature with the largest scale, and the others stop mattering.
• “Does k-means guarantee a global minimum?” — no, only a local one; this is addressed with k-means++ and several restarts.
• “Half-moon shaped clusters — what do you use?” — not k-means (it is for convex spheres), but DBSCAN or spectral clustering.

The EM Algorithm and Gaussian Mixtures (GMM) (in Russian)

• “Why is GMM better than k-means?” — soft assignment (probabilities instead of hard labels) and elliptical clusters of different shape, tilt and size thanks to the full covariance; k-means is its degenerate special case.
• “What do the E- and M-steps do?” — E, with parameters fixed, computes responsibilities (soft memberships); M, with responsibilities fixed, updates the centers, covariances and weights; each cycle does not decrease the likelihood.
• “Why does EM get stuck in a local optimum?” — the likelihood is non-convex; EM increases it monotonically, but only up to the nearest local maximum, so the result depends on the start — k-means++ and several runs help.
• “How do you choose the number of components?” — not by likelihood (it grows with k), but by BIC/AIC with a complexity penalty.

Dimensionality Reduction and PCA (in Russian)

• “What does PCA maximize?” — the variance of the projections; equivalently, it minimizes the mean squared error of linear reconstruction.
• “What are principal components mathematically?” — eigenvectors of the covariance matrix; the eigenvalues are the variances along them.
• “Why standardize before PCA?” — without it, the components will be dominated by features with large units of measurement.
• “Can you trust distances between clusters on t-SNE?” — no: the method preserves local neighborhoods, the global geometry is distorted.

Features and Data Leakage (in Russian)

• “AUC 0.99 offline, 0.6 in production — what happened?” — a classic: leakage. Check features filled in after the event, the split scheme and preprocessing done before the split.
• “How do you do target encoding without leakage?” — out-of-fold: the encoding for a row is computed on folds without it; plus smoothing for rare categories.
• “Why is a random split of time data a lie?” — the model sees the future: the offline metric is inflated, and that information will not exist in production.
• “How do you look for leaks in practice?” — audit features by when they become available, look at suspiciously important features, compare the metric under a time-based and a random split.

Time Series and Forecasting (in Russian)

• “How do you validate a sales forecasting model?” — only a time-based split, ideally walk-forward: several consecutive “train on the past — test on the future” windows.
• “Why is a naive baseline mandatory?” — it sets the bar almost for free; a complex model that doesn't beat seasonal naive adds no value.
• “Which features must not be used?” — anything that uses data after the prediction moment: centered windows, aggregates over “the whole period”, statistics computed over the whole series before the split.
• “What would you use to forecast 10,000 product series?” — boosting on lags and calendar features: one model for all series, and external features are trivial to add.

Reinforcement Learning: Q-learning and PPO (in Russian)

• “What are Q-learning and the Bellman equation?” — Q(s,a) is the expected return of an action; the Bellman equation Q(s,a)=r+γ·max Q(s′,·) links it to the next step; Q-learning fits Q to this target with the update Q ← Q+α·[r+γ·max Q(s′,·)−Q], off-policy (it learns from the max, not from the move actually taken).
• “Exploration vs exploitation?” — explore something new for a potentially bigger reward, or exploit the best found so far; the typical compromise is ε-greedy: with probability ε a random move, otherwise argmax.
• “How is RLHF related to RL?” — it is PPO against a reward model with a KL penalty: the LLM policy generates answers, the reward model scores them, PPO increases the reward, and KL keeps the model close to the reference.
• “How are policy methods different from value methods?” — value methods (Q-learning) learn the value of actions and take the argmax; policy methods (REINFORCE, PPO) parameterize and move the policy itself. For LLMs with a huge action space, it is policy methods that work.

Statistics for ML

Sampling, the CLT and Standard Error

• “State the CLT” — the mean of a sample of n independent observations with finite variance is approximately normally distributed, centered at μ with spread σ/√n, whatever the shape of the original distribution.
• “How is the standard error different from the standard deviation?” — SD is about the data, SE is about the estimate; SE = SD/√n and shrinks as the sample grows.
• “Model A scored 0.91, model B 0.92 on a test of 500 examples. Which is better?” — estimate SE ≈ √(0.9·0.1/500) ≈ 1.3 pp: a 1 pp difference is within the noise; you need a paired test or more data.
• “How many times larger must the sample be to halve the spread of the estimate?” — four times, because of the square root in the formula.

Confidence Intervals (in Russian)

• “What does a 95% confidence interval mean?” — if the procedure is repeated many times, 95% of the constructed intervals will cover the true value; saying “95% probability” about a specific interval is incorrect.
• “Why the t-distribution and not the normal?” — σ is estimated from the sample and is itself random; t with n − 1 degrees of freedom accounts for this uncertainty and converges to z for large n.
• “How do you get an interval for AUC or NDCG?” — bootstrap over the test set: resampling with replacement, percentiles of the metric's distribution.
• “How do you halve the interval?” — four times more data; lowering the confidence level also narrows the interval, but at the cost of reliability.

P-Values and Hypothesis Testing (in Russian)

• “What is a p-value?” — the probability of getting the observed or a more extreme result if H₀ is true; not the probability that H₀ is true, and not the probability of an error.
• “Type I and Type II errors — which is worse?” — it depends on the cost: a false alarm in medical screening is cheaper than a miss, while a false “the model is better” costs a release; α is chosen according to the cost of errors.
• “How do you fairly compare two models on the same test set?” — a paired test on the same examples: McNemar or a bootstrap of the difference; an interval for the difference, not two separate ones.
• “The result is significant at p = 0.01, but the effect is 0.01%. Do we ship it?” — statistical significance is not practical significance: with large n everything is significant; look at the effect size and the cost of shipping.

Neural Networks

From a Neuron to a Multilayer Perceptron (in Russian)

• “Why can't a perceptron solve XOR?” — its decision boundary is a hyperplane, and the XOR classes are not linearly separable; you need a hidden layer that reshapes the feature space.
• “What happens if you remove the activations from a 10-layer network?” — a product of linear maps is linear; the network is equivalent to a single linear layer.
• “What does the universal approximation theorem guarantee?” — the existence of an approximation of a continuous function by a wide MLP; it guarantees neither a reasonable network size nor that gradient descent will find those weights.
• “Why the bias?” — without it the decision boundary has to pass through the origin.

Activation Functions (in Russian)

• “What's wrong with sigmoid in hidden layers?” — saturation (maximum derivative 0.25 → vanishing gradients) and non-centered output (zigzag descent).
• “What is a dead ReLU and how do you fix it?” — a neuron stuck at zero forever; Leaky ReLU/GELU, a careful learning rate, sensible initialization.
• “Why is softmax not an ordinary activation?” — it ties all components of the vector together: output i depends on all logits, and they sum to 1.
• “What is used in transformers?” — GELU/SiLU in the FFN blocks; softmax in the attention mechanism.

Backpropagation (in Russian)

• “Explain backprop in one sentence” — recursive application of the chain rule from the loss to the weights, reusing intermediate values; one backward pass costs about as much as two forward passes.
• “How is backprop different from SGD?” — backprop computes the gradients, the optimizer takes the step. These are two different stages of the training loop.
• “Why were deep networks hard to train and what helped?” — vanishing gradients; ReLU, normalization, initialization and residual connections helped.
• “Why zero_grad() in PyTorch?” — gradients accumulate between backward calls by default, so they must be zeroed before a new step.

Optimizers: SGD, Momentum, Adam (in Russian)

• “How is Adam different from SGD with momentum?” — Adam adds a second moment (the mean of squared gradients) and divides the step by its square root: an adaptive per-parameter scale + bias correction.
• “Why bias correction in Adam?” — m and v are initialized to zero and are underestimated in the first steps; the 1/(1−β^t) correction removes the bias.
• “What is the difference between Adam and AdamW?” — weight decay is applied separately from the gradient rather than through an L2 penalty, which gets distorted by the adaptive normalization.
• “Why warmup?” — it protects against blowing up in the first steps, before the moment statistics have accumulated.

Training in Practice: Batches, Dropout, Batchnorm (in Russian)

• “Why can't you initialize weights with zeros?” — symmetry: identical neurons get identical gradients and never become different.
• “What does dropout do at inference?” — nothing: it is turned off, and the scale is compensated by dividing by 1 − p during training.
• “Why is layer norm better than batch norm for transformers?” — it does not depend on batch size or sequence length, has no running statistics and no train/eval differences.
• “The loss became NaN at step 50 — what do you do?” — lower the lr, enable warmup and gradient clipping, check the data for outliers/zeros inside logarithms.

Recognizing Digits from Pixels (in Russian)

• “How many parameters does a linear MNIST classifier have?” — 784×10 weights + 10 biases = 7850. The ability to quickly count parameters is tested all the time.
• “What will we see if we visualize the weights as images?” — blurry digit templates: positive weights where the class's stroke usually goes.
• “Why softmax + cross-entropy and not MSE?” — cross-entropy gives a strong gradient on a confident mistake and works correctly with class probabilities.
• “Why normalize the pixels?” — a uniform input scale speeds up and stabilizes gradient descent; the predictions of an ideal model would not change with scale.

Convolutional Neural Networks (CNN) (in Russian)

• “Count the parameters of a convolutional layer” — (K·K·C_in + 1)·C_out: for example, a 3×3 kernel, 3 input and 16 output channels → (9·3+1)·16 = 448. Compare it with 150M for a fully connected layer — a striking argument.
• “What will a 16×16 output become after a 3×3 convolution without padding?” — 14×14: the formula is (N − K)/stride + 1.
• “Why pooling?” — to shrink the maps (cheaper), widen the receptive field and add robustness to small shifts.
• “How is a CNN better/worse than a ViT?” — CNN: strong inductive bias, good with small data and on weak hardware; ViT: more flexible and scales better on large datasets.

Recurrent Neural Networks (RNN, LSTM) (in Russian)

• “Write the formula for an RNN step” — h_t = tanh(W·x_t + U·h_t−1 + b); stress that W and U are shared across all steps.
• “Why do gradients vanish?” — in BPTT the gradient is multiplied by the transition Jacobian T times; a product of factors < 1 decays exponentially.
• “Why does an LSTM have three gates?” — controlled writing/erasing/reading of memory; the additive cell update gives the gradient a “highway” through time.
• “Why did the transformer win?” — parallel training over the whole length and direct access to distant context instead of T sequential transitions.

Language Models: From N-grams to the Transformer

Bigrams and N-grams (in Russian)

• “What is a language model?” — a probability distribution over the next token given the context; everything else is details of the parameterization.
• “Why can't you just keep increasing n?” — the number of contexts grows as Vⁿ⁻¹, almost all long contexts are unique: the table is sparse and the counts are unreliable.
• “What is perplexity?” — the exponent of the average cross-entropy; intuitively, the average number of options the model is hesitating between. It is comparable only with the same tokenizer and test set.
• “How is GPT different from an n-gram model?” — the task is the same, but the distribution is parameterized by a neural network that generalizes to unseen contexts instead of exact string matching.

Tokenization and BPE (in Russian)

• “Why subwords and not words or characters?” — the vocabulary is finite, OOV disappears, sequences have moderate length: a compromise between the two extremes.
• “How does BPE tokenize a word that wasn't in the corpus?” — it splits it into characters and applies the learned merges in order; in the worst case the word stays a set of characters/bytes.
• “Why does an LLM make mistakes counting the letters in a word?” — to the model the word is one or two tokens; individual letters are not visible without explicit splitting.
• “What does the vocabulary size affect?” — sequence length versus the size of the embedding matrix and the quality of rare tokens.

Embeddings and word2vec (in Russian)

• “What's wrong with one-hot?” — all words are equidistant, the geometry carries no meaning, and the dimension equals the vocabulary size.
• “Why are ‘cat’ and ‘dog’ close if they rarely appear next to each other in text?” — what matters is not co-occurrence but similarity of contexts: they share neighbors.
• “Skip-gram vs CBOW?” — skip-gram predicts the neighbors from the word (better for rare words), CBOW predicts the word from the neighbors (faster).
• “Why are contextual embeddings better than static ones?” — ambiguous words get different vectors depending on the sentence.

The Attention Mechanism (in Russian)

• “Why divide by √d?” — without it the scores grow with the dimension, softmax saturates and the gradients are almost zero. Check it in the interactive: a small divisor ⇒ the weights collapse onto one token.
• “What are Q, K, V in your own words?” — a soft dictionary: query, key, value; the answer is a weighted sum of values by how similar the query is to the keys.
• “Why does the decoder need a mask?” — during training the “next token” is already in the data; without a mask the model cheats and doesn't learn to predict.
• “What is the complexity of self-attention?” — O(n²·d) in time and O(n²) in memory for the weight matrix; hence FlashAttention, GQA, sliding windows.

The Transformer Architecture (in Russian)

• “Draw a transformer block” — attention and an MLP, each wrapped as LN → sublayer → add to the input (pre-norm). Be sure to say that the residual is addition, not concatenation.
• “How is BERT different from GPT?” — the mask and the task: the encoder without a mask learns to reconstruct masked tokens, the decoder with a mask learns to predict the next one.
• “Why did the transformer replace RNNs?” — parallel training over the whole length + direct access to any token instead of a fading memory.
• “Where are the parameters?” — about ⅔ of a block in the MLP, ⅓ in attention; plus the embeddings, noticeable in small models and almost negligible in large ones.

Modern Architectures: RoPE, MoE, GQA (in Russian)

• “Why RoPE and not learned embeddings?” — relative positions without parameters, better length extrapolation; rotating Q/K makes the dot product depend only on the offset.
• “What does GQA reduce and what does that affect?” — the KV-cache (and memory traffic during decoding), i.e. inference speed and cost; quality barely suffers.
• “MoE — what's the trade-off?” — lots of knowledge with few FLOPs per token, but the whole model has to live in memory, plus router balancing and complex sharding.
• “How do models handle 128k context?” — RoPE scaling, GQA against cache bloat, FlashAttention against quadratic memory, sometimes a sliding window.

LLMs: Training and Inference

Pretraining and Scaling Laws (in Russian)

• “Why is pretraining called self-supervised?” — labels are not created by humans: the next token is the label, so all the text in the world is usable.
• “What is the Chinchilla finding?” — at fixed compute, N and D should grow together (~20 tokens per parameter); GPT-3 was undertrained on data.
• “Why deduplication?” — repeats make the model memorize, distort the loss and contaminate benchmarks that leaked into the training set.
• “Why isn't a base model usable as an assistant?” — its goal is to plausibly continue text, not to fulfil a request; the “helpful answer” format is given to it only in post-training.

Fine-tuning: SFT, LoRA, PEFT (in Russian)

• “Prompting, RAG or fine-tuning?” — in order of cost: first the prompt and few-shot, RAG for facts, fine-tuning when you need a stable style/format/domain and have the data and budget for quality evaluation.
• “How does LoRA work and why doesn't it slow down inference?” — ΔW = A·B of rank r is trained separately and after training is added to W — the architecture and latency don't change.
• “How much memory does a full fine-tune of a 7B model need?” — ~16 bytes per parameter with Adam and mixed precision: around 112 GB, hence the popularity of PEFT.
• “What is catastrophic forgetting and how do you mitigate it?” — loss of general skills during narrow fine-tuning; a small learning rate, mixing in general data, and LoRA instead of full FT help.

Alignment: RLHF and DPO (in Russian)

• “Why the KL penalty in RLHF?” — the RM is imperfect; without a leash the policy finds its holes (reward hacking) and degenerates. KL keeps the model close to the reference.
• “How is DPO different from RLHF?” — the same objective (preferences + KL), but a direct supervised loss on pairs: no separate RM, no PPO, no online generation — simpler and more stable.
• “What is sycophancy and where does it come from?” — the model learns to agree with the user because labelers more often prefer agreeable answers; it is a bias in the preference data, not the model's “character”.
• “Why is RLVR used for math and code?” — the reward is verifiable (the answer matches, the tests pass), so it can't be gamed with a nice-looking but wrong text.

Reasoning Models and Test-Time Compute (in Russian)

• “How is a reasoning model different from a regular one with a CoT prompt?” — a CoT prompt merely asks an already trained model to write out steps; a reasoning model is trained via RL (RLVR) to develop strategies — decomposition, self-checking, backtracking — and its reasoning actually finds and fixes its own mistakes.
• “When do you use a reasoning model and when a regular one?” — by task difficulty versus cost and latency: math, code, planning — yes; extraction, classification, chat that requires an instant reply — no.
• “What is RLVR and why no labelers?” — RL with verifiable rewards: the task's answer is checked, code is run against tests; the reward is objective, and human comparisons of answers aren't needed.
• “How else can you spend test-time compute besides long reasoning?” — in parallel: self-consistency (voting over N chains) and best-of-N with a scorer.

Sampling: Temperature, Top-k, Top-p (in Russian)

• “What does temperature do mathematically?” — it divides the logits before softmax: T < 1 sharpens the distribution, T > 1 flattens it; in the limits you get argmax and uniform.
• “Why is top-p better than top-k?” — the cutoff size adapts to the model's confidence instead of being fixed in advance.
• “Why does greedy decoding degenerate?” — a locally probable token ≠ globally good text; the typical symptom is repetition loops, fixed with sampling and penalties.
• “What parameters would you set for SQL generation?” — low temperature (≈0–0.2), possibly a lower top-p; diversity is harmful here.

Inference: KV-Cache, Quantization, Speculative Decoding (in Russian)

• “Why is decode slower than prefill?” — prefill is parallel across tokens and compute-bound; decode is one token per step and is bound by memory traffic (weights + cache for every token).
• “What does the KV-cache store and how big is it?” — keys and values of all past tokens across all layers and heads; its size is linear in context length (≈0.5 MB/token for a 7B model in fp16).
• “Does speculative decoding hurt quality?” — no: the acceptance rule preserves the large model's distribution; the speedup comes from parallel verification.
• “How do you evaluate serving?” — TTFT, tokens/s per request and total throughput; continuous batching and PagedAttention increase throughput specifically.

Distillation and Model Compression (in Russian)

• “How is distillation different from quantization?” — quantization compresses the numbers of the same model (fp16 → int4), distillation transfers knowledge into a different, smaller architecture by training on the teacher's predictions; they are combined.
• “Why temperature in distillation?” — to soften the teacher's softmax: at T = 1 the distribution is almost one-hot, while at T = 3–5 the relations between classes become visible — that very dark knowledge.
• “How do you transfer reasoning into a small model?” — sequence-level distillation: the teacher generates reasoning chains, the student is fine-tuned on them with SFT (this is how the R1 distills were made).
• “Can the student outperform the teacher?” — not on the teacher's distribution, that is the ceiling; the win is in cost and latency, and on a narrow domain the gap is minimal.

Evaluating LLMs: Benchmarks and LLM-as-Judge (in Russian)

• “How can you tell a benchmark is contaminated?” — a suspiciously high score with weakness on fresh/paraphrased tasks of the same type; checking n-grams against the training corpus.
• “What biases does an LLM judge have and how do you catch them?” — position (rerun with A/B swapped), length (correlation of the verdict with length), self-preference (compare judges from different model families).
• “How do you evaluate an LLM feature in production?” — a golden set of real queries, rubrics, an LLM judge calibrated against humans, regression runs for every release.
• “Model A beat B in 6 out of 10 comparisons — is it better?” — no, the sample is tiny: the confidence interval covers 50/50.

Working with Context and Message Markup (in Russian)

• “Why does role markup matter more than plain text?” — roles set priority and boundaries: the model is trained to treat system as rules and user as a request; without roles this information is lost.
• “What is a chat template?” — the format that wraps messages in special tokens (for example, ChatML with <|im_start|>/<|im_end|>), marking turn boundaries the way the model saw them in training.
• “Where do you put the important stuff in a long context?” — at the beginning and at the end: because of the lost-in-the-middle effect, the middle is used worse.
• “How is context engineering broader than prompt engineering?” — prompting is about wording the instruction, context is about assembling and marking up the whole input: roles, block order, delimiters, token budget.

Prompt Patterns and Structured Outputs (in Russian)

• “How do you get valid JSON from an LLM?” — a ladder: ask in the prompt → few-shot with examples → the provider's JSON mode → schema-constrained decoding; only the last level gives a guarantee, because it masks invalid tokens during sampling.
• “Why is few-shot stronger than a verbal instruction?” — the model continues patterns: examples set the format in exactly the form it must be reproduced, with no “words → behavior” translation.
• “What's wrong with a giant prompt for many tasks?” — instructions compete and contradict each other, errors can't be localized, eval is uninformative; fixed by decomposing into a chain of prompts.
• “Does a schema guarantee correct data?” — no: it guarantees the shape (parsing, field types), while correctness of the values is checked only by evaluation on a reference set.

Multimodal Models

How LLMs See Images (in Russian)

• “How does an image get into an LLM?” — it is cut into patches → each patch is linearly projected into a vector (ViT) → a projector aligns the vectors with the LLM's space → visual tokens go into the sequence together with text tokens.
• “What does CLIP pretraining give?” — contrastive training on image–text pairs puts images and captions into a shared space: the encoder “knows the meaning” of an image, and zero-shot classification works.
• “Why is an image expensive in tokens?” — the token count is N = (H/P)×(W/P): hundreds per image, thousands with tiling; that is both API cost and consumed context.
• “Why can't a VLM read small text?” — details smaller than a patch are lost during encoding; fixed by raising the resolution/tiling or pairing with OCR.

Audio: Speech Recognition and Synthesis (in Russian)

• “How does sound get into a neural network?” — waveform → windows → spectra → mel spectrogram (a time×frequency image); the modern alternative is discrete audio tokens via a neural codec.
• “What does the CTC loss do?” — it learns the alignment of frames and letters without boundary labels: a prediction for every frame, a blank token, collapsing repeats (“hhe-l-lll-o” → “hello”).
• “What is WER?” — (substitutions + deletions + insertions) / number of words in the reference; the main ASR metric, it can exceed 100%.
• “Why is Whisper so good?” — the scale of the data (weak supervision on hundreds of thousands of hours), not a clever architecture.

Omni Models: All Modalities in One (in Russian)

• “How is an omni model different from an LLM with adapters and from an ASR→LLM→TTS pipeline?” — pipeline: three models, latencies add up, only text passes between them. Adapters: an encoder + a projector to a frozen LLM, multimodal input, text output. Omni: one model, pretrained on interleaved tokens, both input and output are multimodal.
• “Why is GPT-4o's voice faster and livelier than the old voice mode?” — the old one was a pipeline of three models; omni hears audio tokens (intonation, pauses) and replies with audio tokens directly — hundreds of ms of latency instead of seconds.
• “What are early and late fusion?” — early: modalities are mixed into one token stream at the input of a single model; late: separate models per modality, combined at the end.
• “What is the main difficulty in training omni models?” — little interleaved data, expensive compute and balancing the mix so that quality on plain text doesn't drop.

OCR: From Classic Methods to Recognition with LLMs (in Russian)

• “Design passport/receipt recognition” — a pipeline: detect the document and fields → recognition (classic/CRNN for printed fields) → validation (check digits, date formats, reference lists) → manual review at low confidence. A VLM is for complex layouts, but with mandatory validation.
• “Why is VLM-OCR dangerous for critical data?” — hallucinations: the model confidently “fills in” unreadable text, and the error is invisible without checking against the source.
• “Why is CRNN+CTC better than per-character segmentation?” — it recognizes the whole line without needing to cut it into letters; merged characters stop being a problem.
• “When is Tesseract enough?” — clean scans of printed text, large volumes, cost/privacy requirements (everything runs locally).

VAEs and GANs: Generation Before Diffusion (in Russian)

• “How is a VAE different from an autoencoder?” — the encoder predicts a distribution (μ, σ) rather than a point, plus a KL penalty pulls the latent towards N(0, 1): the space is smooth, you can sample and interpolate. An AE has “holes” between codes.
• “What is mode collapse?” — a GAN generator outputs a small number of similar examples for any z: it found a way to fool the discriminator and lost the diversity of the distribution.
• “Why the reparameterization trick?” — z = μ + σ·ε moves the randomness into ε so that the gradient flows through μ and σ; otherwise backprop doesn't work through the “sample” operation.
• “Where are VAEs and GANs used today?” — the VAE is the latent encoder/decoder in Stable Diffusion; GANs are in upscalers (ESRGAN) and the adversarial loss in diffusion distillation.

Diffusion Models: How Stable Diffusion Draws (in Russian)

• “What does a diffusion network predict?” — the noise ε added at step t (not the image!); the loss is a simple MSE between the true and predicted noise.
• “Why latent diffusion?” — denoising in the VAE latent is tens of times cheaper than in pixels; the VAE decoder returns pixels at the very end.
• “What does classifier-free guidance do?” — it amplifies the difference between the conditional and unconditional predictions: higher w means closer prompt following, but overdoing it “overcooks” the image.
• “Why did diffusion beat GANs?” — a stable MSE loss instead of a minimax game, no mode collapse, higher quality and diversity; the price is many steps, which DDIM and distillation reduce.

Video Models: Generation with Temporal Consistency (in Russian)

• “Why is video generation harder than image generation?” — a time axis is added: you need temporal consistency (objects keep their shape/color/trajectory), and the token count = frames × patches — the cost goes up by orders of magnitude.
• “How does spatio-temporal attention work?” — spatial attention links patches within a frame, temporal attention links the same region of the scene across frames; in Sora-class models the video latent is cut into spatio-temporal “cube” tokens for a DiT.
• “What is a world model?” — a model that predicts how a scene evolves (“what happens if”) — a video model as an environment simulator for planning by agents and robots.
• “Why can't frames be generated independently?” — each frame is an independent sample from the distribution: details (color, position, background) are chosen anew → flickering and teleporting objects.

Hardware and Deployment

GPUs and VRAM: Hardware for Neural Networks (in Russian)

• “How much VRAM does a 13B model need in 4 bits?” — weights ≈ 13 × 0.5 = 6.5 GB, plus the KV-cache and ~18% overhead: in practice you budget 8–10 GB, and more for long context. The bare formula is the weights; the full answer is weights + cache + overhead.
• “Why is inference bound by memory bandwidth?” — for every token all the weights and the whole cache are pushed through the chip; little arithmetic, many bytes, so speed is determined by VRAM bandwidth, not teraflops.
• “Why quantization?” — it cuts both the size of the weights (the model fits on the card) and the per-token traffic (generation speeds up almost proportionally) by 2–4×, at the cost of a small quality loss.
• “The model doesn't fit — what do you do?” — in order: quantization, then offloading layers to RAM (at the cost of speed), then tensor/pipeline parallelism across several GPUs.

Distributed Training: DP, TP, PP and ZeRO (in Russian)

• “A 70B model doesn't fit on an 80 GB A100 — what do you do?” — step by step: compute the memory (70B × 16 bytes ≈ 1.1 TB for training), enable ZeRO-3/FSDP across enough cards, TP within a node, PP across nodes; for inference, quantization + TP is enough.
• “What is a pipeline bubble?” — idle time of pipeline stages at the beginning and end of a step; fixed by increasing the number of micro-batches (GPipe, 1F1B).
• “How is TP different from PP?” — TP cuts every layer across, the cards compute a layer together and communicate at every layer (NVLink is needed); PP gives different layers to different cards and communicates rarely, but pays with the bubble.
• “What does ZeRO shard at each stage?” — 1: optimizer states; 2: + gradients; 3: + the weights themselves (gathered with all-gather for the duration of a layer's computation).

Local Models vs SaaS and APIs (in Russian)

• “When self-hosted and when an API?” — an API for a quick start, small/bursty volume and access to frontier quality; self-hosted for strict privacy requirements, a large steady flow and the need for control/fine-tuning.
• “What makes up the real cost of a local solution?” — not just hardware: plus electricity, depreciation, engineers' salaries, ops and the cost of downtime; you have to compare TCO, not the price of a card.
• “How do you find the break-even point?” — equate the linear cost of the API (price per token × volume) with the nearly fixed cost of your own/rented hardware; the crossover volume is the threshold above which your own is cheaper.
• “How do you combine privacy and frontier quality?” — a hybrid: private/simple requests on a local model, complex ones to the API; or a self-hosted open-weight model if its quality is enough.

Agents and Tools

Tool Calling: How a Model Calls Tools (in Russian)

• “Does the model execute the tools itself?” — no, it generates a structured request; execution always happens in your code. This is the main trick question.
• “How do you improve call accuracy?” — fewer tools, detailed “what and when” descriptions, strict JSON schemas, returning validation errors to the model for a retry.
• “What is MCP and why is it needed?” — a standard for describing and connecting tools: write a server once, use it from any client.
• “How is tool calling different from JSON mode?” — both mechanisms are about generating to a schema; tool calling adds the loop “call → execution → result into the context → continue”.

The Agent Loop and Stopping Criteria (in Russian)

• “How is an agent different from a pipeline?” — in a pipeline the sequence of steps is fixed in code in advance; an agent chooses the next step and how many steps to take based on the situation.
• “How does an agent know it's time to stop?” — list all four criteria and stress that a verifiable success condition is more reliable than the model's self-assessment.
• “Why do agents break on long tasks?” — compounding errors: 95% per step is only ≈36% over 20 steps; fixed with verification and short, verifiable subtasks.
• “How do you debug an agent?” — read whole trajectories, not final answers; look for loops, lost observations, wrong tool choices.

Memory, Planning and Multi-Agent Systems

• “The context is overflowing — what do you do?” — compaction of old history, moving artifacts out to files, selective loading, delegating bulky subtasks to subagents.
• “What is lost in the middle?” — the model uses information from the middle of a long context worse; so “stuff everything into the window” is not a strategy.
• “The main pro and the main con of multi-agent systems?” — pro: a clean context for each subagent and parallelism; con: losses in handoff, more tokens, harder debugging.
• “When is one agent better than several?” — a short linear task or tightly coupled subtasks with shared state.

MCP: The Standard for Connecting Tools (in Russian)

• “Why MCP if there is function calling?” — the main question. Answer: different levels — function calling describes how the model asks to call a tool, MCP describes where tools come from and how they are delivered to the application.
• “What problem does MCP solve quantitatively?” — N applications × M sources = N×M integrations turn into N+M connections to the standard.
• “What are the risks of a third-party MCP server?” — someone else's code with permissions: supply chain, prompt injection via tool results, excessive privileges. Mitigated by least privilege and confirmation of dangerous actions.
• “What does an MCP server expose besides tools?” — resources (data for the context, read by the application) and prompts (ready-made templates).

Evaluating Agents: pass@k and Trajectories (in Russian)

• “How do you evaluate an agent for production?” — automatic task success + k runs: pass^k for reliability, cost (steps/tokens), checking the trajectory for dangerous actions.
• “Why isn't pass@1 enough?” — a single run is stochastic; pass@1 doesn't distinguish a stable agent from a lucky one, and the pass@1 ↔ pass@k gap shows instability.
• “Why a sandbox?” — reproducibility: a fixed initial state and isolation from the outside world, otherwise the metric is flaky and results are not comparable.
• “What's wrong with a 60% average success rate?” — it may be bimodal or hide reward hacking; look at the distribution across tasks and read the trajectories.

RAG: Retrieval + Generation

What RAG Is and Why You Need It (in Russian)

• “RAG or fine-tuning?” — knowledge vs skill: fresh/private facts — RAG; style, format, domain behavior — fine-tuning; often a combination.
• “Draw the RAG pipeline” — two phases: indexing (documents → chunks → embeddings → vector DB) and query (question embedding → top-k → prompt → generation).
• “Why RAG with a million-token window?” — corpus scale, query cost and latency, attention degradation; retrieval selects only what's needed.
• “How does RAG affect hallucinations?” — it reduces them and makes them verifiable through citations, but doesn't eliminate them: garbage in — fantasy out.

Vector Search and ANN Indexes (in Russian)

• “Cosine or Euclidean?” — on normalized vectors the ranking is almost the same; what matters more is which metric the embedding model was trained with and the index was built with.
• “Explain HNSW in simple terms” — a multi-layer graph: “highways” at the top for long jumps, “streets” at the bottom for precise refinement; greedy descent through the layers.
• “What do you tune in IVF if recall is low?” — increase nprobe (slower, more complete) or the number/quality of centroids; check the case of queries near cell boundaries.
• “How does ANN coexist with metadata filters?” — post-filtering may leave fewer than k results; you need an engine that filters inside the index traversal.

Vector Databases and Their Types (in Russian)

• “How is a vector DB different from a relational one?” — a relational DB looks for an exact match by value (equals/greater than) and does JOINs with ACID; a vector DB looks for semantic proximity through an ANN index. Different questions — different data structures.
• “Do you need a separate vector DB or is pgvector enough?” — if Postgres is already in the stack and you have up to ~1M vectors, pgvector is simpler to operate: one database, transactions, backups. A dedicated service — when scale and load require it.
• “Pre-filtering or post-filtering?” — a post-filter can knock out almost all results under a strict condition; a pre-filter is more accurate but more expensive. Ideally — filtering inside the index traversal (filter-aware search).
• “When do you not need a vector DB at all?” — with thousands of documents: numpy/FAISS in memory, no separate service.

Chunking and Data Preparation (in Russian)

• “How do you choose the chunk size?” — it's a trade-off: larger means more context but a blurred vector; smaller means a precise vector but lost meaning. It is chosen experimentally by retriever metrics (see RAG evaluation).
• “Why overlap and what's wrong with it?” — it saves thoughts at the boundaries, but breeds duplicates in the index and the results.
• “Users complain that answers are cut off mid-sentence. Where do you look?” — check the chunk boundaries: most likely the answer is split; fixed with overlap or structural chunking.
• “What do you put in the metadata?” — date, source, version and access rights; without ACLs the system is insecure.

Hybrid Search and Reranking (in Russian)

• “How is a bi-encoder different from a cross-encoder?” — separate encoding (vectors precomputed, fast search, lower accuracy) versus reading the pair jointly (higher accuracy, but O(N) inferences — only for reranking the top).
• “Why RRF and not a sum of scores?” — BM25 scores and cosine live on incomparable scales; RRF works with ranks and needs no calibration.
• “When is pure vector search not enough?” — exact terms: SKUs, error codes, names, abbreviations, article numbers.
• “What price do we pay for a reranker?” — latency and inference cost; that's why it is applied to the top-N, not the whole database.

MMR and Result Diversity (in Russian)

• “What does the λ parameter do in MMR?” — it balances relevance (λ→1, risk of duplicates in the results) and novelty/diversity (λ→0, risk of irrelevance); in practice λ ≈ 0.5–0.7.
• “Why diversity in RAG?” — top-k by pure similarity often fills the context with near-duplicates; MMR gives the LLM different facets of the topic instead of copies of one point.
• “When does MMR hurt?” — on narrow factual questions where top-1 accuracy matters; then λ is kept high or MMR is turned off.
• “How is MMR different from a reranker?” — a reranker refines the relevance of each document separately, MMR optimizes the composition of the set given what has already been selected (marginal relevance).

Embedding Models: How They Are Trained (in Russian)

• “How do you fine-tune an embedder for your domain?” — query–document pairs from your own data → mining hard negatives → contrastive fine-tuning with a small LR → reindexing and measuring recall@k on a golden set. And mention the alternative: a reranker is often enough.
• “Why hard negatives?” — random negatives are too easy, the gradient on them is almost zero; hard negatives teach the model to tell the similar from the relevant.
• “What are Matryoshka embeddings?” — training in which the first k coordinates are a standalone embedding: the vector can be truncated to a budget without retraining.
• “Why does E5 have query:/passage: prefixes?” — role asymmetry: the query and the document are encoded differently, and the model must know which one it is looking at.

Evaluating RAG Quality (in Russian)

• “How do you evaluate the quality of a RAG system?” — separately: the retriever (recall@k, precision@k, MRR/nDCG on a golden set) and the generator (faithfulness, answer relevance via LLM-as-judge).
• “The answers are bad — what do you check first?” — retrieval: most often the needed chunk simply isn't in the context.
• “What happens if you greatly increase k?” — recall goes up, but junk goes into the context: faithfulness and answer accuracy may drop, latency and cost go up.
• “What are the problems with LLM-as-judge?” — preference for long answers, position bias, leniency towards texts of a similar style; fixed by calibration against human labels.

Production and Quality

ML System Design in Interviews (in Russian)

• “Design an anti-fraud system” — start with the cost of FP/FN errors and the latency requirement (real time!), then follow the framework; the metric is precision/recall or PR-AUC, not accuracy.
• “How do you validate a churn model?” — a time-based split: train on January–June, test on July; a random split gives leakage.
• “The model is ready, what next?” — shadow mode, an A/B test on online metrics, drift monitoring, a retraining plan.
• “Why a baseline first?” — a reference point, a check of the data pipeline and an honest answer to the question “do we even need ML here”.

Recommender Systems (in Russian)

• “Design a recommendation feed” — classic ML system design: a two-stage retrieval → ranking, offline/online metrics, a cold-start plan and a fallback to popular items if the model fails.
• “What is two-tower and what does ANN have to do with it?” — separate user and item encoders; item vectors are precomputed and stored in an ANN index — retrieval in milliseconds.
• “How do you deal with cold start?” — content features for new items, popular items and onboarding for new users, bandits for exploration.
• “Offline nDCG went up — do we roll it out?” — no: an offline metric is only a hypothesis; the decision is made by an A/B test on online metrics.

Ranking: Learning to Rank (in Russian)

• “How is LTR different from regression?” — regression minimizes the error of the absolute value for each object; LTR optimizes the relative order within a query, and absolute scores don't matter by themselves.
• “What is NDCG and why normalize?” — DCG sums relevance with a position discount; normalizing by IDCG (the ideal order) puts the metric into [0,1] and makes it comparable across queries with different numbers of relevant documents.
• “Pointwise vs pairwise vs listwise?” — pointwise predicts relevance per document and sorts; pairwise learns the order within a pair (RankNet); listwise optimizes the metric of the whole list (LambdaMART) and usually gives the best NDCG.
• “Why can't you train LTR on raw clicks?” — position bias: top items get clicked because of their position, and the model locks in the old order; you need IPW or click models.

Monitoring and Data Drift (in Russian)

• “How is data drift different from concept drift?” — in the first case the distribution of X changed while the X→y relationship stayed the same, in the second the relationship itself changed; concept drift is not always visible from the inputs.
• “How do you monitor a model if labels arrive six months later?” — proxy signals: PSI/KL on features, the distribution of predictions, the share of missing values; labels are connected to monitoring retrospectively.
• “PSI = 0.25 on a key feature — what do you do?” — first investigate (a pipeline bug? a new segment?), then decide: fix the data, retrain or roll back.
• “How do you monitor an LLM application without a quality metric?” — refusal rate, answer length, user feedback, an LLM judge on a sample of traffic.

A/B Tests and Product Metrics (in Russian)

• “What is a p-value?” — the probability of getting such or a more extreme result if H₀ is true. Not the probability that H₀ is true — you lose points for that wording.
• “The offline metric went up, the online one didn't. Why?” — offline data was collected under the old model, the proxy metric ≠ the business metric, a shift in user behavior.
• “Why can't you stop the test at the first p < 0.05?” — peeking inflates the Type I error; the duration is fixed in advance.
• “The test showed no significance — is there no effect?” — not necessarily: maybe there wasn't enough power. Look at the confidence interval and the sample size calculation.

Safety: Guardrails and Prompt Injection (in Russian)

• “How is prompt injection different from a jailbreak?” — in a jailbreak the user attacks the model; in an injection the malicious instructions are hidden in the data (an email, a page), and the user is the victim.
• “How do you protect an agent with access to email?” — in layers: input/output filters, marking data boundaries, least privilege, confirmation of external actions; and name the “lethal trifecta”.
• “Can prompt injection be solved with a system prompt?” — no: architecturally the model doesn't distinguish instructions from data; the prompt lowers the probability, but only permission limits give a guarantee.
• “What is defense in depth?” — several independent layers, each catching part of the attacks; the idea is that what slips past one layer is caught by the next.