Interview Questions: Machine Learning Fundamentals

72 questions on “Machine Learning Fundamentals” — with short answers and a link to an in-depth explanation of each topic.

How to use it: cover the answer, answer out loud within a minute, then compare. All sections are on the interview questions page. Topics marked “in Russian” are not translated yet.

• What Is Machine Learning · 4
• Types of Learning: Supervised, Unsupervised and Reinforcement · 4
• Linear Regression · 4
• Gradient Descent · 4
• Logistic Regression and Classification · 4
• Bayes' Theorem and Naive Bayes · 4
• Overfitting and Regularization · 4
• Bias–Variance: Decomposing the Error · 4
• Quality Metrics · 4
• Cross-Validation and Hyperparameter Tuning · 4
• Decision Trees and Ensembles · 4
• SVM and the Kernel Trick · 4
• kNN and Clustering · 4
• The EM Algorithm and Gaussian Mixtures (GMM) · 4
• Dimensionality Reduction and PCA · 4
• Features and Data Leakage · 4
• Time Series and Forecasting · 4
• Reinforcement Learning: Q-learning and PPO · 4

What Is Machine Learning (in Russian)

• “How is ML different from classical programming?” — the rules are not written by hand; they are derived from data by optimizing parameters.
• “Give examples of tasks for each paradigm” — supervised: spam filter, price prediction; unsupervised: customer segmentation, anomaly detection; RL: games, robotics, RLHF.
• “Why have separate validation and test sets?” — hyperparameters are tuned on validation, so it is “worn out” by that tuning; the test set has to stay untouched for the final evaluation.
• “Classification or regression?” — look at the type of output: a discrete class or a continuous number. Trick question: predicting click probability is regression in form, but it solves a classification task.

Types of Learning: Supervised, Unsupervised and Reinforcement (in Russian)

• “How is self-supervised different from unsupervised?” — self-supervised has a target label extracted from the data and an explicit prediction loss; unsupervised has no label, only structure. Tasks like clustering are unsupervised; masking and next-token prediction are self-supervised.
• “Why is an LLM self-supervised?” — the model is trained to predict the next token; the “correct answer” is the text itself, nothing has to be labeled by humans, so the internet becomes free labeled data.
• “What is exploration/exploitation?” — the RL dilemma: explore new actions for a potentially bigger reward, or exploit what has already been found to work.
• “Why transfer learning?” — to reuse learned representations so you can train on small data faster and more accurately than from scratch.

Linear Regression (in Russian)

• “Why MSE and not MAE?” — the square is differentiable and penalizes large errors more; MAE is more robust to outliers.
• “When the normal equation, and when gradient descent?” — the closed form is good with few features; descent scales to large data and streaming.
• “What happens with correlated features?” — the weights are unstable; regularization (L2/ridge) or feature selection helps.
• “How will the model react to an outlier?” — it will shift a lot because of the squared error; check in the interactive, mentally, where a point high above the rest would pull the line.

Gradient Descent (in Russian)

• “Why is the step taken against the gradient?” — the gradient points in the direction of steepest increase of the function, and we are minimizing.
• “What happens with a too large/too small learning rate?” — divergence or oscillation / slow convergence and getting stuck. An ideal answer mentions a learning rate schedule: start larger, decrease towards the end.
• “Why is SGD better than full-batch gradient?” — a step is thousands of times cheaper, and the noise helps explore the surface; convergence per step is worse, but per unit of time it is faster.
• “Why can't we just solve ∇L = 0 analytically?” — neural networks have no closed-form solution; the equations are nonlinear in billions of parameters.

Logistic Regression and Classification (in Russian)

• “Why doesn't linear regression with a threshold work for classification?” — the output is not bounded to [0, 1], MSE wrongly penalizes confident correct answers, and distant points shift the boundary.
• “Why cross-entropy and not MSE?” — with a sigmoid, MSE is non-convex and gives vanishing gradients on confident mistakes; cross-entropy is convex and strongly penalizes confident mistakes.
• “Is the logistic regression boundary linear?” — yes: σ is monotonic, so the condition p = 0.5 is equivalent to w·x + b = 0. Nonlinearity is achieved by adding features (x², products).
• “How are sigmoid and softmax related?” — softmax for two classes reduces to a sigmoid; the sigmoid is a special case.

Bayes' Theorem and Naive Bayes (in Russian)

• “A disease affects 1%, the test is 95% accurate, the test is positive — what is the probability of disease?” — work it out step by step on 10,000 people: 95 true positives versus 495 false positives → 95/590 ≈ 16%. Name the effect: base rate neglect.
• “How is a generative model different from a discriminative one?” — a generative model learns P(x|y)·P(y) and derives the answer via Bayes (NB); a discriminative model learns P(y|x) directly (logistic regression).
• “Why Laplace smoothing?” — without it, a word never seen in a class gives zero probability and zeroes out the whole product.
• “Why compute in logarithms?” — a product of hundreds of small probabilities underflows; a sum of logarithms is numerically stable.

Overfitting and Regularization (in Russian)

• “How do you tell overfitting from underfitting?” — by the pair of train/val errors: a large gap is overfitting, both high is underfitting.
• “How is L1 different from L2?” — L1 zeroes out weights and selects features, L2 shrinks all weights evenly; L2 is smooth and easier to optimize.
• “What do you do if the model overfits?” — in order of increasing cost: regularization/early stopping → simplify the model → collect more data.
• “Why k-fold if you have a train/val split?” — on small data a single split is noisy; averaging over k folds gives a stable estimate and lets you tune hyperparameters fairly.

Bias–Variance: Decomposing the Error (in Russian)

• “What is the bias-variance trade-off?” — expected error = bias² + variance + noise; increasing complexity lowers bias but inflates variance; the optimum is at the bottom of the U-curve.
• “Is overfitting high bias or high variance?” — high variance: the model is perfect on train but jumps around from sample to sample. High bias is underfitting.
• “How do ensembles affect the decomposition?” — bagging/RF average models and reduce variance; boosting builds up accuracy and reduces bias.
• “Can the noise σ² be removed?” — no, it is an irreducible limit caused by randomness in the data itself; error on new data will not go below it.

Quality Metrics (in Russian)

• “Classes 1:100, accuracy 0.99 — is the model good?” — no: the constant “everyone is healthy” gives the same. Look at precision/recall, PR-AUC.
• “What happens to precision and recall when you raise the threshold?” — precision usually goes up, recall goes down: we catch fewer, but with more confidence.
• “How do you explain AUC?” — the probability that a random positive is ranked above a random negative; AUC says nothing about a specific threshold.
• “Which metric would you use for X?” — start with the cost of FP and FN errors in money/risk, then name the metric, not the other way around.

Cross-Validation and Hyperparameter Tuning (in Russian)

• “Why do you need validation if you have a test set?” — the test set is touched once for an honest final estimate; tuning on the test set is leakage and inflates quality. Hyperparameters are chosen on validation/CV.
• “What is nested CV and when is it needed?” — the outer loop fairly evaluates the pipeline, the inner loop tunes hyperparameters; it is needed so that the estimate is not inflated by the very act of picking the best option.
• “Grid search or random search?” — with low dimensionality and discrete values, grid; in a high-dimensional space where only a couple of parameters matter, random search (Bergstra) or Bayesian optimization is more efficient.
• “How do you cross-validate a time series?” — only forward-chaining/walk-forward; random folds leak information from the future.

Decision Trees and Ensembles (in Russian)

• “How is Random Forest different from bagging?” — it adds a random subset of features at each node: trees are more diverse and less correlated.
• “RF or boosting — which when?” — RF: a fast, robust baseline with minimal tuning. Boosting: maximum quality on tabular data, but tune the learning rate, depth and early stopping.
• “Why can't boosting train trees in parallel?” — each tree depends on the errors of the previous ones; the parallelism is inside a single tree.
• “Why does boosting beat neural networks on tables?” — heterogeneous features, categories, missing values, small and medium samples are where trees shine; networks need more data and preprocessing.

SVM and the Kernel Trick (in Russian)

• “What are support vectors?” — points on the edge of the street (or inside it with a soft margin); only they define the boundary, removing the others does not change it.
• “Explain the kernel trick in words” — the data is implicitly mapped into a high-dimensional space where it becomes linearly separable, and the kernel cheaply computes dot products there without building the mapping itself.
• “What do C and gamma do?” — C penalizes margin violations (large C → narrow street, overfitting), gamma sets the radius of a point's influence in RBF (large gamma → islands around points, overfitting).
• “Why isn't SVM used on large data?” — the n×n kernel matrix and O(n²)–O(n³) training.

kNN and Clustering (in Russian)

• “Why is kNN called lazy?” — there is no training phase: the model stores the sample and all the work happens at query time; hence the expensive O(n) prediction.
• “What happens to kNN without feature normalization?” — the distance is dominated by the feature with the largest scale, and the others stop mattering.
• “Does k-means guarantee a global minimum?” — no, only a local one; this is addressed with k-means++ and several restarts.
• “Half-moon shaped clusters — what do you use?” — not k-means (it is for convex spheres), but DBSCAN or spectral clustering.

The EM Algorithm and Gaussian Mixtures (GMM) (in Russian)

• “Why is GMM better than k-means?” — soft assignment (probabilities instead of hard labels) and elliptical clusters of different shape, tilt and size thanks to the full covariance; k-means is its degenerate special case.
• “What do the E- and M-steps do?” — E, with parameters fixed, computes responsibilities (soft memberships); M, with responsibilities fixed, updates the centers, covariances and weights; each cycle does not decrease the likelihood.
• “Why does EM get stuck in a local optimum?” — the likelihood is non-convex; EM increases it monotonically, but only up to the nearest local maximum, so the result depends on the start — k-means++ and several runs help.
• “How do you choose the number of components?” — not by likelihood (it grows with k), but by BIC/AIC with a complexity penalty.

Dimensionality Reduction and PCA (in Russian)

• “What does PCA maximize?” — the variance of the projections; equivalently, it minimizes the mean squared error of linear reconstruction.
• “What are principal components mathematically?” — eigenvectors of the covariance matrix; the eigenvalues are the variances along them.
• “Why standardize before PCA?” — without it, the components will be dominated by features with large units of measurement.
• “Can you trust distances between clusters on t-SNE?” — no: the method preserves local neighborhoods, the global geometry is distorted.

Features and Data Leakage (in Russian)

• “AUC 0.99 offline, 0.6 in production — what happened?” — a classic: leakage. Check features filled in after the event, the split scheme and preprocessing done before the split.
• “How do you do target encoding without leakage?” — out-of-fold: the encoding for a row is computed on folds without it; plus smoothing for rare categories.
• “Why is a random split of time data a lie?” — the model sees the future: the offline metric is inflated, and that information will not exist in production.
• “How do you look for leaks in practice?” — audit features by when they become available, look at suspiciously important features, compare the metric under a time-based and a random split.

Time Series and Forecasting (in Russian)

• “How do you validate a sales forecasting model?” — only a time-based split, ideally walk-forward: several consecutive “train on the past — test on the future” windows.
• “Why is a naive baseline mandatory?” — it sets the bar almost for free; a complex model that doesn't beat seasonal naive adds no value.
• “Which features must not be used?” — anything that uses data after the prediction moment: centered windows, aggregates over “the whole period”, statistics computed over the whole series before the split.
• “What would you use to forecast 10,000 product series?” — boosting on lags and calendar features: one model for all series, and external features are trivial to add.

Reinforcement Learning: Q-learning and PPO (in Russian)

• “What are Q-learning and the Bellman equation?” — Q(s,a) is the expected return of an action; the Bellman equation Q(s,a)=r+γ·max Q(s′,·) links it to the next step; Q-learning fits Q to this target with the update Q ← Q+α·[r+γ·max Q(s′,·)−Q], off-policy (it learns from the max, not from the move actually taken).
• “Exploration vs exploitation?” — explore something new for a potentially bigger reward, or exploit the best found so far; the typical compromise is ε-greedy: with probability ε a random move, otherwise argmax.
• “How is RLHF related to RL?” — it is PPO against a reward model with a KL penalty: the LLM policy generates answers, the reward model scores them, PPO increases the reward, and KL keeps the model close to the reference.
• “How are policy methods different from value methods?” — value methods (Q-learning) learn the value of actions and take the argmax; policy methods (REINFORCE, PPO) parameterize and move the policy itself. For LLMs with a huge action space, it is policy methods that work.