Interview Questions: Production and Quality

24 questions on “Production and Quality” — with short answers and a link to an in-depth explanation of each topic.

How to use it: cover the answer, answer out loud within a minute, then compare. All sections are on the interview questions page. Topics marked “in Russian” are not translated yet.

• ML System Design in Interviews · 4
• Recommender Systems · 4
• Ranking: Learning to Rank · 4
• Monitoring and Data Drift · 4
• A/B Tests and Product Metrics · 4
• Safety: Guardrails and Prompt Injection · 4

ML System Design in Interviews (in Russian)

• “Design an anti-fraud system” — start with the cost of FP/FN errors and the latency requirement (real time!), then follow the framework; the metric is precision/recall or PR-AUC, not accuracy.
• “How do you validate a churn model?” — a time-based split: train on January–June, test on July; a random split gives leakage.
• “The model is ready, what next?” — shadow mode, an A/B test on online metrics, drift monitoring, a retraining plan.
• “Why a baseline first?” — a reference point, a check of the data pipeline and an honest answer to the question “do we even need ML here”.

Recommender Systems (in Russian)

• “Design a recommendation feed” — classic ML system design: a two-stage retrieval → ranking, offline/online metrics, a cold-start plan and a fallback to popular items if the model fails.
• “What is two-tower and what does ANN have to do with it?” — separate user and item encoders; item vectors are precomputed and stored in an ANN index — retrieval in milliseconds.
• “How do you deal with cold start?” — content features for new items, popular items and onboarding for new users, bandits for exploration.
• “Offline nDCG went up — do we roll it out?” — no: an offline metric is only a hypothesis; the decision is made by an A/B test on online metrics.

Ranking: Learning to Rank (in Russian)

• “How is LTR different from regression?” — regression minimizes the error of the absolute value for each object; LTR optimizes the relative order within a query, and absolute scores don't matter by themselves.
• “What is NDCG and why normalize?” — DCG sums relevance with a position discount; normalizing by IDCG (the ideal order) puts the metric into [0,1] and makes it comparable across queries with different numbers of relevant documents.
• “Pointwise vs pairwise vs listwise?” — pointwise predicts relevance per document and sorts; pairwise learns the order within a pair (RankNet); listwise optimizes the metric of the whole list (LambdaMART) and usually gives the best NDCG.
• “Why can't you train LTR on raw clicks?” — position bias: top items get clicked because of their position, and the model locks in the old order; you need IPW or click models.

Monitoring and Data Drift (in Russian)

• “How is data drift different from concept drift?” — in the first case the distribution of X changed while the X→y relationship stayed the same, in the second the relationship itself changed; concept drift is not always visible from the inputs.
• “How do you monitor a model if labels arrive six months later?” — proxy signals: PSI/KL on features, the distribution of predictions, the share of missing values; labels are connected to monitoring retrospectively.
• “PSI = 0.25 on a key feature — what do you do?” — first investigate (a pipeline bug? a new segment?), then decide: fix the data, retrain or roll back.
• “How do you monitor an LLM application without a quality metric?” — refusal rate, answer length, user feedback, an LLM judge on a sample of traffic.

A/B Tests and Product Metrics (in Russian)

• “What is a p-value?” — the probability of getting such or a more extreme result if H₀ is true. Not the probability that H₀ is true — you lose points for that wording.
• “The offline metric went up, the online one didn't. Why?” — offline data was collected under the old model, the proxy metric ≠ the business metric, a shift in user behavior.
• “Why can't you stop the test at the first p < 0.05?” — peeking inflates the Type I error; the duration is fixed in advance.
• “The test showed no significance — is there no effect?” — not necessarily: maybe there wasn't enough power. Look at the confidence interval and the sample size calculation.

Safety: Guardrails and Prompt Injection (in Russian)

• “How is prompt injection different from a jailbreak?” — in a jailbreak the user attacks the model; in an injection the malicious instructions are hidden in the data (an email, a page), and the user is the victim.
• “How do you protect an agent with access to email?” — in layers: input/output filters, marking data boundaries, least privilege, confirmation of external actions; and name the “lethal trifecta”.
• “Can prompt injection be solved with a system prompt?” — no: architecturally the model doesn't distinguish instructions from data; the prompt lowers the probability, but only permission limits give a guarantee.
• “What is defense in depth?” — several independent layers, each catching part of the attacks; the idea is that what slips past one layer is caught by the next.