Interview Questions: Language Models: From N-grams to the Transformer

24 questions on “Language Models: From N-grams to the Transformer” — with short answers and a link to an in-depth explanation of each topic.

How to use it: cover the answer, answer out loud within a minute, then compare. All sections are on the interview questions page. Topics marked “in Russian” are not translated yet.

• Bigrams and N-grams · 4
• Tokenization and BPE · 4
• Embeddings and word2vec · 4
• The Attention Mechanism · 4
• The Transformer Architecture · 4
• Modern Architectures: RoPE, MoE, GQA · 4

Bigrams and N-grams (in Russian)

• “What is a language model?” — a probability distribution over the next token given the context; everything else is details of the parameterization.
• “Why can't you just keep increasing n?” — the number of contexts grows as Vⁿ⁻¹, almost all long contexts are unique: the table is sparse and the counts are unreliable.
• “What is perplexity?” — the exponent of the average cross-entropy; intuitively, the average number of options the model is hesitating between. It is comparable only with the same tokenizer and test set.
• “How is GPT different from an n-gram model?” — the task is the same, but the distribution is parameterized by a neural network that generalizes to unseen contexts instead of exact string matching.

Tokenization and BPE (in Russian)

• “Why subwords and not words or characters?” — the vocabulary is finite, OOV disappears, sequences have moderate length: a compromise between the two extremes.
• “How does BPE tokenize a word that wasn't in the corpus?” — it splits it into characters and applies the learned merges in order; in the worst case the word stays a set of characters/bytes.
• “Why does an LLM make mistakes counting the letters in a word?” — to the model the word is one or two tokens; individual letters are not visible without explicit splitting.
• “What does the vocabulary size affect?” — sequence length versus the size of the embedding matrix and the quality of rare tokens.

Embeddings and word2vec (in Russian)

• “What's wrong with one-hot?” — all words are equidistant, the geometry carries no meaning, and the dimension equals the vocabulary size.
• “Why are ‘cat’ and ‘dog’ close if they rarely appear next to each other in text?” — what matters is not co-occurrence but similarity of contexts: they share neighbors.
• “Skip-gram vs CBOW?” — skip-gram predicts the neighbors from the word (better for rare words), CBOW predicts the word from the neighbors (faster).
• “Why are contextual embeddings better than static ones?” — ambiguous words get different vectors depending on the sentence.

The Attention Mechanism (in Russian)

• “Why divide by √d?” — without it the scores grow with the dimension, softmax saturates and the gradients are almost zero. Check it in the interactive: a small divisor ⇒ the weights collapse onto one token.
• “What are Q, K, V in your own words?” — a soft dictionary: query, key, value; the answer is a weighted sum of values by how similar the query is to the keys.
• “Why does the decoder need a mask?” — during training the “next token” is already in the data; without a mask the model cheats and doesn't learn to predict.
• “What is the complexity of self-attention?” — O(n²·d) in time and O(n²) in memory for the weight matrix; hence FlashAttention, GQA, sliding windows.

The Transformer Architecture (in Russian)

• “Draw a transformer block” — attention and an MLP, each wrapped as LN → sublayer → add to the input (pre-norm). Be sure to say that the residual is addition, not concatenation.
• “How is BERT different from GPT?” — the mask and the task: the encoder without a mask learns to reconstruct masked tokens, the decoder with a mask learns to predict the next one.
• “Why did the transformer replace RNNs?” — parallel training over the whole length + direct access to any token instead of a fading memory.
• “Where are the parameters?” — about ⅔ of a block in the MLP, ⅓ in attention; plus the embeddings, noticeable in small models and almost negligible in large ones.

Modern Architectures: RoPE, MoE, GQA (in Russian)

• “Why RoPE and not learned embeddings?” — relative positions without parameters, better length extrapolation; rotating Q/K makes the dot product depend only on the offset.
• “What does GQA reduce and what does that affect?” — the KV-cache (and memory traffic during decoding), i.e. inference speed and cost; quality barely suffers.
• “MoE — what's the trade-off?” — lots of knowledge with few FLOPs per token, but the whole model has to live in memory, plus router balancing and complex sharding.
• “How do models handle 128k context?” — RoPE scaling, GQA against cache bloat, FlashAttention against quadratic memory, sometimes a sliding window.