Interview Questions: Multimodal Models

28 questions on “Multimodal Models” — with short answers and a link to an in-depth explanation of each topic.

How to use it: cover the answer, answer out loud within a minute, then compare. All sections are on the interview questions page. Topics marked “in Russian” are not translated yet.

• How LLMs See Images · 4
• Audio: Speech Recognition and Synthesis · 4
• Omni Models: All Modalities in One · 4
• OCR: From Classic Methods to Recognition with LLMs · 4
• VAEs and GANs: Generation Before Diffusion · 4
• Diffusion Models: How Stable Diffusion Draws · 4
• Video Models: Generation with Temporal Consistency · 4

How LLMs See Images (in Russian)

• “How does an image get into an LLM?” — it is cut into patches → each patch is linearly projected into a vector (ViT) → a projector aligns the vectors with the LLM's space → visual tokens go into the sequence together with text tokens.
• “What does CLIP pretraining give?” — contrastive training on image–text pairs puts images and captions into a shared space: the encoder “knows the meaning” of an image, and zero-shot classification works.
• “Why is an image expensive in tokens?” — the token count is N = (H/P)×(W/P): hundreds per image, thousands with tiling; that is both API cost and consumed context.
• “Why can't a VLM read small text?” — details smaller than a patch are lost during encoding; fixed by raising the resolution/tiling or pairing with OCR.

Audio: Speech Recognition and Synthesis (in Russian)

• “How does sound get into a neural network?” — waveform → windows → spectra → mel spectrogram (a time×frequency image); the modern alternative is discrete audio tokens via a neural codec.
• “What does the CTC loss do?” — it learns the alignment of frames and letters without boundary labels: a prediction for every frame, a blank token, collapsing repeats (“hhe-l-lll-o” → “hello”).
• “What is WER?” — (substitutions + deletions + insertions) / number of words in the reference; the main ASR metric, it can exceed 100%.
• “Why is Whisper so good?” — the scale of the data (weak supervision on hundreds of thousands of hours), not a clever architecture.

Omni Models: All Modalities in One (in Russian)

• “How is an omni model different from an LLM with adapters and from an ASR→LLM→TTS pipeline?” — pipeline: three models, latencies add up, only text passes between them. Adapters: an encoder + a projector to a frozen LLM, multimodal input, text output. Omni: one model, pretrained on interleaved tokens, both input and output are multimodal.
• “Why is GPT-4o's voice faster and livelier than the old voice mode?” — the old one was a pipeline of three models; omni hears audio tokens (intonation, pauses) and replies with audio tokens directly — hundreds of ms of latency instead of seconds.
• “What are early and late fusion?” — early: modalities are mixed into one token stream at the input of a single model; late: separate models per modality, combined at the end.
• “What is the main difficulty in training omni models?” — little interleaved data, expensive compute and balancing the mix so that quality on plain text doesn't drop.

OCR: From Classic Methods to Recognition with LLMs (in Russian)

• “Design passport/receipt recognition” — a pipeline: detect the document and fields → recognition (classic/CRNN for printed fields) → validation (check digits, date formats, reference lists) → manual review at low confidence. A VLM is for complex layouts, but with mandatory validation.
• “Why is VLM-OCR dangerous for critical data?” — hallucinations: the model confidently “fills in” unreadable text, and the error is invisible without checking against the source.
• “Why is CRNN+CTC better than per-character segmentation?” — it recognizes the whole line without needing to cut it into letters; merged characters stop being a problem.
• “When is Tesseract enough?” — clean scans of printed text, large volumes, cost/privacy requirements (everything runs locally).

VAEs and GANs: Generation Before Diffusion (in Russian)

• “How is a VAE different from an autoencoder?” — the encoder predicts a distribution (μ, σ) rather than a point, plus a KL penalty pulls the latent towards N(0, 1): the space is smooth, you can sample and interpolate. An AE has “holes” between codes.
• “What is mode collapse?” — a GAN generator outputs a small number of similar examples for any z: it found a way to fool the discriminator and lost the diversity of the distribution.
• “Why the reparameterization trick?” — z = μ + σ·ε moves the randomness into ε so that the gradient flows through μ and σ; otherwise backprop doesn't work through the “sample” operation.
• “Where are VAEs and GANs used today?” — the VAE is the latent encoder/decoder in Stable Diffusion; GANs are in upscalers (ESRGAN) and the adversarial loss in diffusion distillation.

Diffusion Models: How Stable Diffusion Draws (in Russian)

• “What does a diffusion network predict?” — the noise ε added at step t (not the image!); the loss is a simple MSE between the true and predicted noise.
• “Why latent diffusion?” — denoising in the VAE latent is tens of times cheaper than in pixels; the VAE decoder returns pixels at the very end.
• “What does classifier-free guidance do?” — it amplifies the difference between the conditional and unconditional predictions: higher w means closer prompt following, but overdoing it “overcooks” the image.
• “Why did diffusion beat GANs?” — a stable MSE loss instead of a minimax game, no mode collapse, higher quality and diversity; the price is many steps, which DDIM and distillation reduce.

Video Models: Generation with Temporal Consistency (in Russian)

• “Why is video generation harder than image generation?” — a time axis is added: you need temporal consistency (objects keep their shape/color/trajectory), and the token count = frames × patches — the cost goes up by orders of magnitude.
• “How does spatio-temporal attention work?” — spatial attention links patches within a frame, temporal attention links the same region of the scene across frames; in Sora-class models the video latent is cut into spatio-temporal “cube” tokens for a DiT.
• “What is a world model?” — a model that predicts how a scene evolves (“what happens if”) — a video model as an environment simulator for planning by agents and robots.
• “Why can't frames be generated independently?” — each frame is an independent sample from the distribution: details (color, position, background) are chosen anew → flickering and teleporting objects.