What does it mean for a model to be 'multimodal,' and how is using one multimodal model different from running a separate image model and then a text model?
technical-conceptual · Junior level · data-ml
What the interviewer is really asking
Probes whether the candidate grasps that 'multimodal' means joint reasoning over modalities in one model, versus chaining two separate single-modality models, and the trade-offs of each.
What to say
- Define multimodal as a single model that takes more than one input type (such as text and images) and reasons over them together in the same context.
- Contrast that with a pipeline that runs a separate vision model first and pipes a text summary into an LLM, which loses detail at the handoff.
- Explain the trade-off: the joint model can ground its answer in both modalities at once, while the two-model pipeline is simpler and more debuggable but can't 'look back' at the image once it's been summarized.
What to avoid
- Don't say multimodal just means 'it can also output images' — input modality is the more common meaning here.
- Don't claim the two-stage pipeline is always worse; it can be simpler and cheaper for narrow tasks.
- Don't confuse modalities with model size or with multilingual support.
Example answers
Strong: Multimodal means one model takes in more than one kind of input — say text and an image — and reasons over them jointly in the same context. The alternative is a pipeline: a vision model captions the image, then an LLM reads only that caption. The joint model can refer back to the actual image while answering, so it catches details the caption dropped; the pipeline is simpler and easier to debug but loses whatever the first model didn't summarize.
Weak: Multimodal just means the model is bigger and can do more things, like generate pictures as well as text. It's basically the same as using two models.