You've fine-tuned per-tenant variants of an open-weight LLM and now have to serve dozens of them in production cost-effectively, and re-evaluate each variant whenever the base model is upgraded. How would you design the serving and the lifecycle around these fine-tuned models?

technical-conceptual · Senior level · data-ml

What the interviewer is really asking

Assesses LLMOps maturity around fine-tuned models in production — multi-adapter (LoRA) serving on shared GPUs versus a deployment per model, adapter versioning and rollback, and the re-evaluation discipline needed when the base model changes — rather than naively standing up one GPU deployment per fine-tuned variant.

What to say

• Reject one GPU deployment per variant: with LoRA adapters you keep a single base model resident and swap small per-tenant adapters in per request, so dozens of variants share one GPU; servers like vLLM and SGLang load adapters on demand with near-zero switch overhead, which collapses the cost from N deployments to roughly one.
• Treat adapters as versioned artifacts with their own lifecycle: each adapter has a version pinned to the base-model version it was trained against, lives in a registry/object store, loads (and evicts) at runtime, and can be rolled back independently of the base — so a bad variant is a config change, not a redeploy.
• Make base-model upgrades a gated, re-evaluated event: a new base invalidates every adapter's compatibility assumptions, so I'd re-run each variant's offline eval suite (and a canary) against the new base before promoting, and keep the old base servable until every tenant has passed — never silently re-point traffic at a new base.

What to avoid

• Standing up a separate full-model deployment (and GPU) per fine-tuned variant — it's the obvious approach and it's wildly expensive and unscalable past a handful of variants.
• Upgrading the base model under everyone at once without re-evaluating each adapter against it, assuming a fine-tune stays valid across a base change.
• Having no per-adapter versioning or rollback, so when one tenant's variant regresses the only lever is redeploying the whole service.

Example answers

Strong: I wouldn't run a GPU deployment per variant — I'd serve them as LoRA adapters over one resident base model, swapping the small per-tenant adapter in per request. vLLM and SGLang load adapters on demand with negligible switch overhead, so dozens of tenants share one GPU instead of paying for N deployments. Each adapter is a versioned artifact in a registry, pinned to the base version it was trained on, loaded and evicted at runtime, and rollable back independently — so a bad variant is a config flip, not a redeploy. For a base-model upgrade I'd treat it as a gated event: re-run every variant's offline eval and a canary against the new base, keep the old base serving until each tenant passes, and only then cut over.

Weak: I'd containerize each fine-tuned model and give each one its own GPU deployment behind the API so they're isolated. When we upgrade the base model I'd just rebuild the images on the new base and roll them out.