A container keeps getting killed with an OOMKilled status even though the host has plenty of free memory. Explain what's actually happening at the Linux level and how you'd reason about it.
technical-conceptual · Senior level · cloud-devops-security
What the interviewer is really asking
Probes whether the candidate understands the kernel primitives behind containers — cgroups enforce resource limits, namespaces provide isolation — and specifically that a cgroup memory limit (memory.max) triggers a cgroup-scoped OOM kill independent of host free memory, plus the memory.high soft-limit throttling behavior, rather than confusing host and container memory.
What to say
- Separate the two kernel primitives: namespaces control what a process can see, cgroups control what it can use — and the memory limit is a cgroup property, so the kill is scoped to the container's cgroup, not the host.
- Explain the OOM mechanics: in cgroups v2, memory.max is a hard ceiling and when the container's working set exceeds it the kernel OOM-kills a process inside that cgroup regardless of how much memory the host still has free — that's why a roomy host and an OOMKilled container coexist.
- Bring in the soft limit and the fix path: memory.high throttles and reclaims before the hard kill, so I'd look at actual usage versus the limit, right-size the request/limit to the real working set, and check for a leak or an unbounded cache rather than just bumping the limit blindly.
What to avoid
- Concluding the host is out of memory because a container was OOM-killed — the cgroup limit, not host pressure, is what triggered it here.
- Reflexively raising the memory limit without checking whether it's a genuine working-set need, a leak, or an unbounded buffer, which just delays the same kill.
- Conflating namespaces and cgroups — namespaces isolate the view, cgroups enforce the limit; mixing them up signals a shallow model of how containers actually work.
Example answers
Strong: OOMKilled with a free host means it's a cgroup limit, not host memory. Containers are cgroups plus namespaces — namespaces decide what the process sees, cgroups decide what it can use — and the memory limit lives in the cgroup. In cgroups v2, memory.max is a hard ceiling, so when the container's working set crosses it the kernel OOM-kills inside that cgroup no matter how much the host has free. So I'd compare real usage to the limit, set memory.high to get throttling before the hard kill, and figure out if it's a true working-set need or a leak before I just raise memory.max.
Weak: If the container got OOM-killed it ran out of memory, so I'd increase its memory limit so it has more headroom. If it keeps happening I'd move it to a bigger node with more RAM so there's enough memory available for it to run.