What is the Kubernetes operator pattern, and how does a custom controller's reconcile loop actually work?

technical-conceptual · Mid level · cloud-devops-security

What the interviewer is really asking

Assess whether the candidate understands an operator as a CRD plus a controller that runs a level-triggered reconcile loop, and can describe the watch/cache/workqueue mechanics and idempotency requirement rather than reciting 'it automates ops'.

What to say

• Define it structurally: an operator is a Custom Resource Definition (CRD) that adds a new API kind to the cluster, plus a custom controller that watches that kind and drives the real world toward the spec the user declared. It codifies the operational knowledge a human SRE would otherwise apply by hand (provision, back up, fail over) as software.
• Describe the reconcile loop concretely: the controller watches the API server, an informer caches the objects and enqueues the namespace/name key of anything that changed onto a workqueue, the loop pops a key and calls Reconcile(), which reads the current spec, compares desired vs observed state, and issues the create/update/delete calls to close the gap. It is level-triggered, not event-triggered — it reconciles toward the current desired state, not 'the thing that just happened'.
• Call out the two properties that make it correct: Reconcile must be idempotent (running it ten times with the same spec yields the same result as running it once), and on a transient error you return and requeue with exponential backoff rather than retrying in a tight loop. Built with controller-runtime via Kubebuilder or the Operator SDK.

What to avoid

• Describing an operator as just 'a pod that runs scripts' or a Helm chart, with no mention of a CRD and a reconcile loop — that misses what makes it an operator.
• Saying the controller reacts to events imperatively ('on create, do X; on delete, do Y') — that's event-triggered thinking and leads to resources getting stuck; the loop is level-triggered and reconciles toward desired state.
• Forgetting idempotency, so the same reconcile run twice double-creates resources, or retrying a failed reconcile in a tight loop instead of requeuing with backoff.

Example answers

Strong: We had a stateful database that needed careful manual steps for failover and backups, so I wrote an operator. The CRD let teams declare a 'PostgresCluster' with replica count and backup schedule, and the controller's reconcile loop compared that spec to the live StatefulSet, PVCs, and CronJobs and created or adjusted whatever was missing. Because it was level-triggered and idempotent, a controller restart just re-reconciled to the same desired state instead of redoing actions.

Weak: An operator is basically a Helm chart that installs an app with some custom settings.