Maintaining & Automating Workloads
This is Day-2: the pipeline already runs, and the question is how you keep it cheap, repeatable, and standing
Most of this domain starts after the build is done. The data is flowing and people depend on it, so the work shifts from "make it run" to "keep it running well": spend less, rerun safely, share capacity fairly, see what is happening, and survive the day a zone disappears. The single mental model that ties the five subtopics together is the operations lifecycle of a live workload, and the classic exam trap is reaching for a build-time answer to a run-time question. When a query is slow you do not re-architect the pipeline, you change the data layout or the slot pool; when a record is delivered twice you do not redesign ingestion, you make the write idempotent. Read each scenario for which operational stage it is in, then answer from that stage.
The domain unfolds in five operational moves, from cost to survival
Walk the subtopics in order and the page reads as one operational checklist. Optimizing Resources comes first because the cheapest fix is usually shaping the work (partition and cluster a BigQuery table, make a Dataproc cluster ephemeral) before you buy more capacity, and it is where you match a pricing model to the usage shape. Automation & Repeatability is next: make every recurring job idempotent so a rerun is safe, then pick the right rung on the scheduling ladder (a BigQuery scheduled query, Dataform, Cloud Composer, Workflows, or Cloud Scheduler) and pin the environment in Terraform. Organizing Workloads is the slot-economics layer, deciding which jobs draw from which reservation and edition, and queuing deferrable work at BATCH so it never crowds out interactive users. Monitoring & Troubleshooting is how you see the running system, choosing the signal that matches the question: a metric for a backlog, a log for a stack trace, job introspection for the bad run. Failure Awareness & Mitigation is the survival layer, operating an already-designed pipeline through transient errors, zonal and regional loss, and corruption.
When two answers both work, prefer the cheapest option that meets the requirement and keeps a rerun safe
Operations questions usually leave two choices that both function, and the instinct the exam rewards is restraint, not redundancy. Buy capacity only after shaping the work, because slots add concurrency, not per-query efficiency. Reach for the lightest scheduler the job actually needs, because standing up a full Airflow environment for one nightly statement is waste. Pay for exactly the failure domain you must survive and no wider, because a multi-region instance for data you can regenerate is money spent on a blast radius you do not have. And before anything is made redundant, make it idempotent, because replication is wasted on a pipeline that corrupts its own output every time it reruns. Match the spend to the requirement, then make the safe-rerun choice.
The five operational moves, and which subtopic owns each
| Operational move | What you decide here | Anchor concept | Drill into |
|---|---|---|---|
| 1. Spend less | Shape the work and match the pricing model before buying more capacity | Cut bytes and storage first; commit for steady load, pay-per-use for spiky | Optimizing Resources |
| 2. Rerun safely | Make recurring jobs idempotent, then schedule and codify the environment | Idempotent unit of work; the scheduler ladder; Terraform for reproducibility | Automation & Repeatability |
| 3. Share capacity fairly | Decide which jobs draw from which reservation and edition, and queue deferrable work | Slots bought in reservations; assignments route jobs; BATCH yields to INTERACTIVE | Organizing Workloads |
| 4. See the running system | Pick the signal that matches the question and act on the in-flight job | Metrics for backlog, logs for cause, job introspection for the bad run | Monitoring & Troubleshooting |
| 5. Survive a fault | Operate through transient errors, zonal or regional loss, and corruption | Idempotent before redundant; place to the failure domain; a tested restore | Failure Awareness & Mitigation |