Domain 5 of 5 · Chapter 5 of 5

Failure Awareness & Mitigation

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Professional Data Engineer premium course — no separate purchase.

Included in this chapter:

  • Resilient execution: make every retry harmless
  • Placement: pick the failure domain you must survive
  • Replication and failover, store by store
  • Recovering corrupted or missing data
  • Exam-pattern recognition

Failure domain to survive, by managed store

StoreZonal failure (regional config)Regional failure (multi-region / cross-region)Corruption / bad data rewind
Cloud SQLHA: synchronous standby in a second zone, automatic failoverCross-region read replica you promote on regional lossAutomated backups plus point-in-time recovery
BigQueryBuilt into any location (zone-redundant)Multi-region dataset (US or EU) spans regionsTime travel up to 7 days, table snapshots and clones
SpannerRegional instance replicates across 3 zonesMulti-region instance survives a region lossBackups and point-in-time recovery
Memorystore (Redis)Standard tier: replica with automatic failoverCross-region replication on the Cluster tierNone: it is a cache, rehydrate from the source of record
BigtableCluster lives in one zone; add a cluster for zone HAReplication across clusters in different regionsBackups (cluster-level), restore to a new table

Decision tree

Retry duplicates or amessage gets stuck?YesExactly-once +dead-letter topicNoMust survive azone outage?YesRegional config(Cloud SQL HA)NoMust survive aregion outage?YesMulti-region:BigQuery / SpannerNoData already corruptedor deleted?YesTime travel / PITR/ replay sourceNoNo mitigation neededregenerable, no availability needAlways: make retries idempotent first, then enable any rewind tool (time travel, backups, retention)before the incident, and prove a backup by actually restoring it once.
Cheat sheet

Cheat sheet

  • Separate the three failure jobs: keep running, survive an outage, recover bad data
  • Dataflow streaming is exactly-once by default, but only inside the pipeline
  • Make the sink write idempotent so a redelivery is a no-op
  • Route poison messages to a dead-letter topic so they cannot wedge the subscription
  • Retry transient errors with exponential backoff, but not a quota wall
  • A resource survives exactly the failure domain it is replicated across, no larger
  • Cloud SQL HA is a regional instance with automatic ~60s zonal failover
  • Survive a Cloud SQL region loss by promoting a cross-region read replica
  • Spanner picks its resilience from the instance configuration: regional vs multi-region
  • BigQuery multi-region datasets and the location-immutable rule
  • Bigtable gets HA by adding clusters, with eventually-consistent replication
  • Memorystore HA is the Standard tier; the cache is never the recovery target
  • BigQuery time travel undoes bad changes within the last 7 days
  • Use BigQuery table snapshots and clones for restore points beyond 7 days
  • Cloud SQL point-in-time recovery rolls back to just before a bad write
  • Replay the source from Pub/Sub retention or Cloud Storage to rebuild bad output
  • Every rewind tool must be configured before the incident and proven by a restore
  • Route bad Dataflow elements to a dead-letter sink with with_exception_handling
  • Set only the Dataflow regional endpoint for zonal resilience and data residency

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Dataflow exactly-once processing
  2. Pub/Sub: handle message failures (dead-letter topics)
  3. Regional Persistent Disk for high availability
  4. BigQuery dataset locations
  5. Cloud SQL high availability
  6. Cloud SQL replication (read replicas)
  7. Spanner instance configurations (regional and multi-region)
  8. Bigtable replication
  9. Memorystore for Redis high availability
  10. BigQuery time travel
  11. BigQuery table snapshots
  12. BigQuery table clones
  13. Cloud SQL point-in-time recovery
  14. Pub/Sub: replay and purge messages overview