Reliability & Fidelity

The trust chain: prepare, validate, orchestrate, survive

A nightly pipeline lands a revenue table that quietly double-counts refunds, and nobody notices until the finance team does. That single failure is both halves of this subtopic at once: the pipeline ran (reliability held) but produced wrong numbers (fidelity broke). On the Professional Data Engineer exam, reliability means the pipeline keeps delivering through zone and region failures, restarts cleanly, and is watched; fidelity means the data it produces is cleaned, validated, and never silently corrupted or lost. They are not the same property, and a design can pass one while failing the other.

Every scenario in this subtopic decomposes into the same four jobs, in order. Prepare and clean the raw data so it conforms to the target schema. Validate it against rules before it lands, so a bad batch fails the run instead of reaching consumers. Orchestrate and monitor the multi-step run so dependencies, retries, and alerts are explicit. Survive failure so a stopped or crashed pipeline recovers within the business's downtime and data-loss budget without losing or duplicating records. Keep that order in mind: it is also the order data flows, so a term defined in one job is reused, not redefined, in the next.

The rest of this page walks each job and the Google Cloud service that owns it. Prepare maps to Dataflow^[1], Cloud Data Fusion^[2], and Dataform^[3]; validate maps to Dataform assertions and Dataplex auto data quality^[4]; orchestrate maps to Cloud Composer^[5] with Cloud Monitoring^[6]; and survive maps to consistency choice, the Dataflow stop modes^[7], and disaster-recovery sizing.

The four jobs in data-flow order; reliability is the band underneath that keeps the whole chain running and recoverable.

Preparing and cleaning data

Preparing data means turning raw, inconsistent input into records that match the target schema: parsing, deduplicating, standardizing formats, filling or rejecting nulls, and enriching with lookups. The decision the exam tests is not whether to clean but which service cleans, and the answer turns on the team's skills and where the data already lives.

Dataflow: code-based stream and batch transforms

Dataflow^[1] is the serverless, fully managed runner for Apache Beam^[1] pipelines, processing unified stream and batch with one programming model and autoscaling workers. Reach for it when cleaning logic is complex or real-time, when you need custom code (Java or Python), or when the source is a stream such as Pub/Sub. It is the default processing stage of a streaming pipeline: Pub/Sub ingests, Dataflow transforms, BigQuery stores.

Cloud Data Fusion: visual, code-free ETL

Cloud Data Fusion^[2] is a fully managed, cloud-native data integration service for building ETL (extract, transform, load) pipelines through a drag-and-drop visual interface with no code, built on the open-source CDAP project. It is the fit when the builders are data analysts rather than software engineers, or when you want a library of pre-built connectors and transforms. Data Fusion includes Wrangler, an interactive tool for data preparation that parses, filters, and reshapes data with point-and-click directives. Under the hood it executes pipelines on ephemeral Dataproc (Apache Spark) clusters it provisions and deletes per run, so you pay only for the run.

Dataform: SQL transformation inside BigQuery

Dataform^[3] manages the ELT (extract, load, transform) pattern inside BigQuery: raw data is already loaded, and Dataform runs the transformation step with version-controlled SQL. Developers write SQLX (SQL plus metadata) files; the ref() function builds the dependency graph automatically and Dataform compiles everything into a DAG (directed acyclic graph, the ordered task dependency map this page uses throughout) it runs in BigQuery. Choose Dataform when the data is already in BigQuery and the team is SQL-fluent, so you transform in place rather than moving data out to a separate engine.

The quick rule: complex or streaming code, use Dataflow; visual and code-free, use Data Fusion; in-warehouse SQL on data already in BigQuery, use Dataform. You can also prompt an LLM such as Gemini to draft the SQL for a Dataform or BigQuery transformation, but you still review and test the generated query before it runs in production.

Choosing a preparation service by where the data lives and whether builders write code.

Validating data before it lands

Validation is the gate that makes fidelity enforceable: a rule the pipeline checks automatically so a bad batch fails the run rather than publishing to consumers. The most expensive bug is the one that lands quietly and gets read for a week, so put the check upstream of the consumer, not in a dashboard the consumer happens to notice.

Dataform assertions

An assertion in Dataform is a data-quality test expressed as SQL: it queries for rows that violate a condition, and if any rows come back the assertion fails. The documentation defines them as "data quality tests... to check for uniqueness, null values, or a custom condition." Because assertions are nodes in the same dependency DAG as the transformations, you can require a table's assertions to pass before downstream tables that depend on it run, which stops corruption from propagating.

Dataplex auto data quality

Dataplex auto data quality^[4] defines data-quality rules over a BigQuery table and runs a scan (scheduled or on demand) that validates the data and logs alerts on failure. Rules are row-level (evaluated per row against a passing-threshold percentage, such as range, null, or regex) or aggregate (evaluated across the dataset, such as uniqueness or a statistic). Every rule carries one of seven dimensions: freshness, volume, completeness, validity, consistency, accuracy, and uniqueness. A scan produces a data-quality score (the percentage of rules passed), and the standard pattern is to gate the pipeline by failing it when the score drops below target.

Schema validation

Schema validation is the structural layer beneath value rules: it rejects records whose shape is wrong (missing required fields, wrong types) before any value check runs. BigQuery enforces a table schema on load, and a Dataflow pipeline can route records that fail schema parsing to a dead-letter destination so one malformed row does not crash the whole job. Distinguish the two layers on the exam: schema validation catches structure, assertions and data-quality rules catch content.

The pattern across all three is the same: express the correctness rule as code, run it as part of the pipeline, and fail closed. Choose Dataform assertions when transforming in BigQuery with SQLX, Dataplex when you want a managed, dimension-scored scan with alerting across many tables, and schema validation as the always-on structural floor.

Orchestrating and monitoring pipelines

Once a pipeline has more than one step, the steps have dependencies (load before transform, transform before validate, validate before publish), and someone has to run them in order, retry the ones that fail, and tell you when something breaks. That is orchestration plus monitoring, the operational core of reliability.

Cloud Composer: managed Apache Airflow

Cloud Composer^[5] is a fully managed workflow orchestration service built on Apache Airflow. You author a workflow as a DAG (directed acyclic graph) of tasks in Python; Composer schedules the tasks, respects their dependencies, retries failed tasks per the policy you set, and backfills. A Composer environment is a self-contained Airflow deployment whose components (scheduler, workers, triggerer) run as workloads on Google Kubernetes Engine, but Composer manages that GKE cluster for you. Reach for Composer when a pipeline spans several services (for example: trigger a Dataflow job, then load to BigQuery, then run a Dataform assertion, then refresh a Looker dashboard) and you need the dependencies, retries, and schedule declared in one place.

Do not confuse Composer with Dataflow. Dataflow processes the data within one job; Composer orchestrates the multi-step workflow that may include a Dataflow job as one task. A common distractor offers Dataflow as the orchestrator of a cross-service pipeline; the orchestrator is Composer.

Monitoring the pipeline

Cloud Monitoring^[6] and Cloud Logging^[8] are the observability layer. An alerting policy in Cloud Monitoring bundles a condition (a metric, a threshold, an aggregation) with notification channels (email, PagerDuty, Slack, Pub/Sub) and opens an incident when the condition is met. Dataflow publishes job metrics (system lag, data freshness, element throughput, worker utilization) that you alert on directly. Cloud Logging captures the job and task logs; with a log-based metric you turn matching log entries into a Monitoring time series you can chart and alert on, which is how you catch error patterns that no built-in metric exposes. Alert on user-felt symptoms anchored to an objective (rising lag, falling freshness, climbing error rate), not on every internal CPU blip.

Composer runs the ordered DAG (solid arrows); Monitoring and Logging observe every task (dashed arrows) and alert on failure.

Surviving failure: consistency choice, stop modes, and disaster recovery

Surviving failure is two questions a real scenario forces you to answer: when the data store fails, what consistency must it still guarantee, and when the pipeline stops, how do you restart without losing or duplicating data.

ACID and the consistency-vs-scale choice

ACID stands for atomicity (a transaction is all-or-nothing), consistency (it moves the database between valid states), isolation (concurrent transactions do not corrupt each other), and durability (a committed write survives a crash). These are guarantees a transactional store makes per transaction, and the right store depends on how strong the guarantee must be and at what scale:

Store	Model	ACID transactions	When it fits
Cloud SQL	Managed relational (MySQL/PostgreSQL/SQL Server)	Full ACID, single region	One regional transactional app
Spanner	Relational + horizontal scale	Full ACID with external consistency, global	Global scale and strong consistency together
Bigtable	NoSQL wide-column	Single-row atomic only, no multi-row ACID	High-throughput single-key reads/writes
BigQuery	Serverless analytics warehouse	Not for high-QPS transactions	Analytics and reporting, not transaction processing (OLTP)

Spanner^[9] provides external consistency, the strictest concurrency guarantee: the system behaves as if all transactions ran one at a time in the order they committed, even across multiple regions, and it backs this with TrueTime, Google's globally synchronized clock. That is why Spanner is the answer when a workload needs both global scale and strong consistency, and why it carries up to a 99.999% availability SLA. Choose Cloud SQL when one region's relational workload is enough; do not reach for Spanner's cost when the consistency and scale requirement does not demand it.

Stopping a streaming pipeline: drain vs cancel

When you must stop a running Dataflow streaming job (to deploy a new version, for example), the choice between drain and cancel^[7] decides whether you keep the in-flight data. Drain stops reading new input but lets the service "finish processing the buffered data while simultaneously ceasing the ingestion of new data," so nothing in flight is lost; the trade-off is that draining can take significant time. Cancel "stops the Dataflow service from processing any data, including buffered data," so it is immediate but "you might lose in-flight data." The exam rule is simple: to redeploy without losing records, drain; cancel only when data loss is acceptable and you need an immediate stop.

Disaster recovery, checkpointing, and replay

For recovery from a failure, size the architecture to two targets: RTO (recovery time objective, the maximum tolerable downtime) and RPO (recovery point objective, the maximum tolerable data loss). Tighter RTO/RPO values cost more, so match them to the business need rather than always buying the strictest. Practical levers: store critical data multi-region for durability; take backups frequently enough to meet the RPO (a 15-minute RPO means backing up at least every 15 minutes); checkpoint long-running jobs so a restart resumes from the last consistent point instead of the beginning; and keep events in Pub/Sub so a failed downstream pipeline can replay unacknowledged messages and resume without a gap. A backup that has never been restored is not yet proven recoverable, so test recovery regularly, not just during the real outage.

Drain keeps in-flight data; cancel is immediate but may lose it. Restart via Pub/Sub replay or checkpoint to meet the RPO.

Reliability and fidelity: which tool for which job

Job	Primary GCP tool	What it does	Reach for instead when
Prepare and clean	Dataflow / Data Fusion / Dataform	Transform, standardize, and enrich data (code, visual, or in-warehouse SQL)	Data is already clean and only needs loading
Validate	Dataform assertions / Dataplex auto data quality	Test rows against rules and fail the run on violations	No correctness rule can be expressed yet
Orchestrate	Cloud Composer (managed Airflow)	Run multi-step DAGs with dependencies, retries, schedules	A single step with no dependencies
Monitor	Cloud Monitoring + Cloud Logging	Alert on job health, latency, errors, and SLO burn	An ad hoc one-time run
Choose consistency	Spanner / Cloud SQL / Bigtable / BigQuery	Match ACID strength, scale, and availability to the workload	Requirement is purely offline analytics
Survive failure	Drain, checkpoint, multi-region, backups, Pub/Sub replay	Stop cleanly, recover within RTO/RPO, lose no records	Workload can be re-run from source with no urgency

Decision tree

Cheat sheet

Sharp facts the exam loves — give these one last read before exam day.

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Reliability is the pipeline surviving; fidelity is the data being correct

Treat the two as separate properties a design must satisfy independently. Reliability is operational: the pipeline keeps delivering through zone and region failures, restarts cleanly, and is monitored. Fidelity is data correctness: records are cleaned, validated, and never silently corrupted or lost. A pipeline can run flawlessly and still publish wrong numbers, so a scenario that asks about trustworthy data is usually testing the fidelity half even when everything is up.

Use Dataflow for code-based stream and batch transforms

When cleaning logic is complex, real-time, or needs custom code, use Dataflow, the serverless managed runner for Apache Beam pipelines that handles stream and batch with one model and autoscaling workers. It is the standard processing stage of a streaming pipeline (Pub/Sub ingests, Dataflow transforms, BigQuery stores). Reach for Cloud Data Fusion instead when the builders are analysts who want a visual, code-free tool, or Dataform when the data is already in BigQuery and the team writes SQL.

Trap Reaching for Cloud Composer to do the transformation; Composer orchestrates the steps but does not process the data, Dataflow does.

Use Cloud Data Fusion for visual, code-free ETL

When the people building pipelines are data analysts rather than engineers, use Cloud Data Fusion, a fully managed data integration service with a drag-and-drop interface built on the open-source CDAP project. Its Wrangler tool prepares and cleans data with point-and-click directives, and it runs pipelines on ephemeral Dataproc (Spark) clusters it provisions and deletes per run. Choose Dataflow instead when you need custom code or low-latency streaming logic.

Use Dataform for version-controlled SQL transformation inside BigQuery

When data is already loaded in BigQuery and the team is SQL-fluent, use Dataform to run the transform step of ELT in place. Developers write SQLX (SQL plus metadata) files and the ref() function builds the dependency DAG automatically, so you transform without moving data to a separate engine. This is the ELT pattern (transform after load), distinct from Dataflow's ETL where transformation happens in flight before the warehouse.

3 questions test this

Dataform assertions are SQL data-quality tests that fail the run on bad rows

An assertion in Dataform is a data-quality test written as SQL that queries for rows violating a condition; if any violating rows return, the assertion fails. They check uniqueness, null values, or a custom condition, and because they are nodes in the same dependency DAG as the transforms, you can require a table's assertions to pass before downstream tables run, stopping corruption from propagating. Put the check in the pipeline so a bad batch fails closed rather than reaching consumers.

Trap Validating data only by inspecting the output dashboard afterward; that catches bad data after consumers have already read it, not before it lands.

Dataplex auto data quality scores a table against rules across seven dimensions

Use Dataplex auto data quality when you want a managed, scheduled scan that validates a BigQuery table and alerts on failure. Rules are row-level (per-row, with a passing-threshold percentage: range, null, regex) or aggregate (across the dataset: uniqueness, statistic), and each carries one of seven dimensions: freshness, volume, completeness, validity, consistency, accuracy, uniqueness. A scan produces a data-quality score (percentage of rules passed); the standard pattern is to gate the pipeline by failing it when the score drops below target.

1 question tests this

Your enterprise data platform has BigQuery tables across multiple projects and regions containing customer transaction data. Data quality…

Schema validation catches structure; assertions catch content

Separate the two validation layers. Schema validation rejects records whose shape is wrong (missing required fields, wrong types) and is enforced by BigQuery on load; value rules (assertions, Dataplex rules) check the content of structurally-valid rows. A Dataflow pipeline routes records that fail schema parsing to a dead-letter destination so one malformed row does not crash the whole job. Use both: schema is the structural floor, assertions are the content gate.

Trap Treating a passed schema check as proof the data is correct; schema validation only confirms shape, not that the values are valid or complete.

Route malformed records to a dead-letter sink instead of failing the whole job

When a streaming Dataflow job hits records it cannot parse or that fail validation, send them to a dead-letter destination (a separate Pub/Sub topic, BigQuery table, or Cloud Storage path) rather than letting one bad record crash the pipeline. The good data keeps flowing and the rejected records are captured for later inspection and reprocessing. This preserves both reliability (the job stays up) and fidelity (bad rows never silently enter the clean dataset).

2 questions test this

Cloud Composer orchestrates multi-step pipelines; Dataflow processes data

When a pipeline spans several services with dependencies, retries, and a schedule, orchestrate it with Cloud Composer, the managed Apache Airflow service where you declare the workflow as a DAG of tasks in Python. Composer schedules tasks, respects dependencies, and retries failures; a Composer environment runs Airflow components on a GKE cluster that Composer manages. Do not confuse it with Dataflow, which processes the data inside one job and may itself be a single task in a Composer DAG.

Trap Naming Dataflow as the orchestrator of a cross-service pipeline; Dataflow runs one processing job, Composer sequences the whole multi-step workflow.

Alert on pipeline health with Cloud Monitoring alerting policies

Monitor a pipeline with Cloud Monitoring alerting policies, which bundle a condition (metric, threshold, aggregation) with notification channels and open an incident when met. Dataflow exposes job metrics such as system lag and data freshness that you alert on directly. Anchor alerts to user-felt symptoms (rising lag, falling freshness, climbing error rate), not to every internal CPU spike, so on-call gets signal rather than noise.

2 questions test this

Turn log patterns into alertable metrics with log-based metrics

When the failure you care about appears in logs but no built-in metric exposes it, create a Cloud Logging log-based metric that turns matching log entries into a Cloud Monitoring time series you can chart and alert on. A counter metric counts matching entries; a distribution metric extracts a number from each entry and builds a histogram. This is how you alert on error strings or parsed values that the standard Dataflow or service metrics do not surface.

ACID is the per-transaction guarantee: atomicity, consistency, isolation, durability

ACID describes what a transactional store guarantees for each transaction: atomicity (all-or-nothing), consistency (only valid state transitions), isolation (concurrent transactions do not corrupt each other), and durability (a committed write survives a crash). It is the property you match a database choice against, so a scenario demanding that a multi-row financial update never partially apply is asking for an ACID-compliant transactional store, not an analytics warehouse.

Choose Spanner for ACID with global scale and strong consistency together

When a relational workload needs both horizontal global scale and strong consistency, choose Spanner, which provides ACID transactions with external consistency, the strictest concurrency guarantee, backed by the TrueTime synchronized clock, and carries up to a 99.999% availability SLA. External consistency means the system behaves as if all transactions ran one at a time in commit order, even across regions. Use Cloud SQL instead when a single region's relational workload is enough; Spanner's cost is justified only when global scale and strong consistency are both required.

Trap Reaching for Spanner for a single-region transactional app that Cloud SQL handles; you pay for global scale you do not need.

Cloud SQL fits single-region relational ACID; BigQuery is not for transactions

Match the store to the consistency and access pattern. Cloud SQL (managed MySQL, PostgreSQL, SQL Server) gives full ACID for one regional transactional application. BigQuery is a serverless analytics warehouse for reporting, not a high-QPS transactional system, so it is the wrong answer for transaction processing (OLTP) even though it speaks SQL. Bigtable is NoSQL wide-column with single-row atomic operations only and no multi-row ACID, suited to high-throughput single-key access such as time-series or IoT data.

Trap Using BigQuery as the transactional database because it runs SQL; it is an analytics warehouse, not an OLTP store for frequent small transactions.

Drain to keep in-flight data; cancel to stop a streaming job immediately

When stopping a running Dataflow streaming job, drain stops reading new input but lets the service finish processing buffered, in-flight data before shutting down, so no records are lost; the trade-off is that draining can take significant time. Cancel stops all processing immediately, including buffered data, so it is fast but you may lose in-flight data. To redeploy a new pipeline version without losing records, drain; cancel only when data loss is acceptable and you need an immediate stop.

Trap Cancelling a streaming job to deploy a new version; cancel discards buffered and in-flight data, so use drain to preserve it during a redeploy.

Size disaster recovery to the RTO and RPO the business accepts

RTO (recovery time objective) is the maximum tolerable downtime and RPO (recovery point objective) is the maximum tolerable data loss; both drive the DR design, and tighter values cost more. Set backup frequency to meet the RPO (a 15-minute RPO means backing up at least every 15 minutes), and match readiness posture to the RTO: cold (backup-and-restore, cheapest, slowest), warm (scaled-down standby), or hot (full second deployment, fastest, priciest). Buy the strictest posture only when the business requirement demands it.

Trap Buying an always-on multi-region hot standby for a workload whose RTO/RPO budget a cheaper backup-and-restore posture already meets.

Replay from Pub/Sub and checkpoint long jobs to recover without gaps

Make a stopped pipeline resumable rather than restarting from scratch. Keep events in Pub/Sub so a failed downstream pipeline can replay unacknowledged messages and resume without a gap, and checkpoint long-running jobs so a restart resumes from the last consistent point. These recover within the RPO by not losing the records that were in flight when the failure hit. Pub/Sub retains undelivered messages, which is what makes the replay possible.

3 questions test this

A backup is only proven recoverable once you have restored it

Testing recovery is part of reliability, not an afterthought. The reliability discipline requires regularly restoring backups in a non-production environment to confirm they contain consistent, usable snapshots, because a backup that has never been restored is not yet proven recoverable. Schedule recovery drills (for example, periodic regional-failover and restore-from-backup tests) rather than discovering a broken backup during the real outage.

Trap Assuming a successful backup job means the data is recoverable; only a completed test restore proves the snapshot is consistent and usable.

You can prompt an LLM to draft transformation SQL, but you still test it

Generative AI such as Gemini can draft the SQL for a Dataform or BigQuery transformation from a natural-language description, which speeds authoring. The generated query is a draft, not a verified artifact, so you review it and run it through the same assertion and data-quality gates as hand-written SQL before it touches production. The LLM accelerates writing the transform; it does not replace validating its output.

Back up BigQuery beyond the 7-day time-travel window with scheduled-query table snapshots

Time travel only recovers data for 7 days by default, so for longer retention (14, 30, 365 days) automate CREATE SNAPSHOT TABLE DDL with a scheduled query and an expiration matching the requirement, writing snapshots to a separate dataset so they survive deletion of the source. Snapshots are cheap because BigQuery only charges for data that differs from the base table.

Trap Relying on time travel or copying full tables for long-term retention instead of expiring snapshots, which only bill for the delta from the base table.

6 questions test this

Cross-region BigQuery failover needs managed DR on Enterprise Plus with a failover reservation

For automated failover of both compute and storage across regions with a cross-regional SLA, use BigQuery managed disaster recovery with an Enterprise Plus edition failover reservation and attach the replicated datasets to it. A soft failover needs both regions available and replicates everything (no loss); a hard failover promotes the secondary immediately and can lose unreplicated data. This is separate from BigQuery's native single-region zonal redundancy.

Trap Assuming BigQuery's built-in zonal redundancy already covers a full regional outage; cross-region failover requires the Enterprise Plus managed-DR feature.

3 questions test this

Pass only a Cloud Storage URI through XCom; keep the large payload in Cloud Storage

XCom is for small metadata. Pushing megabyte-scale objects through it bloats the Airflow metadata database, degrades scheduler performance, and breaks environment snapshots and upgrades. Store the actual dataset in Cloud Storage and pass just the GCS path/URI via XCom, letting the downstream task fetch it.

Trap Treating XCom as a general data-transfer channel and serializing the whole intermediate dataset into it instead of a reference.

4 questions test this

Make Composer tasks atomic and idempotent with retries so reruns don't duplicate data

Maintenance and worker restarts terminate and reschedule tasks, so design each task to do one operation that is safe to re-run, and configure retries (default_args / default_task_retries, with retry_exponential_backoff for flaky APIs). Achieve idempotent writes with BigQuery MERGE or WRITE_TRUNCATE keyed on the logical run date, and use Jinja macros like {{ ds }} / {{ execution_date }} rather than datetime.today() so historical reruns target the correct partition.

Trap Using datetime.today() or append-only inserts in a DAG, so a retry stamps the current date and produces duplicate rows on re-execution.

8 questions test this

Keep the dead-letter sink simple and branch off the original record before transforms

A dead-letter path should use the simplest, most reliable sink (often Cloud Storage rather than BigQuery) because a failure writing to the dead-letter sink can fail the whole pipeline. Capture the original input record plus the error at the earliest validation stage; branching later only saves a partially transformed element you can't reprocess. In Dataflow, route multiple outcomes with a ParDo and TupleTag side outputs, and note Pub/Sub dead-lettering only fires for failures in the first fused stage because Dataflow acks the message after that stage.

Trap Putting the dead-letter branch after transformations, so the captured record is partial and can't be matched back to the source for reprocessing.

5 questions test this

Spark Streaming on Dataproc needs a Cloud Storage checkpoint plus an idempotent sink for exactly-once

Store Spark Structured Streaming checkpoints in Cloud Storage, with a unique checkpoint path per query, so progress survives cluster failures. Checkpoints and write-ahead logs alone are not enough for exactly-once delivery: the output sink must be idempotent, e.g. use foreachBatch with the batchId and MERGE/upsert on a business key so reprocessed records don't duplicate. For driver-failure resilience add an HA cluster and StreamingContext.getOrCreate.

Trap Believing checkpointing by itself gives exactly-once output; without an idempotent sink, recovery replays can write duplicates.

7 questions test this

Validate Cloud Storage integrity with CRC32C; composite objects have no MD5

Supply a precomputed checksum on upload and Cloud Storage validates it server-side, rejecting the object with a 400 if it doesn't match so corrupt data is never stored. For parallel composite uploads and compose results, only CRC32C is available (MD5 is not), so validate each component and the final composite with CRC32C; Python clients need google-crc32c or crcmod to compute it efficiently.

Trap Expecting an MD5 hash on a composite object for integrity checks, when composite objects only carry a CRC32C.

7 questions test this

Prevent Cloud Storage race conditions with generation-match and metageneration-match preconditions

When multiple writers update the same object, read its generation (content) and metageneration (metadata) numbers, then include generation-match / metageneration-match preconditions in the write; the request fails with 412 if another process changed it first, so no silent overwrite occurs. Pin the generation number in ranged reads too, so all chunks of a large object come from one consistent version.

Trap Relying on retries or object locking for concurrent writers instead of optimistic generation preconditions that make a stale write fail fast.

4 questions test this

Object Versioning recovers overwritten objects; soft delete recovers a deleted bucket

Enable Object Versioning to keep readable noncurrent versions of overwritten or deleted objects, and use Object Lifecycle Management to expire those noncurrent versions and cap storage cost. Object Versioning does not protect against deleting the bucket itself; soft delete does, letting you restore a deleted bucket and its objects. Enable both for layered protection against accidental and malicious deletion.

Trap Assuming Object Versioning protects against bucket deletion; only soft delete can restore a deleted bucket.

4 questions test this

The trust chain: prepare, validate, orchestrate, survive

Preparing and cleaning data

Dataflow: code-based stream and batch transforms

Cloud Data Fusion: visual, code-free ETL

Dataform: SQL transformation inside BigQuery

Validating data before it lands

Dataform assertions

Dataplex auto data quality

Schema validation

Orchestrating and monitoring pipelines

Cloud Composer: managed Apache Airflow

Monitoring the pipeline

Surviving failure: consistency choice, stop modes, and disaster recovery

ACID and the consistency-vs-scale choice

Stopping a streaming pipeline: drain vs cancel

Disaster recovery, checkpointing, and replay

Reliability and fidelity: which tool for which job

Decision tree

Cheat sheet

Also tested in

References