Designing Data Processing Systems

A scenario says "finance data must stay in the EU, the nightly load must survive a zone outage, the warehouse will triple next year, and it all lives on-premises today." None of that is about which transform to write; it is about the shape the system must take before processing starts. This first domain is the design-time domain: it decides the four things a data platform has to get right up front, because changing them later means a rebuild. Each one maps to exactly one of this domain's four subtopics, so the exam-winning habit is to name which requirement a question is really testing, then jump to the subtopic that owns it. The classic trap is reading a design question as an implementation question and reaching for a pipeline tool when the answer is a placement, a key-ownership, or a service-selection decision.

Read this page as a map, then follow the subtopics in order. Security & Compliance decides who may touch the data and where it may live: IAM roles bound at the right level in the resource hierarchy, encryption key ownership (Google-managed keys, CMEK, or CSEK), de-identification with Sensitive Data Protection, and residency pinned with Organization Policy and VPC Service Controls. Reliability & Fidelity is the two-part promise that the pipeline keeps running and the numbers it lands are correct: validation gates such as Dataplex auto data quality and Dataform assertions, matching a store's ACID and availability profile (Cloud SQL, Spanner, BigQuery, Bigtable) to the workload, and stopping streams safely with drain versus cancel. Flexibility & Portability designs for change without a rewrite: decoupling storage from compute, open formats (Parquet, Iceberg) read in place through BigLake and BigQuery Omni, and portable engines on Dataproc or GKE. Data Migrations covers the one-time move onto Google Cloud: choosing the managed service that fits the source and the downtime budget (DMS, Datastream, BigQuery Data Transfer Service, Storage Transfer Service, Transfer Appliance) and validating the cutover.

Across all four dimensions the Data Engineer exam rewards the same instinct: pick the option that meets the stated requirement with the least standing risk and the least custom plumbing. That means binding IAM at the narrowest level that works rather than Editor on the project, leaning on encryption that is already on by default and only adding CMEK when a regulator asks, gating data quality inside the pipeline instead of catching it on a dashboard later, and naming the purpose-built managed migration service rather than scripting a transfer by hand. Let the requirement, not a favorite tool or raw dataset size, drive every choice; the design that absorbs the constraint cleanly is almost always the exam-correct one.