Domain 1 of 5 · Chapter 3 of 4

Flexibility & Portability

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Professional Data Engineer premium course — no separate purchase.

Included in this chapter:

  • From requirements to a flexible architecture
  • Data portability: open formats, BigLake, Omni
  • Application portability across clouds
  • Cataloging, profiling, discovery as governance
  • Residency and exam-pattern recognition

Portability mechanisms by what they keep portable

MechanismWhat it keeps portableHow it worksChoose when
Open formats + BigLakeData (one copy, many engines)Parquet/Iceberg files on Cloud Storage with a governed table over themYou want to avoid warehouse lock-in and share one dataset across engines
BigQuery OmniAnalytics across cloudsRuns the BigQuery engine in AWS/Azure regions, querying S3/Blob in placeData must stay in another cloud but you want BigQuery SQL on it
DataprocProcessing code (Spark/Hadoop)Managed open-source clusters running stock Spark, Hadoop, Hive, PrestoLifting existing OSS jobs with minimal redevelopment
GKE / GKE EnterpriseApplications (containers)Same Kubernetes across Google Cloud, on-prem, AWS, Azure under one control planeHybrid or multi-cloud containerized pipelines need consistent operations
Resource-location policyCompliance (where data sits)Pins dataset/bucket region and blocks out-of-region creation via org policyA regulation fixes data residency

Decision tree

Data in another cloud,cannot copy it?BigQuery OmniYesMoving existingSpark/Hadoop or containers?NoDataprocSpark / Hadoop jobsOSS data jobsGKE Enterprisemulti-cloud containerscontainersNeed one copy readby many engines?NoOpen formats + BigLakeParquet / Iceberg on Cloud StorageYesRegulated datalocation?NoPin location + org policyconstraints/gcp.resourceLocationsYesDataplexcatalog / profile / discoverNo: find/trust data
Cheat sheet

Cheat sheet

  • Separate fixed constraints from growth axes before picking a product
  • Portability is a cost you pay only when a requirement demands it
  • Open formats make one copy of data readable by many engines
  • BigLake puts governed table access over open-format files
  • BigQuery Omni analyses S3 and Blob data without copying it
  • Omni cross-cloud transfer is metered, so query in place
  • Dataproc runs stock open-source so jobs lift and shift
  • Dataproc vs Dataflow: portable OSS code vs serverless Beam
  • Dataproc separates storage from compute via Cloud Storage
  • GKE Enterprise runs the same Kubernetes across clouds
  • Anthos is now GKE Enterprise; the same product, renamed
  • Residency is enforced placement, set at creation and by org policy
  • Residency controls location; sovereignty controls access
  • Catalog and discovery keep a spread-out estate usable
  • Dataplex absorbed Data Catalog with no migration needed
  • Profiling and quality scans gate pipelines on a data quality score
  • Use Dataplex lineage to scope migration and schema changes
  • Speed up BigLake queries over many files with metadata caching and a staleness bound
  • Join Cloud SQL with BigQuery in place using EXTERNAL_QUERY over a BigQuery connection
  • Avro for write-heavy streaming ingestion, Parquet for read-heavy column analytics
  • Dataplex curated zones require columnar Hive-partitioned Parquet, Avro, or ORC
  • Keep Beam pipelines portable with core transforms and runner choice in PipelineOptions
  • Reuse a Java Beam connector from a Python pipeline via Dataflow Runner v2 multi-language pipelines
  • GKE Autopilot for hands-off managed nodes; GKE Standard for fine-grained control
  • Use Infrastructure Manager with Terraform and versioned Cloud Storage state

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. https://cloud.google.com/architecture/framework
  2. https://cloud.google.com/bigquery/docs/biglake-intro
  3. https://cloud.google.com/bigquery/docs/omni-introduction
  4. https://cloud.google.com/dataproc/docs/concepts/overview
  5. https://cloud.google.com/kubernetes-engine/multi-cloud/docs
  6. https://cloud.google.com/dataplex/docs/introduction
  7. https://cloud.google.com/dataplex/docs/auto-data-quality-overview
  8. https://cloud.google.com/architecture/framework/security/data-residency-sovereignty