Domain 3 of 5 · Chapter 3 of 4

Data Lake

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Professional Data Engineer premium course — no separate purchase.

Included in this chapter:

  • The lake on Google Cloud: store, govern, query
  • Managing the lake: storage classes and aging policies
  • Governing the lake with Dataplex
  • Querying lake data: external tables vs BigLake
  • Processing, monitoring, and reading the stem

Ways to make Cloud Storage lake data queryable and governed

CapabilityExternal tableBigLake tableDataplex discovery
What it isBigQuery table over GCS filesExternal table with access delegationAuto scan that registers tables for you
User needs GCS permission?Yes, direct bucket readNo, only table permissionTables it creates can be BigLake (no bucket read)
Row/column security + maskingNoYesInherited from the BigLake tables it creates
SchemaYou define itYou define itInferred automatically from the files
Best forQuick ad-hoc reads of trusted dataGoverned, fine-grained access to lake dataCataloging and exposing a sprawling bucket

Decision tree

What is the job on the lake?Cloud Storage bucket = the lakeCut costGovern / discoverQuery in placeAccess schedule known?storage class by read frequencyDataplexdiscover, profile, data qualityBucket access / masking?read GCS from BigQueryYes: by ageLifecycle rulesSetStorageClass / DeleteNo: variesAutoclassno retrieval / early-del feeYes: governedBigLake tableaccess delegation, row/col securityNo: trustedExternal tableuser needs bucket readMigrate existing Spark/Hadoop over the lake = Dataproc (ephemeral over GCS); Beam batch = Dataflow.Dataplex watches data content (quality); Cloud Monitoring + Logging watch infrastructure health.All engines read one Cloud Storage copy in place: keep one cheap copy, many engines over it.
Cheat sheet

Cheat sheet

  • On Google Cloud the data lake is a Cloud Storage bucket, not a service
  • Query Cloud Storage from BigQuery in place with an external table, no load needed
  • A plain external table makes every querying user hold direct bucket read
  • Use a BigLake table for access delegation and fine-grained security on lake data
  • A BigLake table is still external by storage; it adds controls, not a copy
  • BigLake metadata caching speeds queries over many files
  • Dataplex is the lake management plane across Cloud Storage and BigQuery
  • A Dataplex discovery scan turns bucket files into queryable BigQuery tables
  • Dataplex auto data quality validates content; pick the right quality dimensions
  • Separate data-content health from infrastructure health when monitoring a lake
  • Set the Cloud Storage class by read frequency, not by data importance
  • Nearline/Coldline/Archive charge for the minimum duration even if you delete early
  • Object Lifecycle Management ages data by rule when the schedule is known
  • Autoclass moves objects by access pattern with no transition or retrieval fees
  • A bucket cannot use both Autoclass and a SetStorageClass lifecycle rule
  • Process lake data in place; Dataproc keeps Spark/Hadoop clusters ephemeral over GCS
  • Many engines read one Cloud Storage copy in place, no per-engine copy
  • Parquet for columnar analytics reads, Avro for write-heavy ingest and schema evolution
  • Drive lifecycle off a business date with Custom-Time, not creation time
  • Target lifecycle rules by prefix or suffix, and Delete wins ties
  • Uniform bucket-level access disables ACLs; fine-grained keeps them for S3 parity
  • Grant prefix-scoped access with managed folders under uniform access

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Data lake vs. data warehouse
  2. Cloud Storage storage classes
  3. Object Lifecycle Management
  4. Cloud Storage Autoclass
  5. Dataplex (Knowledge Catalog) introduction
  6. Automatic discovery of Cloud Storage data
  7. Dataplex auto data quality overview
  8. Query Cloud Storage data in external tables
  9. Introduction to BigLake tables
  10. Dataproc overview
  11. Cloud Monitoring overview