Automation & Repeatability — PDE Study Guide

Bundled into the existing Professional Data Engineer premium course — no separate purchase.

Included in this chapter:

Recurring work shape	Tool	Schedule mechanism	Coordinates downstream steps?
One SQL statement on a cadence	BigQuery scheduled query	Cron-like, minimum 5 minutes between runs	No, single statement only
SQL models, dev-to-prod promotion	Dataform workflow configuration	Unix-cron, runs a release config's actions	Within the SQL graph, not across services
Many cross-service tasks, retries, backfill	Cloud Composer (Airflow) DAG	schedule_interval plus catchup/backfill	Yes, full task dependency graph
Short chain of services or APIs, branching	Workflows	Triggered, often by Cloud Scheduler	Yes, ordered steps with switch and retry
Fire one target on a timetable	Cloud Scheduler	Unix-cron cron job	No, it only triggers a single target

Cheat sheet

Make every recurring task idempotent so a rerun is safe
Overwrite the run's partition instead of appending to stay idempotent
Key the processing window off the logical date, not the wall clock
Pin a DAG's start_date to a fixed value
An operator describes the work; a task is an instance of it
Use a sensor to wait on data arriving, not a guessed time offset
Deploy a Composer DAG by dropping the .py into the /dags bucket folder
Catchup and backfill rerun missed intervals, but only safely if tasks are idempotent
Use a BigQuery scheduled query for a single recurring SQL statement
Schedule a SQL transformation graph with a Dataform workflow configuration
A Scheduler target may fire more than once, so make it idempotent
Define the data environment in Terraform so a second copy is one apply away
Keep Terraform state in a remote Cloud Storage backend, not on a laptop
Use Infrastructure Manager to let Google run your Terraform
Match the recurring scheduler to the shape of the work, lightest first
Order Terraform resources by referencing an output attribute, not depends_on
Manage BigQuery dataset IAM with google_bigquery_dataset_access to keep authorized views
Migrate Deployment Manager to Terraform with DM Convert, then run it in Infrastructure Manager
Schedule recurring batch Dataflow with a Dataflow data pipeline on Cloud Scheduler
Enable a Composer triggerer so deferrable operators free worker slots while waiting

Unlock with Premium — includes all practice exams and the complete study guide.

Cheat sheet