Pipeline Orchestration — DEA-C01 Study Guide

Bundled into the existing AWS Certified Data Engineer - Associate premium course — no separate purchase.

Included in this chapter:

Decision axis	AWS Step Functions	Amazon MWAA	AWS Glue workflows	Amazon EventBridge
What it is	Serverless state-machine workflow	Managed Apache Airflow (Python DAGs)	Glue-native crawler + job chaining	Event bus + cron/rate scheduler
Best fit	Serverless workflows with branching and error handling	Teams on Airflow / rich operator ecosystem	Pipelines built only of Glue crawlers and jobs	Event-driven or scheduled triggering of a pipeline
Authoring model	Amazon States Language (JSON), visual designer	Python DAG files in an S3 bucket	Triggers and conditions in the Glue console/API	Rules with event patterns or schedule expressions
Servers to manage	None (serverless)	Managed Airflow environment (sized workers)	None (serverless)	None (serverless)
Error handling	Per-state Retry / Catch, Map, Parallel built in	Airflow retries, dependencies, sensors, backfill	Conditional triggers; job retries within Glue	Retry policy + dead-letter queue per target
Pricing model	Per state transition (Standard) or per request + duration (Express)	Per environment-hour plus worker compute	No extra charge beyond the Glue resources run	Per million events / scheduler invocations

Cheat sheet

The orchestrator coordinates steps; it never moves the data itself
Reach for AWS Step Functions as the default serverless orchestrator
Step Functions Standard is exactly-once; only synchronous Express is at-most-once
Choose Express Step Functions for high-volume, short, event-driven workflows
In Step Functions, Retry handles transient failures and Catch handles permanent ones
Use a Step Functions Map state to process every item in a collection in parallel
Use the .sync service integration so the orchestrator waits for a long job to finish
Use Amazon MWAA when the team already runs Airflow DAGs
Use AWS Glue workflows when the pipeline is only Glue crawlers and jobs
A Glue conditional trigger expresses a dependency edge between Glue steps
Use EventBridge to trigger a pipeline; it starts the workflow but is not the workflow engine
Use EventBridge Scheduler for cron and rate schedules at scale
SNS is pub/sub fan-out; SQS is a durable pull queue
Fan out with SNS to multiple SQS queues so each consumer gets its own durable backlog
Set the SQS visibility timeout to at least six times the consumer's processing time
SQS retains messages 4 days by default, up to 14 days, and quarantines failures in a DLQ
Make pipeline steps idempotent so a retry or duplicate delivery never double-writes
Use Lambda as small glue logic, not as a multi-step workflow engine

Unlock with Premium — includes all practice exams and the complete study guide.

Cheat sheet