Data Processing Automation

• Automate by trigger: an event or the clock starts the work, not a person
• Use an S3 Event Notification to run code the instant an object lands
• S3 must have permission to invoke your Lambda, or the event silently never arrives
• Never write a function's output back to the prefix that triggered it
• EventBridge Scheduler gives a schedule its own retry policy and dead-letter queue
• Use an EventBridge event-pattern rule to react to something that happened
• Prefer EventBridge Scheduler over the older scheduled EventBridge rule
• Lambda is the event-driven runner a trigger invokes for short processing
• A Lambda invocation caps at 15 minutes, 10 GB memory, and 10 GB of /tmp
• When work exceeds the Lambda limits, start a job from Lambda instead of stretching it
• Drive any AWS service from code with the SDK, boto3 in Python
• SDK start calls are asynchronous; poll for completion
• A boto3 AccessDenied in Lambda is an execution-role fix, not a code fix
• Use Glue DataBrew for visual, no-code data preparation
• A DataBrew recipe is the reusable, versioned list of transformation steps
• SageMaker Unified Studio is the single workspace that unifies data, analytics, and AI
• Use Athena for serverless SQL directly over S3
• Automate Athena with start_query_execution, then poll for the result
• Size an MWAA environment by its class, scale it with workers
• MWAA tasks queuing for a long time means add capacity, not retries
• An MWAA DAG that never appears is usually an import error or wrong S3 path
• Diagnose a Step Functions failure from its execution history
• States.Permissions is a missing IAM permission on the execution role
• A Distributed Map's MaxConcurrency caps fan-out to a downstream service's own limit
• Pick Standard for long or exactly-once workflows and Express for short high-volume ones
• The .sync pattern for Glue needs StartJobRun, GetJobRun, GetJobRuns, and BatchStopJobRun
• Use a Parallel state to run independent branches and wait for all of them
• Use a Choice state to branch a workflow on the data's value
• Use Retry for transient errors and Catch for ones you want to route
• A Glue conditional trigger's Logical AND or ANY decides whether all or one watched job must finish
• A Glue EventBridge event trigger batches arrivals with BatchSize and BatchWindow
• Airflow datasets trigger a consumer DAG by data availability, not the clock

Unlock with Premium — includes all practice exams and the complete study guide.