Domain 1 of 4 · Chapter 4 of 4

Programming Concepts

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing AWS Certified Data Engineer - Associate premium course — no separate purchase.

Included in this chapter:

  • Lambda concurrency: reserved vs provisioned
  • Ephemeral storage and mounting EFS
  • IaC: CloudFormation, CDK, and SAM
  • CI/CD, distributed computing, and exam patterns

IaC tools for a data pipeline

DimensionCloudFormationAWS CDKAWS SAM
Authoring formDeclarative YAML/JSON templateCode (Python/TypeScript/Java) synthesized to a templateShort YAML with the AWS::Serverless-2016-10-31 transform
Relationship to CloudFormationIs the base engineCompiles down to CloudFormationA CloudFormation superset/extension, same engine
Best fitGeneral infrastructure, full controlTeams wanting loops, conditions, and unit tests in a real languageServerless pipelines: Lambda, Step Functions, DynamoDB
Deploy commandaws cloudformation deploy / change setscdk deploy (synth then deploy)sam build then sam deploy
Local serverless testingNone built inVia SAM/local toolingsam local invoke / sam local start-api

Decision tree

Fits 15 min, 10 GB mem,10 GB /tmp on Lambda?NoMove to Glue or EMRdistributed Spark batchYesFan-out exhausts adownstream database?YesReserved concurrencycap parallel connectionsNoCold-start latency hurts asynchronous transform?YesProvisioned concurrencypre-warm; billed when idleSnapStart if Java 11/17NoUnreserved poolscratch > 10 GB or shared?then mount EFS, else /tmpAlways: define it as IaC (CloudFormation / CDK / SAM)
Cheat sheet

Cheat sheet

  • Lambda gives 1,000 concurrent executions per Region by default, shared by every function
  • Reserved concurrency caps a function and carves its slice out of the shared pool
  • Provisioned concurrency pre-warms environments to kill cold starts, and bills while idle
  • Lambda always holds back 100 unreserved units; you can reserve up to 900 by default
  • Use Lambda SnapStart for a free cold-start cut on Java 11/17 functions
  • Lambda scales by at most 1,000 environments every 10 seconds per function
  • Lambda /tmp is configurable from 512 MB to 10 GB and is private to one environment
  • Mount EFS to Lambda when scratch exceeds 10 GB or must be shared
  • A Lambda transform must fit 15 minutes, 10 GB memory, 10 GB /tmp, and a 6 MB payload
  • Pick the IaC tool by who authors it, since CDK and SAM both compile to CloudFormation
  • A SAM template needs Transform: AWS::Serverless-2016-10-31 to expand its shorthand
  • Use AWS SAM to package and deploy a serverless data pipeline from one template
  • Preview a stack update with a change set, not drift detection
  • CodePipeline orchestrates, CodeBuild builds and tests, CodeDeploy rolls out
  • Use CloudWatch Logs and metrics to observe a data pipeline
  • In distributed computing the shuffle, not data size alone, drives how a job scales
  • Match the language to the data-engineering task

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Understanding Lambda function scaling (concurrency)
  2. Configuring reserved concurrency for a Lambda function
  3. Configuring provisioned concurrency for a Lambda function
  4. Lambda scaling behavior (concurrency scaling rate)
  5. Configuring file system access for Lambda functions (Amazon EFS)
  6. Lambda quotas
  7. AWS::Serverless transform (AWS SAM)
  8. Updating stacks using change sets (CloudFormation)