Domain 4 of 4 · Chapter 2 of 3

Optimize ML and AI Costs and Performance

Unlock the complete study guide.

Bundled into the existing AWS Certified Machine Learning Engineer - Associate premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Cost levers, telemetry and billing data
  • Selecting an inference instance family
  • Sizing Serverless Inference: memory and feature limits
  • Locating latency: model, overhead and network time
  • Dashboards for performance metrics
  • Alarms that notify and act
  • Matching serving capacity to demand
  • Billing data: Cost Explorer, tags and Budgets
  • Purchasing options for ML compute
  • Estimating FM inference cost from the token mix
  • Batch inference for delay-tolerant prompts
  • Intelligent prompt routing
  • Prompt caching: when reuse pays
  • Attributing FM usage to applications and callers
  • RAG costs: knowledge base sync and vector storage
  • Agent resource consumption with AgentCore
  • Requirements, fixes and rejected alternatives

Purchasing options for ML and FM compute

Decision factorSageMaker AI Savings PlansCompute Savings PlansEC2 Instance Savings PlansManaged Spot TrainingBedrock Provisioned Throughput
Applies toSageMaker AI instance usage, such as notebooks, training and inferenceEC2 instance usage, plus Fargate and LambdaEC2 instance usage in one instance family and one RegionSageMaker AI training jobs run on Spot capacityModel units for one supported Bedrock model
What you give upA commitment whose terms can't change after purchaseA commitment whose terms can't change after purchaseA commitment whose terms can't change after purchaseScheduling certainty: jobs can wait for capacity and be interruptedPer-token billing: model units bill hourly, and a 1-month or 6-month term blocks deletion before it ends
Discount survivesChanges of instance family, size, Region and componentChanges of instance family, size, Region, OS and tenancy; moves to Fargate or LambdaChanges of size, OS and tenancy inside the familyNot applicable: Spot applies per training jobNot applicable: the hourly price covers the purchased model units
Main cost riskUnused commitment (low utilization)Unused commitment (low utilization)Unused commitment, or a migration outside the family or RegionJob stops at the maximum wait time; no on-demand fallbackBilling continues until you delete it
Best fitA predictable SageMaker AI baselineEC2-based ML workloads that may change family or RegionA stable EC2 family in one RegionInterruption-tolerant training with checkpointsA Bedrock workload whose steady throughput need justifies reserved capacity

Decision tree

Bedrock FM with a steady throughput need?YesProvisioned ThroughputBedrock, billed hourlyNoInterruption-tolerant SageMaker AI training job?YesManaged Spot Trainingcheckpoints in S3NoPredictable SageMaker AI instance baseline?YesSageMaker AI Savings Plansany family, size, RegionNoSteady EC2 usage that may change family or Region?YesCompute Savings PlansEC2, Fargate, LambdaNoSteady EC2 usage in one family and Region?YesEC2 Instance Savings Plansone family, one RegionNoOn-demand rates
Cheat sheet

Cheat sheet

  • Inference Recommender selects configurations using workload measurements
  • Inferentia-backed instances are an inference-specific accelerator option
  • GPU memory utilization and GPU compute utilization reveal different constraints
  • Serverless inference memory settings also affect available compute
  • Serverless cost savings apply only when its feature set meets the workload requirements
  • Serving latency must be separated into model latency, overhead latency and network time
  • CloudWatch dashboards provide a shared operational view across resources
  • An operational dashboard needs alarms for automatic notifications
  • Missing-data treatment must match how a metric is emitted
  • Composite alarms combine alarm states to reduce alert noise
  • Distributed traces identify which downstream dependency adds latency
  • Target tracking adjusts serving capacity toward a chosen workload target
  • Scheduled scaling prepares capacity before a predictable traffic increase
  • Cooldown periods balance scaling responsiveness against oscillation
  • Asynchronous inference can queue work while capacity is zero
  • Serverless provisioned concurrency trades idle savings for predictable startup latency
  • Cost Explorer investigates historical spending and usage trends
  • Resource tags must be activated for cost allocation reporting
  • Budget alerts can warn before forecast spending reaches the limit
  • A budget notification is not a hard real-time spending cap
  • Commitment utilization and coverage measure different cost risks
  • Application inference profiles separate foundation-model usage by application
  • SageMaker AI Savings Plans apply to eligible managed ML instance usage
  • Compute Savings Plans retain discounts across more EC2 configuration changes
  • Managed Spot Training exchanges scheduling certainty for lower training cost
  • Managed Spot Training does not automatically switch to on-demand capacity
  • Bedrock Provisioned Throughput creates an hourly capacity commitment
  • Foundation-model cost comparisons must include both input and output usage
  • Output-token limits control generation length but can truncate required content
  • Prompt caching benefits repeated stable context rather than arbitrary repeated requests
  • Prompt-cache savings depend on actual reuse and cache pricing
  • Prompt-cache minimums and expiry determine whether reuse is possible
  • Batch inference fits independent prompts whose results can arrive asynchronously
  • Intelligent prompt routing can select among supported models using predicted response quality
  • Incremental knowledge-base sync avoids reprocessing unchanged documents
  • Vector storage optimization must preserve embedding compatibility
  • OpenSearch Serverless vector-collection generation affects idle resource behavior
  • New-session counts and active-session gauges measure different agent usage
  • Agent resource telemetry supports cost investigation but is not the final bill
  • Invocation logs can identify callers responsible for token-heavy usage

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html
  2. https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain4.html
  3. https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain3.html
  4. https://aws.amazon.com/ec2/instance-types/
  5. https://aws.amazon.com/machine-learning/inferentia/
  6. https://aws.amazon.com/ai/machine-learning/trainium/
  7. https://docs.aws.amazon.com/sagemaker/latest/dg/inference-recommender.html
  8. https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html
  9. https://docs.aws.amazon.com/sagemaker/latest/dg/inference-gpu-drivers.html
  10. https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference-create-endpoint.html
  11. https://docs.aws.amazon.com/sagemaker/latest/dg/host-vpc.html
  12. https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html
  13. https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-runtime-metrics.html
  14. https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-telemetry.html
  15. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html
  16. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html
  17. https://docs.aws.amazon.com/autoscaling/application/userguide/target-tracking-scaling-policy-overview.html
  18. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/alarms-and-missing-data.html
  19. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/alarm-combining.html
  20. https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-scheduled-scaling.html
  21. https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference-autoscale.html
  22. https://docs.aws.amazon.com/cost-management/latest/userguide/ce-what-is.html
  23. https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html
  24. https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html
  25. https://docs.aws.amazon.com/savingsplans/latest/userguide/plan-types.html
  26. https://docs.aws.amazon.com/ec2/latest/instancetypes/instance-type-names.html
  27. https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html
  28. https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html
  29. https://aws.amazon.com/bedrock/pricing/
  30. https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html
  31. https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html
  32. https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html
  33. https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
  34. https://docs.aws.amazon.com/sagemaker/latest/dg/model-cards.html
  35. https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles.html
  36. https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html
  37. https://docs.aws.amazon.com/bedrock/latest/userguide/kb-data-source-sync-ingest.html
  38. https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html
  39. https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-vector-search.html