Domain 3 of 4 · Chapter 2 of 3

Provision and configure ML and AI workload resources

Unlock the complete study guide.

Bundled into the existing AWS Certified Machine Learning Engineer - Associate premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Provisioning resources based on existing architecture
  • Deploying a model programmatically
  • On-demand versus provisioned capacity
  • Choosing the scaling signal
  • Waking an endpoint from zero
  • Sharing GPU instances with inference components
  • Sharing resources between CloudFormation stacks
  • Running SageMaker AI jobs from AWS Step Functions
  • Building and maintaining ML container images
  • SageMaker AI endpoints in a VPC
  • Knowledge base ingestion: chunking strategies
  • Knowledge base vector index configuration
  • AgentCore Runtime compute types
  • Managing agent state with AgentCore
  • Provisioning an Amazon Bedrock flow
  • Matching requirements and symptoms to resource fixes

Scalable target, per-unit load signal and zero-capacity behavior by SageMaker AI endpoint type

Decision pointReal-time endpoint (production variant)Real-time endpoint with inference componentsAsynchronous endpointServerless endpoint
Scalable target (what is counted)Instance count of the variantCopy count of each inference componentInstance count of the variantProvisioned concurrency
Per-unit load signalInvocationsPerInstance; ConcurrentRequestsPerModel for token streamingInvocationsPerCopy or ConcurrentRequestsPerCopyApproximateBacklogSizePerInstanceSageMakerVariantProvisionedConcurrencyUtilization
Can capacity reach zero?No: scaling to zero requires inference componentsYes, with managed instance scaling MinInstanceCount set to 0 and minimum copy capacity 0 for every hosted inference componentYes, with MinCapacity 0 on the scalable targetYes, on-demand capacity scales to zero automatically
Request that arrives at zero capacityNot applicableFails until an instance is provisioned; a NoCapacityInvocationFailures alarm starts step scalingWaits in the queue; a HasBacklogWithoutCapacity alarm starts step scalingServed after a cold start; provisioned concurrency avoids it

Decision tree

Can the caller wait for a queued result?Idle gaps, cold starts tolerable, no GPU?Several models share GPU instances?Token streaming, varied response lengths?Asynchronous endpointApproximateBacklogSizePerInstanceServerless Inferenceon demand; provisioned concurrencycost-effective for predictable burstsInference componentscopy scaling, InvocationsPerCopy(ConcurrentRequestsPerCopy if streaming)Real-time variantConcurrentRequestsPerModelReal-time variantInvocationsPerInstanceYesYesYesYesNoNoNoNo
Cheat sheet

Cheat sheet

  • Serverless provisioned concurrency keeps configured serving capacity warm
  • Per-instance invocation metrics support load-based endpoint scaling
  • Queue backlog per instance is a useful asynchronous inference scaling signal
  • A backlog-without-capacity policy helps asynchronous endpoints wake from zero
  • Concurrent-request metrics reflect the occupancy of streaming model requests
  • Inference-component scaling adjusts model copies on shared accelerator infrastructure
  • Real-time scale-to-zero requires inference components and tolerating startup failures
  • CloudFormation exports let independently managed stacks share resource identifiers
  • Imported CloudFormation exports constrain updates to the exporting stack
  • Nested stacks group reusable infrastructure under a parent lifecycle
  • Step Functions job integration waits for supported SageMaker AI jobs to finish
  • State machine role permissions and execution role permissions are separate
  • Multi-stage builds exclude build-only tools from the runtime image
  • Pinning a base image digest makes the selected image content reproducible
  • Refreshing a base image and disabling the build cache solve different update problems
  • ECR repository tag immutability rejects overwriting existing release tags
  • Enhanced ECR scanning covers language packages as well as operating-system packages
  • EKS image pulling uses the infrastructure's image-pull role
  • Programmatic endpoint deployment separates model, configuration, and serving resource
  • A model VPC configuration connects its inference container to private resources
  • A Runtime interface endpoint keeps client inference calls on the AWS network
  • Private DNS lets standard SageMaker AI hostnames resolve to interface endpoints
  • Network isolation blocks container calls even to private VPC resources
  • The SageMaker Python SDK packages and deploys models through a higher-level interface
  • No chunking treats each source document as a single retrieval chunk
  • Semantic chunking groups text by meaning rather than only by fixed size
  • Chunk overlap preserves context across adjacent retrieval units
  • A knowledge-base vector index must match the embedding representation
  • Knowledge-base field mappings connect vectors, text, and source metadata
  • OpenSearch knowledge-base indexes need a supported vector engine
  • S3 metadata sidecars must correspond to their source documents
  • Binary embeddings require a vector store that supports binary vectors
  • Short-term memory preserves turn-by-turn context within a session
  • Long-term memory retains extracted knowledge across agent sessions
  • AgentCore microVM sessions isolate execution state between users
  • AgentCore Runtime compute types support different execution requirements
  • Bedrock Flows connects configured nodes through explicit data relationships

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html
  2. https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain3.html
  3. https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-deploy-models.html
  4. https://docs.aws.amazon.com/step-functions/latest/dg/connect-sagemaker.html
  5. https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html
  6. https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-modelbuilder-creation.html
  7. https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html
  8. https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-autoscale.html
  9. https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference-autoscale.html
  10. https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling-zero-instances.html
  11. https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling-add-code-define.html
  12. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-stack-exports.html
  13. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-nested-stacks.html
  14. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-anatomy.html
  15. https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-sagemaker-model.html
  16. https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/intrinsic-function-reference-importvalue.html
  17. https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html
  18. https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html
  19. https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html
  20. https://docs.aws.amazon.com/eks/latest/best-practices/image-security.html
  21. https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html
  22. https://docs.aws.amazon.com/lambda/latest/dg/python-image.html
  23. https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-tag-mutability.html
  24. https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_ImageIdentifier.html
  25. https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-pull-ecr-image.html
  26. https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecr_assets.DockerImageAsset.html
  27. https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_on_EKS.html
  28. https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html
  29. https://docs.aws.amazon.com/sagemaker/latest/dg/interface-vpc-endpoint.html
  30. https://docs.aws.amazon.com/sagemaker/latest/dg/host-vpc.html
  31. https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain4.html
  32. https://docs.aws.amazon.com/sagemaker/latest/dg/mkt-algo-model-internet-free.html
  33. https://docs.aws.amazon.com/bedrock/latest/userguide/kb-chunking.html
  34. https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-setup.html
  35. https://docs.aws.amazon.com/bedrock/latest/userguide/kb-data-source-sync-ingest.html
  36. https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agents-tools-runtime.html
  37. https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-instances-how-it-works.html
  38. https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory-types.html
  39. https://docs.aws.amazon.com/bedrock/latest/userguide/flows.html