Previous version: MLA-C01. The current version is MLA-C02.
Domain 3 of 4 · Chapter 2 of 3

Provisioning & Scaling

Endpoint hosting and scaling options for inference

CriterionReal-time + target trackingReal-time + step scalingServerless inferenceScheduled scaling
Best-fit trafficSpiky but continuousSpiky, or scale from zeroIntermittent with idle gapsPredictable by clock
Who sets instance countPolicy holds a metric near targetAlarm thresholds map to instance stepsFully managed, no instancesCapacity set by schedule
Scales to zeroNo (floor = min count)Yes (step from zero)Yes (idle = 0)No
Idle costPay for min instancesPay for min instancesZero when idlePay for scheduled floor
Cold startsNone (always warm)Possible from zeroPossible (use provisioned concurrency)None
VPC / GPU supportYesYesNoYes

Decision tree

Idle gaps with no traffic?long silent stretchesYesNo, near-continuousPredictable bursts?cold start acceptable?Bursty, no cold startIdle OKNeed VPC or GPU?private networking?Serverless +provisioned concurrencyServerless endpointscales to zero when idleNoYesScale out from zero?or known time-of-day peaksReal-time in VPCVpcConfig + auto scalingFrom zero / by clockAlways ≥ 1 instanceStep scaling / scheduled scalingstep required to scale from 0Target tracking onSageMakerVariantInvocationsPerInstance
Cheat sheet

Cheat sheet

  • Choose provisioning by traffic shape, not by data size
  • Attach auto scaling to a real-time endpoint instead of pinning instance count
  • Use target tracking as the default endpoint scaling policy
  • Track SageMakerVariantInvocationsPerInstance for load-proportional scaling
  • Use step scaling when you must scale an endpoint out from zero
  • A real-time endpoint with auto scaling never drops below its minimum count
  • Cooldown periods default to 300 seconds for scale-in and scale-out
  • Add provisioned concurrency to keep a serverless endpoint warm for predictable bursts
  • Serverless endpoints cap memory at 6 GB and max 200 concurrency
  • Use managed spot training to cut training cost up to 90%
  • Spot is for training jobs, never for real-time inference endpoints
  • Use SageMaker hosting for managed serving; ECS or EKS only when you own orchestration
  • Put an endpoint in your VPC with VpcConfig and at least two subnets
  • An endpoint placed in a VPC reaches S3 via a gateway VPC endpoint, other services via interface VPC endpoints
  • Set EnableNetworkIsolation to cut all outbound calls from the container
  • Both CloudFormation and CDK provision infrastructure as code with rollback on error
  • Share values across stacks with CloudFormation exports and Fn::ImportValue
  • Use scheduled scaling for demand that follows the clock
  • Let Karpenter right-size GPU nodes from nvidia.com/gpu pod requests
  • Enable SMDDP distributed training via the estimator's distribution parameter
  • Set compute, VPC, volume encryption, and timeout on a boto3 create_training_job call
  • Reference SageMaker resource names with Fn::GetAtt so CloudFormation orders creation
  • Wire SageMaker endpoint auto scaling in CloudFormation with ScalableTarget + ScalingPolicy

References

  1. https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html
  2. https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html
  3. https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling-policy.html
  4. https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms.html
  5. https://docs.aws.amazon.com/sagemaker/latest/dg/host-vpc.html