Previous version: MLA-C01. The current version is
MLA-C02.
SageMaker AMT tuning strategies and when each fits
| Strategy | How it searches | Best for | Constraint |
|---|
| Bayesian optimization | Learns from completed jobs to propose better next configurations | Continuous ranges where each job is expensive; the efficient default | Less parallelizable because each round informs the next |
| Hyperband | Multi-fidelity: stops weak runs early and reallocates budget to strong ones | Long deep-learning jobs with a meaningful intermediate metric | Needs an objective that improves during training |
| Random search | Samples configurations independently at random | A fair baseline; fully parallel across many jobs | No learning between jobs, so it can waste budget |
| Grid search | Exhaustively tries every combination | Small, fully categorical spaces you must cover completely | Categorical parameters only; combinations explode fast |
Decision tree
Cheat sheet Cheat sheet
- An epoch is one full pass over the data; a step processes one batch
- Hyperparameters are set before training; parameters are learned during it
- Use AMT Bayesian optimization when each training job is expensive
- Use AMT Hyperband for long jobs with a useful intermediate metric
- AMT grid search is categorical-only and exhaustive
- Early stopping ends training when the validation metric stops improving
- Pick distributed training by what does not fit: model vs data
- Do not distribute a model that fits comfortably on one GPU
- Managed spot training cuts training cost up to 90%
- Pair managed spot with checkpointing so a reclaim resumes, not restarts
- Warm pools keep clusters alive to skip cold start on the next job
- Model size is driven by architecture, not data volume
- Regularization fixes overfitting (high variance), never underfitting
- L1 zeros weights (feature selection); L2 shrinks them smoothly
- Dropout randomly disables neurons each step to stop co-adaptation
- Use script mode to run your own TensorFlow or PyTorch code
- Fine-tune a pre-trained model with JumpStart or Bedrock instead of training from scratch
- Avoid catastrophic forgetting when fine-tuning on a narrow dataset
- Ensembling: bagging is parallel, boosting is sequential, stacking learns a combiner
- Shrink a model with pruning and quantization for a cheaper endpoint
- Register every model version in the Model Registry for repeatable, auditable promotion
- AMT warm start seeds a tuning job from up to 5 prior tuning jobs
- In script mode, save the model to SM_MODEL_DIR and read inputs from SM_CHANNEL_
- Add extra Python packages with a requirements.txt in source_dir, not a custom image
- AMT early stopping needs TrainingJobEarlyStoppingType=AUTO and per-epoch objective metrics
- Use AMT completion criteria to stop the whole tuning job once it stops improving
- Use Logarithmic scaling for orders-of-magnitude ranges and ReverseLogarithmic near 1.0
- Enable SMDDP for optimized multi-node AllReduce/AllGather on AWS
- Scale the learning rate up (with warmup) as the global batch size grows
- Combine sharding and tensor/context parallelism (SMP v2) to fit and scale huge models
- Match the fine-tuning variant to your data: labeled pairs vs unlabeled domain text
- Track runs with load_run()/log_metric() and pull results with ExperimentAnalytics