Training & Refining Models

• An epoch is one full pass over the data; a step processes one batch
• Hyperparameters are set before training; parameters are learned during it
• Use AMT Bayesian optimization when each training job is expensive
• Use AMT Hyperband for long jobs with a useful intermediate metric
• AMT grid search is categorical-only and exhaustive
• Early stopping ends training when the validation metric stops improving
• Pick distributed training by what does not fit: model vs data
• Do not distribute a model that fits comfortably on one GPU
• Managed spot training cuts training cost up to 90%
• Pair managed spot with checkpointing so a reclaim resumes, not restarts
• Warm pools keep clusters alive to skip cold start on the next job
• Model size is driven by architecture, not data volume
• Regularization fixes overfitting (high variance), never underfitting
• L1 zeros weights (feature selection); L2 shrinks them smoothly
• Dropout randomly disables neurons each step to stop co-adaptation
• Use script mode to run your own TensorFlow or PyTorch code
• Fine-tune a pre-trained model with JumpStart or Bedrock instead of training from scratch
• Avoid catastrophic forgetting when fine-tuning on a narrow dataset
• Ensembling: bagging is parallel, boosting is sequential, stacking learns a combiner
• Shrink a model with pruning and quantization for a cheaper endpoint
• Register every model version in the Model Registry for repeatable, auditable promotion
• AMT warm start seeds a tuning job from up to 5 prior tuning jobs
• In script mode, save the model to SM_MODEL_DIR and read inputs from SM_CHANNEL_
• Add extra Python packages with a requirements.txt in source_dir, not a custom image
• AMT early stopping needs TrainingJobEarlyStoppingType=AUTO and per-epoch objective metrics
• Use AMT completion criteria to stop the whole tuning job once it stops improving
• Use Logarithmic scaling for orders-of-magnitude ranges and ReverseLogarithmic near 1.0
• Enable SMDDP for optimized multi-node AllReduce/AllGather on AWS
• Scale the learning rate up (with warmup) as the global batch size grows
• Combine sharding and tensor/context parallelism (SMP v2) to fit and scale huge models
• Match the fine-tuning variant to your data: labeled pairs vs unlabeled domain text
• Track runs with load_run()/log_metric() and pull results with ExperimentAnalytics

Unlock with Premium — includes all practice exams and the complete study guide.