Domain 2 of 4 · Chapter 2 of 3

Train and customize ML and AI models

Unlock the complete study guide.

Bundled into the existing AWS Certified Machine Learning Engineer - Associate premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Two ways to customize: weights or requests
  • Script mode and SageMaker AI training storage paths
  • Epochs, steps, batch size and learning rate
  • Diagnosing fit: overfitting and underfitting
  • Catastrophic forgetting in FM fine-tuning
  • Automatic model tuning strategies
  • Configuring a tuning job
  • Managed warm pools: shorter startup between matching jobs
  • Managed Spot Training: lower price, with checkpoints to resume
  • Distributed training: data or model parallelism
  • Using multiple models
  • Prompt engineering for task-specific behavior
  • Inference parameters: randomness, length, stopping
  • Optimizing retrieval and embeddings

Automatic model tuning strategies: Bayesian vs Random vs Grid vs Hyperband

CriterionBayesianRandomGridHyperband
How the search proceedsRegression over the results of completed trials picks the next valuesRandom draw from the ranges, independent of earlier resultsWorks through each unrun combination of the categorical valuesGives promising trials more epochs and stops weak ones early
Results that steer the searchResults of completed trialsNoneNoneIntermediate and final results of trials
Concurrency guidanceLimited: its sequential nature cannot massively scaleHighest: trials are independentNo specific guidance; chosen for coverage and reproducibilityScales to many parallel trials
Range typesCategorical, integer, continuousCategorical, integer, continuousCategorical onlyCategorical, integer, continuous
Number of trialsCapped by the total-trial limit (MaxNumberOfTrainingJobs)Capped by the total-trial limit (MaxNumberOfTrainingJobs)Equals the number of distinct combinationsCapped by the total-trial limit (MaxNumberOfTrainingJobs)
Fits whenEach trial is costly and should build on earlier resultsTrials must be independent or run at high concurrencyA small enumerated space must be covered completely or reproducedIterative training reports a metric per epoch or resource level

Decision tree

Cover every categorical combination?Grid searchcategorical values onlyYesNoIterative, with per-epoch metrics?Hyperbandstops weak trials earlyYesNoShould trials learn from earlier results?Bayesian optimizationcannot massively scaleYesNoRandom searchmost parallel trials

Cheat sheet

  • Script mode runs custom training code inside a supported framework environment
  • Training dependencies must be available in the remote framework environment
  • Training scripts read data through the job's configured input channels
  • Final model artifacts must be written to the designated model output directory
  • Bayesian tuning uses previous results to choose promising configurations
  • Random search supports independent concurrent tuning trials
  • Grid search enumerates the supplied categorical combinations
  • Hyperband reallocates training resources using intermediate trial performance
  • A tuning job optimizes one selected objective metric in the specified direction
  • Custom tuning metrics must be emitted and matched by metric definitions
  • Logarithmic search is suitable for positive ranges spanning orders of magnitude
  • Total trial limits and concurrent trial limits control different tuning resources
  • Warm-start tuning reuses search results rather than retained training infrastructure
  • Warm-start type determines whether the tuning dataset can change
  • Managed warm pools reduce startup time between matching training jobs
  • Warm-pool reuse requires matching resource and security settings
  • The warm-pool persistent cache survives matching jobs but not pool termination
  • Data parallelism distributes examples across replicas of a model
  • Model parallelism partitions a model that does not fit on one device
  • Early stopping saves training work when validation performance stops improving
  • Managed Spot Training trades interruption tolerance for lower training cost
  • Checkpoints let interrupted training resume instead of restarting from the beginning
  • More training workers do not guarantee lower training time or cost
  • Strong training performance with weak validation performance indicates overfitting
  • Poor training fit calls for learning capacity or training evidence before stronger regularization
  • Weight penalties and dropout reduce overfitting through different mechanisms
  • Mixing general-capability examples into fine-tuning can reduce catastrophic forgetting
  • Epochs count dataset passes while steps count training iterations
  • Reducing batch size can resolve training out-of-memory failures
  • Learning rate trades update speed against convergence stability
  • Learning-rate warmup gradually introduces the target update size
  • Bagging combines separately trained learners rather than a sequential correction chain
  • Stacking learns how to combine base-model predictions
  • Validation performance determines whether a base model or stacked ensemble is preferable
  • Intelligent prompt routing selects a model per request rather than combining generated answers
  • Few-shot prompting provides task examples without a training job
  • A task prompt needs both the instruction and the material to operate on
  • Temperature changes sampling variation rather than supplying missing knowledge
  • Top K limits candidate count while Top P limits cumulative probability mass
  • Output limits and stop sequences terminate generation for different reasons
  • Direct model requests need the relevant conversation history in the current input
  • Generation parameter support and ranges depend on the selected model
  • Prompt templates separate reusable instructions from changing inputs and support controlled comparisons
  • Hybrid retrieval combines raw-text matching with vector similarity
  • Metadata filters constrain which source documents are eligible for retrieval
  • Reranking reorders retrieved chunks using a relevance model
  • The requested retrieval result count is a maximum rather than a guarantee
  • Logical document segments can improve retrieval even when the embedding model accepts longer input
  • Multilingual input support does not guarantee strong cross-language retrieval
  • Bulk indexing and interactive query embedding have different performance priorities

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. https://docs.aws.amazon.com/machine-learning/latest/dg/model-fit-underfitting-vs-overfitting.html
  2. https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-engineering-guidelines.html
  3. https://docs.aws.amazon.com/sagemaker/latest/dg/algorithms-choose.html
  4. https://docs.aws.amazon.com/nova/latest/userguide/nova-forge-sft.html
  5. https://docs.aws.amazon.com/aws-certification/latest/machine-learning-engineer-associate-02/machine-learning-engineer-associate-02-domain2.html
  6. https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints.html
  7. https://docs.aws.amazon.com/sagemaker/latest/dg/model-train-storage-env-var-summary.html
  8. https://docs.aws.amazon.com/sagemaker/latest/dg/distributed-training.html
  9. https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-llms-finetuning-hyperparameters.html
  10. https://aws.amazon.com/what-is/overfitting/
  11. https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-how-it-works.html
  12. https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-considerations.html
  13. https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-define-metrics-variables.html
  14. https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-define-ranges.html
  15. https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-warm-start.html
  16. https://docs.aws.amazon.com/sagemaker/latest/dg/train-warm-pools.html
  17. https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html
  18. https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-metrics-validation.html
  19. https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html
  20. https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-management.html
  21. https://docs.aws.amazon.com/bedrock/latest/userguide/inference-parameters.html
  22. https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html
  23. https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-config.html
  24. https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-build-graphs.html
  25. https://docs.aws.amazon.com/bedrock/latest/userguide/rerank.html