Domain 2 of 4 · Chapter 3 of 3

Analyzing Model Performance

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing AWS Certified Machine Learning Engineer - Associate premium course — no separate purchase.

Included in this chapter:

  • Reading the confusion matrix and its metrics
  • Baselines, Clarify bias and SHAP, and Debugger
  • Exam patterns: metric choice, shadow vs A/B, tradeoffs

Which evaluation metric for which problem

MetricProblem typeOptimize whenKey caveat
AccuracyBalanced classificationErrors cost about the same and classes are evenMisleads on imbalanced data: a constant predictor can score high
PrecisionClassificationA false positive is expensive (e.g. blocking a good transaction)Ignores missed positives; pair with recall
Recall (sensitivity)ClassificationA missed positive is expensive (e.g. cancer, fraud)Ignores false alarms; pair with precision
F1 scoreImbalanced classificationYou need precision and recall jointly highHarmonic mean hides which of the two is weak
ROC AUCClassificationRanking models across all thresholdsCan look optimistic when negatives vastly outnumber positives
PR AUCHeavily imbalanced classificationPositive class is rare and you care about itLess familiar than ROC; baseline shifts with class ratio
RMSERegressionLarge errors must be penalized hardOutlier-sensitive; no confusion matrix applies
MAERegressionYou want robustness to outliersTreats all error sizes linearly

Decision tree

Continuous target?(regression vs classification)Yes: regressionRMSE or MAERMSE punishes big errors; MAE robust to outliersNo: classificationClasses imbalanced?is the positive class rare?No: balancedAccuracyerrors cost about the sameYes: imbalancedWhich error costs more?or ranking models across thresholdsMissing a positiveRecallTP/(TP+FN), catch the rare classFalse alarmPrecisionTP/(TP+FP), avoid false hitsBoth matterF1 or PR-AUCharmonic mean; PR curve on rare classRank onlyROC AUCthreshold-free; optimistic if negatives swamp positivesAlways: compare against a baseline firstno-skill floor (majority class, prior version, Model Monitor reference)
Cheat sheet

Cheat sheet

  • On imbalanced classes, accuracy is the wrong headline metric
  • Precision is TP/(TP+FP); optimize it when false positives are costly
  • Recall is TP/(TP+FN); optimize it when misses are costly
  • F1 is the harmonic mean of precision and recall
  • Read recall down the row, precision down the column of the confusion matrix
  • ROC AUC ranks models across all thresholds, not at one operating point
  • Prefer the PR curve over ROC AUC on heavily imbalanced data
  • Regression uses RMSE or MAE, never a confusion-matrix metric
  • RMSE punishes large errors; MAE resists outliers
  • A score is meaningless without a baseline
  • SageMaker Clarify computes eleven post-training bias metrics
  • SageMaker Clarify produces SHAP values for explainability
  • SageMaker Debugger captures tensors and fires built-in rules during training
  • Overfitting is low training error but high validation error
  • Exploding gradients give NaN loss; vanishing gradients stall early layers
  • Shadow tests mirror live traffic but never serve the shadow response
  • A/B testing splits real traffic across production variants and serves both
  • SageMaker Experiments tracks runs to make comparisons reproducible
  • Weigh accuracy gains against training time and inference cost
  • Debugger profiler rules diagnose resource bottlenecks (CPUBottleneck, LowGPUUtilization, IOBottleneck)
  • Automate reactions to Debugger findings via IssueFound events to Lambda
  • suggest_baseline generates statistics.json + constraints.json you can edit before use
  • ModelQualityMonitor needs ground-truth labels and time offsets for delayed labels
  • Clarify SHAP agg_method: median resists outliers, mean_abs weighs both directions
  • Compare DPL (pre-training) with DPPL (post-training) to see if training added bias
  • ModelExplainabilityMonitor detects feature-attribution drift via NDCG on SHAP rankings

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. https://docs.aws.amazon.com/sagemaker/latest/dg/autopilot-metrics-validation.html
  2. https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-fairness-and-explainability.html
  3. https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-measure-post-training-bias.html
  4. https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-feature-attribute-shap-baselines.html
  5. https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html
  6. https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-built-in-rules.html
  7. https://docs.aws.amazon.com/sagemaker/latest/dg/experiments.html
  8. https://docs.aws.amazon.com/sagemaker/latest/dg/mlflow.html