Previous version: MLA-C01. The current version is
MLA-C02.
Which evaluation metric for which problem
| Metric | Problem type | Optimize when | Key caveat |
|---|
| Accuracy | Balanced classification | Errors cost about the same and classes are even | Misleads on imbalanced data: a constant predictor can score high |
| Precision | Classification | A false positive is expensive (e.g. blocking a good transaction) | Ignores missed positives; pair with recall |
| Recall (sensitivity) | Classification | A missed positive is expensive (e.g. cancer, fraud) | Ignores false alarms; pair with precision |
| F1 score | Imbalanced classification | You need precision and recall jointly high | Harmonic mean hides which of the two is weak |
| ROC AUC | Classification | Ranking models across all thresholds | Can look optimistic when negatives vastly outnumber positives |
| PR AUC | Heavily imbalanced classification | Positive class is rare and you care about it | Less familiar than ROC; baseline shifts with class ratio |
| RMSE | Regression | Large errors must be penalized hard | Outlier-sensitive; no confusion matrix applies |
| MAE | Regression | You want robustness to outliers | Treats all error sizes linearly |
Decision tree
Cheat sheet Cheat sheet
- On imbalanced classes, accuracy is the wrong headline metric
- Precision is TP/(TP+FP); optimize it when false positives are costly
- Recall is TP/(TP+FN); optimize it when misses are costly
- F1 is the harmonic mean of precision and recall
- Read recall down the row, precision down the column of the confusion matrix
- ROC AUC ranks models across all thresholds, not at one operating point
- Prefer the PR curve over ROC AUC on heavily imbalanced data
- Regression uses RMSE or MAE, never a confusion-matrix metric
- RMSE punishes large errors; MAE resists outliers
- A score is meaningless without a baseline
- SageMaker Clarify computes eleven post-training bias metrics
- SageMaker Clarify produces SHAP values for explainability
- SageMaker Debugger captures tensors and fires built-in rules during training
- Overfitting is low training error but high validation error
- Exploding gradients give NaN loss; vanishing gradients stall early layers
- Shadow tests mirror live traffic but never serve the shadow response
- A/B testing splits real traffic across production variants and serves both
- SageMaker Experiments tracks runs to make comparisons reproducible
- Weigh accuracy gains against training time and inference cost
- Debugger profiler rules diagnose resource bottlenecks (CPUBottleneck, LowGPUUtilization, IOBottleneck)
- Automate reactions to Debugger findings via IssueFound events to Lambda
- suggest_baseline generates statistics.json + constraints.json you can edit before use
- ModelQualityMonitor needs ground-truth labels and time offsets for delayed labels
- Clarify SHAP agg_method: median resists outliers, mean_abs weighs both directions
- Compare DPL (pre-training) with DPPL (post-training) to see if training added bias
- ModelExplainabilityMonitor detects feature-attribution drift via NDCG on SHAP rankings