ML Model Development
Never build what you can buy or borrow
Faced with any modeling question, the winning move is almost always to solve the problem with the least code, the least data, and the least infrastructure you can get away with. Picture a build-vs-buy-vs-tune ladder with four rungs, cheapest first: a managed AI service (a pre-trained API such as Amazon Rekognition or Amazon Comprehend), a foundation model on Amazon Bedrock, a SageMaker built-in algorithm or JumpStart model you train or fine-tune, and finally a fully custom model you write yourself. Climb only as far as the problem forces you and stop at the first rung that meets the requirement. The classic MLA-C01 trap dangles a from-scratch custom model when a pre-trained API would already do the task, so the higher rung is usually the wrong answer even though it also works.
The domain unfolds in three phases: choose, train and refine, then analyze
Read this page as a map, then follow the three subtopics in order. Choosing a Modeling Approach owns the ladder above: it maps the problem type to an algorithm family (regression to XGBoost or Linear Learner, forecasting to DeepAR, clustering to K-Means, anomaly detection to Random Cut Forest) and picks the right rung once you have decided to build. Training and Refining Models takes over once an approach is chosen: hyperparameter tuning with SageMaker Automatic Model Tuning (AMT), cost levers like managed spot training and early stopping, regularization to cure overfitting, and versioning every candidate in the SageMaker Model Registry. Analyzing Model Performance closes the loop, choosing the evaluation metric from the cost of the mistake and diagnosing whether the model actually converged. Each subtopic carries the mechanisms, the numbers, and the traps; this overview only shows how they fit together.
Optimize for the cost of the mistake, not for raw accuracy
The same instinct runs through all three phases: the most accurate-looking model is rarely the exam-correct one. When choosing, weigh interpretability, effort, and cost against accuracy, because a managed API on a small instance often beats a custom GPU fleet that scores a fraction higher. When refining, regularize to fix overfitting rather than chase another point of training accuracy. When analyzing, remember that accuracy lies on imbalanced classes, so a fraud or cancer screen optimizes recall, a costly-false-alarm system optimizes precision, and F1 or PR AUC balances the two. Always anchor a score against a baseline before celebrating it, and use SageMaker Clarify to check that the gains are not hiding unfair bias.
The three phases of model development (and where each is covered)
| Phase | Decides | Key AWS tools | Drill into |
|---|---|---|---|
| Choose | Which approach and algorithm fit the problem | AI services, Bedrock, SageMaker built-in algorithms, JumpStart | Choosing a Modeling Approach |
| Train and refine | How to tune, speed up, and version the model | SageMaker Automatic Model Tuning, managed spot training, regularization, Model Registry | Training & Refining Models |
| Analyze | Whether the model is actually good, and why | Precision/recall/F1, ROC and PR AUC, RMSE/MAE, SageMaker Clarify and Debugger | Analyzing Model Performance |