ML Model and Foundation Model (FM) Development

An insurer needs the claim number, date and total from thousands of scanned invoices: training a document model would need labeled examples, a training job and a hosted model, while the AnalyzeExpense operation of Amazon Textract already reads invoices and receipts. This domain sorts every machine learning (ML) and generative AI option into four build-effort levels, lowest first: a task-specific AWS AI service (such as Amazon Textract, Amazon Comprehend or Amazon Rekognition), a foundation model (FM) used as-is with prompts and Retrieval Augmented Generation (RAG, which retrieves relevant passages and adds them to the prompt at request time), a customized FM (an FM whose weights an FM customization method, such as fine-tuning, has changed), and a model you train yourself in Amazon SageMaker AI. The levels are a frame for this guide, not AWS terminology, but they follow AWS guidance that orders SageMaker AI implementation options by increasing levels of effort and describes a progressive journey from Amazon Bedrock prompting and RAG to customization to training jobs; the decision tree on this page walks the four levels in order. The classic trap is climbing a level out of habit, such as fine-tuning a model to keep up with a catalog that changes daily: RAG supplies knowledge while customization changes behavior, and an Amazon Bedrock knowledge base, an out-of-the-box RAG solution, removes the need to continually train your model on private data.

Choose ML and AI modeling approaches picks the build-effort level and, within it, the AWS AI service, the Bedrock FM, the RAG pattern, the FM customization method or the SageMaker AI algorithm, weighing quality, latency and cost together. Train and customize ML and AI models carries out that choice in two ways: training-time customization changes the model's weights (script mode, automatic model tuning (AMT), faster training, and guarding against overfitting and a fine-tuned FM losing capabilities it had before, called catastrophic forgetting), while inference-time customization changes only what each request carries, such as the prompt's instructions and examples, inference parameters and retrieved passages. This sense is broader than in the build-effort levels, where a customized FM (Level 3) has changed weights: an FM steered only by its requests stays at Level 2. Evaluate ML and AI system performance measures the result with three evaluation families that AWS prescriptive guidance uses: metric-based, model-based (an evaluator model scores the output, called LLM-as-a-judge) and human evaluation, and it adds reproducible experiments, shadow tests, baselines and explanations.

Starting at the lowest build-effort level also settles a choice between two answers that both work: change the request (a prompt, an inference parameter or fresher retrieved passages) before the weights, and, within a level, pick a smaller model that passes the same Amazon Bedrock evaluation before a larger one. Each of those decisions rests on a score, and every score is a comparison against a reference: ground truth (known-correct labels or reference text; metric-based), criteria scored by an evaluator model (model-based), or a person's judgment (human evaluation), so name the reference before you trust the number. For a RAG application the same instinct means evaluating retrieval, for example its context relevance, before blaming the generator model, because no prompt change can use a passage that retrieval never returned.