Monitor ML and AI Inference and Performance

For example, suppose a fraud-scoring endpoint reports no errors and steady latency, yet the fraud team reports that it misses more fraud: if the statistical nature of the data a model receives in production drifts away from the baseline data it was trained on, the model begins to lose accuracy (Model Monitor data quality^[1]). Watching a model in production means watching more than whether requests succeed.

Production signals differ in the reference they are compared with (Model Monitor data quality^[1], Model Monitor model quality^[2]):

• Operational signals answer "is the request path working, and how fast?" They compare telemetry with alarm thresholds: invocation counts, latency, errors, throttles, how a stream ended (stopReason, MidStreamErrors) and which workflow step slowed or failed (StepDuration, StepFailed) (MessageStopEvent^[3], SageMaker AI metrics in CloudWatch^[4]). Amazon CloudWatch metrics and logs, and the traces of Amazon Bedrock AgentCore Observability, supply them.
• Drift signals answer "has live data or model behavior changed?" They compare production data with a baseline: the statistics and constraints computed from an approved reference dataset, usually the training dataset. Data quality drift, bias drift and feature attribution drift are drift signals.
• Quality signals answer "are the predictions or generated answers correct?" They need a reference of correctness: ground truth labels (the outcomes that actually happened), reference answers, a judge model, or human raters.

Amazon SageMaker Model Monitor^[5] lists model quality among its four monitoring types, next to data quality, bias drift and feature attribution drift. This page files model quality under quality signals anyway, because model quality monitoring^[2] compares the predictions that the model makes with the actual ground truth labels, as the figure below shows.

Two availability changes shape the tool choice: Model Monitor^[6] and Amazon SageMaker Clarify^[7] are no longer open to new customers, while existing customers can continue to use them. AWS documents the AWS-published open-source SageMaker AI monitoring solutions (the open-source solutions for short), combined with Amazon QuickSight and Amazon CloudWatch, as a Model Monitor replacement (Model Monitor availability change^[6]).

The AWS MLA-C02 exam guide lists 'Task 4.1: Monitor ML and AI model inference and performance.' under Domain 4^[8], separately from 'Task 2.3: Analyze and evaluate the performance of ML and AI systems.' under ML Model and Foundation Model (FM) Development (Domain 2^[9]) and from 'Task 4.2: Optimize and manage ML and AI infrastructure costs and performance.' and 'Task 4.3: Secure ML and AI workloads and model endpoints.' in Domain 4; see Evaluate ML and AI system performance, Optimize ML and AI Costs and Performance and Secure ML and AI Workloads and Endpoints.

The takeaway: name the question first (working, changed, or correct), because drift is measured against a baseline^[1] and model quality against actual ground truth labels^[2].

Operational signals are an important source of alarms in production, and reading them correctly depends on knowing exactly what each metric counts. SageMaker AI endpoints and the Amazon Bedrock runtime each publish inference metrics to Amazon CloudWatch (SageMaker AI metrics in CloudWatch^[4], Bedrock runtime metrics^[10]).

SageMaker AI endpoint metrics

A SageMaker AI real-time endpoint publishes request metrics to CloudWatch in the AWS/SageMaker namespace, among them Invocations, Invocation4XXErrors, Invocation5XXErrors, ModelLatency and OverheadLatency (SageMaker AI metrics in CloudWatch^[4]). The EndpointName, VariantName dimension filters them per production variant (one model plus its own instances behind the shared endpoint), which the A/B testing section below defines and relies on.

Amazon Bedrock runtime metrics

The Bedrock runtime publishes its metrics in the AWS/Bedrock namespace (Bedrock runtime metrics^[10]). The trap is that client errors, server errors and throttles have different counting rules, and some errors are also counted in Invocations:

Metric	What it measures
Invocations	Requests to Converse, ConverseStream, InvokeModel and InvokeModelWithResponseStream that Bedrock accepted for processing, including requests that later failed with a client-side or server-side error. Throttled requests, and requests rejected during input validation before processing began, are not counted.
InvocationClientErrors	Accepted invocations that ended in a client-side error (also counted in Invocations). This count also includes requests rejected for exceeding a service quota: despite the word "rejected", Bedrock reports them as client errors (so they are also in Invocations), not as throttles.
InvocationServerErrors	Accepted invocations that ended in an AWS server-side error (also counted in Invocations).
InvocationThrottles	Requests the system throttled. They are not counted in Invocations, InvocationClientErrors or InvocationServerErrors, and the number you see depends on your SDK retry settings.
InvocationLatency	Time from sending the request to receiving the last token.
TimeToFirstToken	Time from sending the request to receiving the first token, for the streaming operations ConverseStream and InvokeModelWithResponseStream.

First, a rising failure rate needs its category before a fix: throttles are requests Bedrock never accepted for processing, client errors point at the request itself or at an exceeded service quota, and server errors point at the AWS side. Adding every failure up as a server error sends the investigation to the wrong place. Second, the two latency metrics answer different questions. A streaming chat user feels TimeToFirstToken, the wait before text starts to appear. InvocationLatency runs to the last token, so a longer answer can raise it without increasing the wait for the first word.

Bedrock publishes these metrics both with and without the ModelId dimension, and a request that fails before Bedrock determines which model it targets is published without ModelId. Filtering by ModelId can therefore undercount errors; query without a dimension to account for every request.

The takeaway: read throttles, client errors and server errors as three separate signals, and pick the latency metric that matches the experience you are protecting.

A streamed response starts successfully long before it finishes, so the status of the first bytes says nothing about the last ones. Both services report a stream's outcome after the first response: SageMaker AI as a CloudWatch metric, and Bedrock as events inside the stream (SageMaker AI metrics in CloudWatch^[4], ConverseStream API reference^[11]).

Errors after the stream has started

On a SageMaker AI endpoint that streams responses, an error can occur after the initial response has already been sent to the caller. SageMaker AI counts these in the MidStreamErrors metric (SageMaker AI metrics in CloudWatch^[4]). An application that records success as soon as the first chunk arrives logs these failures as successes; monitor stream completion and MidStreamErrors instead. Bedrock ConverseStream reports the equivalent inside its event stream: a modelStreamErrorException (HTTP status code 424) means a streaming error occurred and the request should be retried (ConverseStream API reference^[11]).

The stop reason

When a Bedrock ConverseStream message ends, its messageStop event carries a stopReason: the reason the model stopped generating output (MessageStopEvent^[3]). The valid values are end_turn, tool_use, max_tokens, stop_sequence, guardrail_intervened, content_filtered, malformed_model_output, malformed_tool_use and model_context_window_exceeded. For example, end_turn, max_tokens, tool_use and guardrail_intervened each call for different handling:

stopReason	What happened	What the application does
end_turn	The model finished its turn	Treats the answer as complete
max_tokens	The output reached the output-token limit	Treats the answer as truncated, not complete
tool_use	The model is requesting a tool call	Runs the tool and returns its result
guardrail_intervened	A guardrail intervened	Treats the response as filtered

A guardrail here is an Amazon Bedrock Guardrails resource, which provides configurable safeguards that detect and filter undesirable content in user inputs or model responses (Amazon Bedrock Guardrails^[12]).

A tool_use stop can be misread in agent monitoring. The stream ended on purpose, because the model handed control to the application, which must run the requested tool and send the result back (client-side tool calling is covered in managing model deployment infrastructure). Treating it as a network truncation and retrying generation throws the tool request away. A rising share of max_tokens stops is the truncated-streaming signal: answers are being cut off by the output limit even when the affected calls return successful API responses.

The takeaway: log the stop reason with every streamed response, and count a stream as successful only when it completed and ended for the reason you expected.

A SageMaker AI pipeline chains processing, training and batch inference steps, so when a run fails or slows down, the first question is which step did it. This section covers how to localize that step with pipeline metrics.

SageMaker AI publishes pipeline metrics to the AWS/Sagemaker/ModelBuildingPipeline namespace at a 1-minute frequency (SageMaker AI metrics in CloudWatch^[4]). Execution metrics (ExecutionStarted, ExecutionFailed, ExecutionSucceeded, ExecutionStopped, ExecutionDuration) use the PipelineName dimension; step metrics (StepStarted, StepFailed, StepSucceeded, StepStopped, StepDuration) use PipelineName, StepName. When a pipeline run gets slower, ExecutionDuration only confirms it, while StepDuration per step shows whether the time went into an upstream processing step or into inference. Scaling the inference endpoint because the whole pipeline got slower treats a cause the step metrics never pointed to.

The takeaway: localize a slow or failed run with per-step metrics such as StepDuration and StepFailed, not with the execution totals.

An agent or inference workflow in AWS Step Functions chains model calls and tools, and any one call can fail transiently or hang. This section covers how to automate recovery from transient errors.

By default, when a state reports an error, Step Functions fails the entire execution (Step Functions error handling^[13]). Two fields on a Task state change that: Retry holds a list of retriers, each a rule whose ErrorEquals field lists the error names it matches (Step Functions error handling^[13]) and that runs the state again after a delay, and Catch holds a list of catchers, each sending a matching error (again listed in its ErrorEquals) to a recovery state when no retrier matches or the retries fail to resolve it, as the figure below shows. Besides ErrorEquals, a retrier is configured with these fields:

Field	What it sets	Default
IntervalSeconds	Seconds before the first retry	1
MaxAttempts	Maximum number of retries; 0 means never retry	3
BackoffRate	Multiplier applied to the interval after each retry	2.0
MaxDelaySeconds	Upper limit on the growing interval	Optional field
JitterStrategy	FULL randomizes each wait; NONE keeps it as calculated	NONE

Backoff and jitter solve different problems. Exponential backoff makes each wait longer than the last (with an interval of 3 seconds and a backoff rate of 2, the waits are 3, 6 and 12 seconds), which gives a struggling dependency room to recover. Jitter spreads retries over a randomized delay so that hundreds of tasks that failed at the same moment do not all retry at the same moment again. Identical fixed intervals can keep synchronized failures retrying together.

A retry only begins after an error is reported, so a step that hangs without answering never triggers one, however high MaxAttempts is. Bound that wait with TimeoutSeconds: a Task state that runs longer reports States.Timeout. A worker that sends heartbeats can also set HeartbeatSeconds, and a gap between heartbeats longer than that value fails the state as well. Watch the wildcards: in a Retry or Catch, States.TaskFailed matches any known error except States.Timeout, while States.ALL must appear alone in the last retrier or catcher. A States.ALL catcher does not catch States.DataLimitExceeded or States.Runtime.

Retry and Catch fields on an agent Task state (example values)

"InvokeAgentStep": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke",
  "TimeoutSeconds": 300,
  "Retry": [
    {
      "ErrorEquals": ["States.TaskFailed"],
      "IntervalSeconds": 2,
      "BackoffRate": 2.0,
      "MaxAttempts": 3,
      "MaxDelaySeconds": 30,
      "JitterStrategy": "FULL"
    }
  ],
  "Catch": [
    {
      "ErrorEquals": ["States.ALL"],
      "Next": "RecordFailure"
    }
  ],
  ...
}

This fragment applies the table above to one Task state. The final ... marks omitted fields: the task's arguments, and a Next state, because a Task state must either set End to true or provide a state in Next (Task workflow state^[14]); add them before you use the definition. Its retrier values are examples, and some differ from the defaults in the table above; for instance, IntervalSeconds is 1 by default (Step Functions error handling^[13]). Because the retrier matches States.TaskFailed, a timeout after 300 seconds is not retried and goes straight to the catcher, whose States.ALL sends remaining catchable errors, including States.Timeout, to the RecordFailure state.

The takeaway: retry transient errors a bounded number of times with backoff and jitter, catch what remains, and put a timeout on every step that can hang.

When a live generative AI request is slow or wrong, an evaluation score (Bedrock evaluations, covered below) cannot show which call inside that request caused it; that needs the request's own metrics, logs and traces. This section covers the two AWS tools that collect them: Amazon CloudWatch generative AI observability and Amazon Bedrock AgentCore Observability.

CloudWatch generative AI observability

CloudWatch generative AI observability^[15] provides pre-configured views into the latency, usage and errors of generative AI workloads, and end-to-end prompt tracing that helps identify issues in components such as knowledge bases, tools and models. It has two pre-built capabilities:

• Model Invocations: a metrics dashboard on model usage and token consumption, with a curated invocation-log table that shows the input and output content of model inferences.
• Amazon Bedrock AgentCore agents: performance and decision metrics for AgentCore primitives such as agents, memory, built-in tools, gateways and identity.

The dashboards include total and average invocations, token usage, latency (average, P90, P99), error rates and throttling events. The invocation-log table depends on a setting you control: you must enable Bedrock model invocation logging^[16] and send the logs to CloudWatch Logs to view invocation content such as inputs and outputs, and you can mask sensitive data in those logs. Without invocation logging, the metrics appear but the prompts and responses do not.

AgentCore Observability for agents

An agent is a workflow of model calls and tool calls coordinated by agent code, so it can fail in the middle: a tool returns an error, or one step hands a bad result to the next (a coordination failure). AgentCore Observability^[17] shows each step in the agent workflow, so you can inspect the execution path, audit intermediate outputs, and find the step where the run went wrong. An end-to-end error count tells you how often an agent fails; a step-level trace tells you which dependency failed.

What you see depends on what you turn on:

• Built-in metrics, by default. AgentCore outputs a set of key metrics for agents, gateway resources and memory resources. The runtime metrics include Invocations, Throttles, System Errors, User Errors, Latency (from receiving the request to sending the final response token) and Session Count (AgentCore runtime metrics^[18]).
• Service-provided spans and application logs, after you enable observability on the agent resource. To view the spans and traces AgentCore generates, you first complete a one-time setup that turns on CloudWatch Transaction Search (add observability to AgentCore resources^[19]). The two bullets fit together: AWS describes the built-in metrics of the first bullet as default data available in Amazon CloudWatch, while this setup is what lets AgentCore deliver spans, because without Transaction Search it can't deliver them to the agent's log group.
• Custom spans, traces and metrics, after you instrument the code. Internal operations such as each tool call inside your agent need framework or application instrumentation; AWS documents the AWS Distro for OpenTelemetry (ADOT) SDK for CloudWatch integration. Opening a runtime metrics dashboard does not produce tool-level traces on its own.

AgentCore emits its telemetry in an OpenTelemetry-compatible format, so it can be correlated with the rest of an existing monitoring stack.

One gap catches investigators: AgentCore Runtime application logs include the request payload of each invocation, along with request, session, trace and span identifiers, but AgentCore Runtime does not capture the agent response, so the records have no response_payload field. If diagnosing wrong answers requires the response, record it from the agent code under your data-protection policy.

Automated recovery from transient tool or model failures in an orchestrated agent workflow uses the bounded retries, jitter and timeouts described in Automate recovery in Step Functions above.

The takeaway: CloudWatch generative AI observability correlates invocation metrics, logged content and traces; AgentCore Observability shows the failing step, provided observability is enabled and the internal operations are instrumented.

A drift detection pipeline is the machinery behind drift signals: it records what the model sees in production and compares it, on a schedule, with a baseline. Two approaches discussed here are Model Monitor for eligible existing customers and the open-source solutions:

• Amazon SageMaker Model Monitor, for existing customers only. Model Monitor is no longer open to new customers^[6]; existing customers can continue to use it, and AWS does not plan to introduce new features.
• The open-source solutions, which AWS documents, combined with Amazon QuickSight and Amazon CloudWatch, as the Model Monitor replacement. Their open-source tooling is SageMaker AI MLflow Apps and Evidently AI (an open-source drift-analysis library), and they run inside your own account (Model Monitor availability change^[6]).

Both follow the same four-step flow, and the figure below traces it. Separately, for lightweight data quality checks without the full monitoring pipeline, AWS also documents publishing custom CloudWatch metrics with anomaly detection alarms (Model Monitor availability change^[6]).

Step by step in Model Monitor

The steps follow the path of the data (How Model Monitor works^[5]):

1. Data capture. For a real-time endpoint, enable capture of incoming requests and the resulting predictions; for a batch transform job, enable capture of its inputs and outputs. The captured records land in Amazon S3.
2. Baseline. Create a baseline from the dataset that was used to train the model. The baseline job computes statistics and suggests constraints; with the SageMaker Python SDK, DefaultModelMonitor.suggest_baseline() writes them to statistics.json and constraints.json (Create a baseline^[20]).
3. Scheduled comparison. A monitoring schedule says what data to collect, how often, how to analyze it and which reports to produce. Each run compares the latest captured data with the baseline and reports values outside the constraints as violations.
4. Alert. Inspect the reports and metrics, and attach CloudWatch alarms to the thresholds you care about. A report sitting in Amazon S3 notifies nobody; the alarm is what turns a violation into a notification.

Batch predictions are covered too. Model Monitor supports continuous monitoring with a batch transform job that runs regularly, and on-schedule monitoring for asynchronous batch transform jobs, so a nightly scoring job does not need to move to an always-on endpoint just to be monitored.

The same flow without Model Monitor

For a new customer with an existing endpoint, the lightweight option among the open-source solutions (not the custom-metric checks above) is the Real-Time Inference Monitoring with Evidently AI + SNS solution (Model Monitor availability change^[6]). It keeps data capture on the endpoint and adds one SageMaker AI Processing step that runs Evidently, an EventBridge Scheduler schedule, and an Amazon SNS topic. Evidently's DataDriftPreset reads the training baseline as the reference dataset and recent inference data as the current dataset, and computes PSI (Population Stability Index) and KS (Kolmogorov-Smirnov) statistics for every feature. An alert fires when the share of drifted columns exceeds DriftThreshold (default 0.05) or when a feature listed in CriticalFeatures drifts. You adapt the monitoring window, the relevant features and the thresholds to the workload. Other solutions in the same set cover batch transform workloads, real-time endpoints deployed at scale with the AWS CDK, and large language model (LLM) output quality.

Whichever build you run, endpoint CPU utilization and invocation-error alarms are not drift detectors. They are operational signals, and they never compare a feature's live distribution with the reference data.

The takeaway: capture, compare with a baseline on a schedule, and alarm on violations; Model Monitor runs that flow for existing customers, and the open-source solutions run it for everyone else.

For the existing customers who still run it, Model Monitor has documented boundaries that can explain confusing monitoring results. This section lists them with the symptom each one produces (Model Monitor notes^[5]).

Two deployment concepts in the table are easy to mix up. An inference pipeline is a single SageMaker AI model made of a sequence of containers that pass each request along, such as a preprocessing container followed by a predictor. A multi-model endpoint hosts several separate models behind one endpoint. Model Monitor supports the first and not the second, and support for one says nothing about the other.

Symptom	Boundary	Correct response
No drift statistics for an image classifier's inputs	Model Monitor computes metrics and statistics on tabular data only; for an image classifier it can monitor the label output, not the image input	Monitor the tabular outputs with Model Monitor, and analyze image-input drift in a separate pipeline
Violations on many features right after a preprocessing change	The training dataset schema and the inference dataset schema should exactly match in the number and order of features	Realign the captured input with the training schema; loosening constraints hides the mismatch without fixing it
Cannot tell which container in an inference pipeline altered the data	Capture and analysis cover the entire inference pipeline, not individual containers	Use the pipeline-boundary data for overall checks, and add your own visibility into the intermediate stage
A multi-model endpoint cannot be monitored	Model Monitor supports only endpoints that host a single model	Plan a different monitoring approach for that endpoint
Endpoint invocations continue but captured records stop	Data capture stops capturing requests at high levels of disk usage, to avoid affecting inference; AWS recommends keeping disk utilization below 75%	Check disk utilization before reading the gap as a traffic drop; regenerating the baseline does not free disk
Violations after a deliberate retrain on new, validated data	The baseline still describes the old training data	Regenerate the baseline from the new dataset and update the schedule to use its statistics and constraints

Two more details sit behind the schema row: the baseline job assumes the prediction (output) columns are the first columns of the training dataset, and baseline column names must be Spark-compliant, using only lowercase characters with _ as the only special character (Create a baseline^[20]). A reordered or extra column can invalidate the comparison, so check the schema before treating the result as drift.

The last row needs a caution: a new baseline is right only when the change is expected and approved. Unexpected drift is the signal the monitor exists to raise, and re-baselining to make it disappear hides it. Retuning the model's hyperparameters is no fix for a stale baseline either, because hyperparameters do not change the reference data the monitor compares against.

The takeaway: before acting on a violation or a gap, check whether the data type, the schema order, the endpoint type, disk utilization or a stale baseline explains it.

A model that passed a fairness check at training time can still drift into unequal outcomes once the population it serves changes, and the features that drive its decisions can shift too. This section covers the two drift signals that watch for this, both powered by Amazon SageMaker Clarify inside Model Monitor. Clarify, like Model Monitor, is no longer open to new customers^[7]; existing customers can continue to use it.

Bias drift

Bias drift monitoring^[21] computes a bias metric on the data each monitoring window collected and raises an alert, viewable through CloudWatch, when the metric leaves an allowed range. A bias metric compares outcomes between groups; DPPL (difference in positive proportions in predicted labels), for example, compares how often each group receives the positive prediction. You specify the allowed range, for instance (-0.1, 0.1) for DPPL.

The subtle part is the alert rule. A short window may hold few samples, so its bias estimate can land outside the range purely by chance. Clarify therefore builds a confidence interval around the estimate (the Normal Bootstrap Interval method) and raises an alert only when that interval and the allowed range are disjoint. An estimate outside the range whose interval still overlaps the range is not a violation. Aggregate accuracy cannot stand in for this check, because a model can hold its overall accuracy while its outcomes for one group move.

For new customers, AWS documents segment-sliced metrics instead: Evidently computes outcome and performance metrics such as selection rate and true positive rate for each protected-attribute segment, logs them to SageMaker AI MLflow and Amazon QuickSight, and an Amazon SNS alert fires when the gap between segments crosses a threshold you define (Model Monitor availability change^[6]).

Feature attribution drift

Feature attribution drift asks a different question: has what drives the predictions changed? Clarify feature attribution monitoring^[22] compares the ranking of feature attributions in live data with the ranking in the training data using NDCG (normalized discounted cumulative gain). NDCG is sensitive to rank changes and weighs them by each feature's attribution score in the training data, so a high-importance feature falling down the ranking counts more than a minor one doing the same. A value of 1 means the ranking is unchanged, and Clarify raises an alert when NDCG falls below 0.90. In AWS's college admissions example, SAT score falls from the highest attribution in training (0.70) to the lowest in live data (0.10) while class rank rises from 0.05 to 0.70, and NDCG drops to 0.69.

The takeaway: bias drift alerts on a confidence interval rather than a single estimate, and feature attribution drift alerts on a ranking change.

Not every bias question needs a standing schedule. For example, suppose an auditor asks once whether a trained model's predictions differ between groups: an existing Clarify customer answers with a Clarify processing job^[23] launched whenever needed, not a monitoring schedule.

To launch it with the SageMaker Python SDK, define DataConfig (the input dataset and output location), ModelConfig (the model or endpoint to analyze) and BiasConfig (the bias analysis parameters), create a SageMakerClarifyProcessor, and call run_post_training_bias to measure bias in the trained model's predictions. run_pre_training_bias measures bias in the dataset before training, so it cannot answer a question about a model's predictions. The AWS MLA-C02 exam guide lists 'Task 1.3: Validate data quality and manage bias.' (Domain 1^[24]); see Validate data quality and manage bias.

The takeaway: a one-off post-training bias analysis is a processing job rather than a schedule, and run_post_training_bias is the call that measures a model's predictions.

Drift tells you something changed; only model-quality monitoring tells you whether predictions are still right, and it cannot run without the real outcomes. This section covers how those outcomes, the ground truth labels, reach the monitor (Ingest ground truth labels^[25]).

Data-quality monitoring and model-quality monitoring need different inputs. Data quality compares feature statistics with baseline constraints using captured data alone, so it works before any outcome is known. Model quality compares each captured prediction with its actual label and computes metrics that depend on the problem type: regression metrics such as MAE, MSE and RMSE, binary classification metrics such as accuracy, precision and recall, and multiclass metrics such as weighted F1 (Model quality metrics^[26]). Data-quality constraints describe inputs, so they cannot measure prediction accuracy even when labels exist.

Joining labels to the right predictions

Labels can arrive later than predictions (for example, a loan can default months after the credit decision). Model Monitor joins them through a unique identifier on each record:

Ground truth record format for Model Monitor model quality

{
  "groundTruthData": {
    "data": "1",
    "encoding": "CSV"
  },
  "eventMetadata": {
    "eventId": "aaaa-bbbb-cccc"
  },
  "eventVersion": "0"
}

In this record, groundTruthData.data holds the label and eventMetadata.eventId names the prediction it belongs to. That identifier is one of two values. Either way it goes in the record's eventMetadata.eventId field; the key is named eventId even when it holds an inferenceId (Ingest ground truth labels^[25]). If the caller supplied an inferenceId when invoking the endpoint and it is present in the captured records, Model Monitor merges on inferenceId, and you are responsible for making the label records carry the same value. Otherwise Model Monitor uses the eventId that was generated automatically when the endpoint was invoked. Joining by upload order would pair predictions with the wrong outcomes as soon as labels arrive out of sequence.

Where the labels go

Upload labels to Amazon S3 in JSON Lines format (.jsonl) under a path with the same format as captured data, s3://<bucket>/<prefix>/yyyy/mm/dd/hh. The date in that path is the date the label was collected, and it does not have to match the date the inference was made, so a label stored under a later date than its prediction is normal and must not be discarded. The labels are not embedded in the capture files: you pass their S3 location when you create the monitoring job, as GroundTruthS3Input in a Boto3 create_model_quality_job_definition call or as ground_truth_input to the SageMaker Python SDK's create_monitoring_schedule.

Alerting on quality

With enable_cloudwatch_metrics set to True on the schedule, model quality jobs send all their metrics to CloudWatch, under aws/sagemaker/Endpoints/model-metrics for real-time endpoints and aws/sagemaker/ModelMonitoring/model-metrics for batch transform jobs. A CloudWatch alarm on a metric then fires when it does not meet your threshold; the figure below traces the path from captured predictions and labels to that alarm.

For new customers, the open-source solutions compute ROC-AUC, precision, recall, F1 and the confusion matrix with Evidently's ClassificationPreset, and include a pattern that reconciles delayed ground truth with predictions by inference ID (Model Monitor availability change^[6]).

The takeaway: quality monitoring needs labels joined by a stable inference identifier and stored by collection date; add an alarm when you need notifications.

An FM application can return every response quickly and without errors and still answer badly, so FM monitoring needs quality signals of its own. Amazon Bedrock evaluations supply them for Bedrock models and knowledge bases, and also for models and RAG (Retrieval Augmented Generation) sources outside Bedrock (Bedrock evaluations^[27]). There are four kinds of evaluation (RAG evaluation splits further into two job types, one per stage, described below), and the choice depends on who or what can judge a good answer:

Evaluation type	Who scores	Fits when
Programmatic model evaluation	Computed metrics on your own prompt dataset or a built-in dataset	You need fast, repeatable scores for a task such as summarization or question answering
Judge-model evaluation	A second LLM, the judge model (AWS also calls it the evaluator model), scores each response and explains the score	You need qualitative scoring, such as correctness or helpfulness, at scale
Human-based evaluation	Employees of your company or subject-matter experts rate responses	Good answers depend on expert judgment or user preference
RAG evaluation	A judge model scores retrieval, or retrieval plus generation	The application answers from a knowledge base

Prompts must be representative of the application: the scores describe how the model handles the dataset you give it, so a dataset unlike production traffic produces scores unlike production quality. A judge-model job can also score responses you supply from a model outside Bedrock, in which case Bedrock skips the invocation step (judge-model evaluation^[28]).

Splitting a RAG failure into retrieval and generation

A RAG answer can fail at two stages, and Bedrock offers one RAG evaluation job type per stage (RAG evaluation metrics^[29]). A retrieve-only job scores the retrieved passages with context relevance (are they relevant to the question?) and context coverage (do they cover all the information in the ground truth texts?); context coverage requires ground truth texts in the prompt dataset. A retrieve-and-generate job scores the answer with correctness, completeness, helpfulness, logical coherence, faithfulness, citation precision, citation coverage, harmfulness, stereotyping and refusal. Faithfulness measures how well the response avoids hallucination with respect to the retrieved texts, while correctness measures how accurately it answers the question. The figure below places each job type on the request path.

The split tells you where to look. Missing source passages are a retrieval problem, and judging the fluency of the answers will not reveal them. Relevant passages followed by invented details are a generation problem: high context relevance does not prove faithfulness, because a model can add unsupported claims after it receives the right passages.

Feeding production back into the evaluation dataset

AWS's GenAIOps guidance treats evaluation as a continuous loop: valuable production interactions, especially user-reported failures, are triaged, converted into new test cases, and added to a version-controlled evaluation dataset (GenAIOps preproduction hardening^[30]). Signals worth collecting include user ratings such as thumbs up and thumbs down, user comments, and behavior such as rephrasing a query. Review each new case against reliable ground truth before it becomes a benchmark, because a generated answer or a rating is evidence, not a verified answer. Protect customer interactions under your privacy and regulatory obligations before reusing them.

Recurring failures usually point to a cause the next change should target: users asking new kinds of questions, an unclear prompt, or knowledge the retrieval source lacks. Answers that fail because a newly released product is missing from the knowledge base are a retrieval gap. Add the documents to the data source and sync it, which re-indexes only the added, modified or deleted documents (update a knowledge base data source^[31]), then run the evaluation again. Fine-tuning changes model weights, and it is not the first fix for missing knowledge (the knowledge-versus-behavior distinction in choosing ML and AI modeling approaches).

The takeaway: operational metrics say the FM answered; Bedrock evaluations on representative, production-refreshed prompts say whether it answered well, and RAG evaluations say which stage failed.

An A/B test answers which of two candidate models performs better on real traffic, and on SageMaker AI it runs on one endpoint with several production variants (Testing models with production variants^[32]). A production variant is one model plus its own instance type and instance count, served behind the shared endpoint.

There are two ways to decide which variant serves a request:

• Weighted routing. Each variant has a weight in the endpoint configuration, and SageMaker AI distributes requests in proportion to the weights: two variants with a weight of 1 each receive half the traffic each. The weight is the traffic control. Adding instances to a variant adds capacity, not traffic share.
• Targeted routing. The caller sets TargetVariant on the InvokeEndpoint call, and SageMaker AI processes the request on that variant. Targeted routing overrides the weighted random distribution, so it is how you keep a controlled cohort (internal testers, for example) on one variant; random weights cannot pin a particular customer to a variant.

The figure below traces both paths.

Reading the results per variant

Each inference response names the production variant that processed the request, and SageMaker AI emits metrics such as latency and invocations for each variant in CloudWatch, filterable with the EndpointName, VariantName dimension. Keep the variant name with every prediction you later score against outcomes, and compare accuracy, latency and errors variant by variant. An endpoint-wide average blends both candidates and can hide a poorly performing one.

Shifting traffic to the winner

When one variant performs better, call UpdateEndpointWeightsAndCapacities to change the weights. It changes the traffic distribution without requiring updates to the endpoint, so there is no second endpoint to build. In AWS's example the split moves from 50/50 to 25/75 while the metrics are watched, then the old variant's weight goes to 0 and the winner's to 1, and only then is the old variant removed from the endpoint.

Shadow testing, which copies live requests to a candidate without returning its responses to callers, is a different technique covered in automating ML and AI delivery pipelines. In an A/B test, every response a variant produces goes back to a real caller.

The takeaway: weights split traffic, TargetVariant pins it, results are read per variant, and the winner gets more traffic through a weight update rather than a new endpoint.

In these monitoring scenarios, distinguish the kind of signal (operational, drift or quality) before selecting a tool. The AWS MLA-C02 exam guide lists 'Monitor workflows to detect anomalies or errors in data processing or model inference' and 'Detect changes in data distribution that can affect model performance' among the Task 4.1 skills (AWS MLA-C02 exam guide, Domain 4^[8]).

Scenario	Right move	Tempting wrong answer, and why it fails
Endpoint healthy, predictions getting worse, outcome labels arrive weekly	Model-quality monitoring with labels joined by inferenceId (or the automatically generated eventId)	Data-quality constraints: they describe inputs, not prediction correctness
New account needs scheduled drift detection	The open-source solutions: scheduled Evidently Processing step plus SNS	A new Model Monitor schedule (closed to new customers); CPU or error alarms (no distribution comparison)
Invocations continue but captured records disappear	Check endpoint disk utilization (keep below 75%)	Regenerating the baseline, which does not free disk
Many violations right after a preprocessing column change	Realign the inference schema with the training schema	Raising drift tolerances, which hides the mismatch
Approved retrain on a new validated dataset now raises violations	Regenerate the baseline from the new dataset and update the schedule	Retuning hyperparameters, which leaves the monitor's reference data unchanged
Small monitoring window shows a bias estimate outside the allowed range	Check whether the confidence interval still overlaps the range	Treating the single estimate as a confirmed violation
A controlled cohort must always reach the new model	TargetVariant on InvokeEndpoint	Variant weights, which route randomly
One variant wins the A/B test	UpdateEndpointWeightsAndCapacities on the existing endpoint	A second endpoint, or more instances as a traffic control
Streaming chat feels slow to start	Bedrock TimeToFirstToken	InvocationLatency, which runs to the last token
A stream ends with tool_use	Run the tool and return its result	Retrying generation as if the stream were truncated
Fluent RAG answers contain claims the sources do not support	Retrieve-and-generate evaluation, faithfulness	Context relevance alone, which scores the passages, not the answer
An agent fails intermittently inside a multi-tool flow	AgentCore Observability traces, with ADOT instrumentation for tool spans	Restarting components based on aggregate latency, or relying on the runtime dashboard alone
Hundreds of workflow tasks fail at the same moment	Retry with backoff, JitterStrategy FULL, a finite MaxAttempts, then Catch	The same fixed retry interval for every task
A workflow step never returns	TimeoutSeconds (or HeartbeatSeconds) with a catcher	More retries, which start only after an error is reported
A SageMaker AI pipeline run got slower	StepDuration by PipelineName, StepName	Scaling the inference endpoint on the overall duration alone

A useful approach to these examples: first decide whether the question is about the request path working, about change against a baseline, or about correctness, then pick a relevant tool and localize by variant, step or stop reason.