Manage deployment infrastructure for ML and AI models

For example, suppose a checkout cannot complete until a fraud score comes back, while a marketing team wants churn scores for every customer by morning; both need predictions, but their response-time and workload requirements differ, and AWS matches them to different options: Real-Time Inference suits millisecond latency requirements, and Batch Transform suits offline predictions on large batches of data available up front^[1]. Matching each requirement to the mechanism that meets it is the AWS MLA-C02 exam guide skill 'Select model inference strategies (for example, real-time and batch processing)'^[2] of Task 3.1, Manage deployment infrastructure for ML and AI model types, whose other skills include shared endpoints, models built outside of AWS, foundation models, agents and retrieval-augmented generation. For Task 3.2 (provisioning and configuring resources) and Task 3.3 (orchestration and CI/CD pipelines) of the same domain, see Provision and configure ML and AI workload resources and Automate ML and AI delivery pipelines.

The four options

Amazon SageMaker AI provides four options to deploy a trained model for predictions^[1]. The first question is whether each prediction answers an individual request at all:

• Batch transform has no per-request caller. A job starts compute instances, distributes the input files in Amazon S3 among them, and writes one output file per input file back to S3^[3]. AWS lists it for running inference when you don't need a persistent endpoint^[3], so nothing keeps running between nightly jobs; by contrast, a real-time endpoint kept running between scheduled scoring jobs holds serving instances while the offline workload sends no requests. It can split input files into mini-batches^[3], groups of records sent to the model container as one payload^[1].
• Real-time inference is a persistent endpoint (an HTTPS address backed by instances you choose) that answers each request synchronously. It suits sustained traffic that needs lower and consistent latency, with payloads up to 25 MB and processing times up to 60 seconds, or 8 minutes for streaming responses^[1].
• Serverless Inference also answers synchronously, but SageMaker AI launches compute when requests arrive and scales it in and out with traffic, so you choose no instance type and manage no scaling policy^[4]. AWS positions on-demand Serverless Inference for workloads that have idle periods between traffic spurts and can tolerate cold starts^[4]. Its payload limit is 4 MB^[1].
• Asynchronous Inference queues each request and writes its result to Amazon S3 later. It fits payloads up to 1 GB and processing times up to one hour^[5]; the next section follows its request path.

Cold starts: the price of on-demand idle savings

A cold start is the delay when a serverless endpoint that has been idle, or whose concurrent requests exceed its current usage, must spin up compute before it can process requests^[4]; its length depends on model size, model download time and container start-up time. Accepting that delay is what buys the saving: with on-demand Serverless Inference you pay for the duration of running the inference code and the amount of data processed, not for idle periods^[1]. Provisioned Concurrency keeps a set amount of serverless capacity warm for predictable bursts.

What Serverless Inference does not support

Serverless Inference is not a drop-in replacement for a real-time endpoint. Its feature exclusions^[4] are GPUs, AWS Marketplace model packages, private Docker registries, multi-model endpoints, VPC configuration, network isolation, data capture (recording endpoint requests and responses to Amazon S3), multiple production variants (several model versions behind one endpoint, each with its own instances), Amazon SageMaker Model Monitor and inference pipelines. If a scenario needs any one of them, for example a GPU-backed model or captured traffic for monitoring, Serverless Inference is ruled out even when traffic is bursty. Provisioned Concurrency does not lift these exclusions; it only keeps serverless compute initialized. The conversion is also one-way: you cannot update a real-time endpoint to serverless, and a serverless endpoint converted to real-time cannot be rolled back to serverless^[4].

The selection follows the caller. An offline dataset available up front suits batch transform; individually queued requests whose results are collected later suit Asynchronous Inference; a caller waiting on the response can use real-time inference, or Serverless Inference when traffic is intermittent, cold starts are acceptable and none of the excluded features is required. Check payload and processing-time limits as well. The comparison table The four SageMaker AI inference options and the decision tree at the end of this page lay out the same choice^[1].

This section follows one asynchronous request from submission to result, because its path through Amazon S3 is what separates Asynchronous Inference from the two synchronous options in the previous section.

You create an asynchronous endpoint the same way as a real-time one, adding an AsyncInferenceConfig object to the endpoint configuration^[5]. The request path^[5] then runs in five steps:

1. The client uploads the request payload to Amazon S3.
2. The client calls InvokeEndpointAsync, passing a pointer to that S3 object rather than the payload itself.
3. SageMaker AI queues the request and immediately returns an identifier and the output location.
4. When an instance processes the request, SageMaker AI writes the result to the S3 output location.
5. Optionally, Amazon SNS sends a success or error notification, so the application does not have to poll the output location.

The figure below follows the request from submission to notification. Two endpoint properties matter here. First, an endpoint configuration that contains AsyncInferenceConfig means the endpoint can only receive asynchronous invocations^[5], so an application that also needs synchronous answers needs a separate endpoint. Second, the instance count can scale down to zero; requests received while there are zero instances are queued for processing once the endpoint scales up^[1].

Serverless Inference can also sit at zero instances, so the two are easy to confuse. A serverless endpoint still returns each prediction in the HTTP response and accepts at most 4 MB per request, while an asynchronous endpoint never returns the prediction in the response and accepts payloads up to 1 GB. Intermittent traffic alone therefore points to Serverless Inference; a requirement to submit large inputs and collect results later points to Asynchronous Inference.

Hosting each model on its own endpoint can leave a company paying for mostly idle instances when each customer model receives infrequent traffic. SageMaker AI offers four ways for models to share an endpoint's instances, and they differ in how a request reaches a model:

Topology	How a request reaches a model	Fits when
Multi-model endpoint (MME)	The request names a model with TargetModel; SageMaker AI loads it into the shared container if needed	Many models on the same ML framework, with a mix of frequent and infrequent traffic
Multi-container endpoint with direct invocation	The request names one container with TargetContainerHostname; only that container runs	A set of independent models, possibly on different frameworks, kept in memory
Inference pipeline	The request enters the first container; each container's output is the next one's input	Dependent stages such as preprocessing, prediction and postprocessing
Inference components	Each model is deployed as its own component with declared CPU, accelerator and memory needs	Several models that must be added, removed or updated independently with explicit resource shares

Multi-model endpoints

A multi-model endpoint uses the same fleet of resources and a shared serving container to host all of your models^[6], which raises endpoint utilization compared with single-model endpoints. Because one container serves every model, it is ideal for a large number of models that use the same ML framework^[6], and it works best when the models are fairly similar in size and invocation latency^[6].

Multi-container endpoints with direct invocation

A multi-container endpoint deploys multiple containers, which can use different models or frameworks, on a single endpoint^[7]. With direct invocation, up to 15 inference containers share the endpoint, and each request specifies the container to invoke with the TargetContainerHostname parameter^[8]; only the container with that hostname processes the request^[8]. The hosting FAQ describes these containers as running with no cold starts, which suits keeping all the models in memory^[1]. Here a cold start is a model-loading delay: on a multi-model endpoint, less frequently used models may incur some cold start latencies since the models are dynamically loaded to an instance^[6], which differs from the serverless compute start-up described in the first section. This suits independent models with separate framework-specific containers; different frameworks alone do not rule out an MME, since GPU-backed MMEs can use Triton (NVIDIA Triton Inference Server, which can be used to deploy models from all major ML frameworks^[9]) to serve multiple frameworks.

Inference pipelines

An inference pipeline (listed among SageMaker AI's advanced endpoint options as Serial Inference Pipelines^[10]) is a SageMaker AI model composed of a linear sequence of two to fifteen containers^[11]. SageMaker AI handles each invocation as a sequence of HTTP requests: the first container handles the initial request, its intermediate response becomes the request to the second container, and so on, and SageMaker AI returns the final response to the client^[11]. All the containers run on each instance of the endpoint or transform job^[11], so one pipeline serves real-time requests or batch transform jobs, and the client never coordinates separate stage calls. The same multi-container endpoint is configured for one mode or the other: its containers run in sequence as an inference pipeline, or each container is accessed individually by direct invocation^[7]. A request to an endpoint configured for direct invocation must specify a target container^[8], so direct invocation runs one container and never the whole chain. The figure below contrasts the two modes: chained in an inference pipeline, or picked one at a time by direct invocation.

Inference components

An inference component is a SageMaker AI hosting object that deploys a model to an endpoint and specifies how that model uses the endpoint's resources^[12]: the CPU cores, accelerators and memory it requires. Because the component decouples the model from the endpoint, you can add or remove models as needed and update each model independently^[12]. Production-variant weights do a different job: they set the percentage of traffic routed to each variant^[13] and reserve no accelerator memory for any model.

The choice follows how the models relate: one framework and many models suggests an MME, independent models on different frameworks suggest direct invocation, dependent stages suit an inference pipeline, and explicit per-model resources with independent updates call for inference components.

This section explains why latency on a multi-model endpoint varies from model to model, and why adding a model needs no endpoint change. Both follow from one design choice: SageMaker AI does not download every model when you create the endpoint; it loads and caches models when you invoke them^[6].

Each request names its model in the TargetModel parameter, the model's path relative to the Amazon S3 prefix given when the SageMaker AI model was created^[14]. For each request, SageMaker AI does the following^[6]:

1. Routes the request to an instance behind the endpoint.
2. Downloads the model from the S3 bucket to that instance's storage volume.
3. Loads the model into the container's memory (CPU or GPU memory, depending on the instance type). If the model is already loaded, the invocation is faster because neither the download nor the load is needed.

The figure below shows the three outcomes: a model already in memory answers at once, a model still on the storage volume needs only a load, and a model on neither needs a download first. When an instance's memory utilization is high and another model must be loaded, SageMaker AI unloads unused models from the container; unloaded models remain on the storage volume and can be loaded again without another download from S3, and models are deleted from the volume only when it reaches capacity^[6]. SageMaker AI keeps routing a model's requests to the instance where it is already loaded, and sends some requests to another instance when the model is busy^[6].

What the cache means for design

• Latency depends on cache state. The application should tolerate occasional cold start latency when it invokes infrequently used models^[6]; here the cold start is the S3 download and memory load shown in the figure above, not the serverless compute start-up described in the first section. Models with significantly higher transactions per second or strict latency requirements belong on dedicated endpoints^[6], not in a cache where they compete with infrequent models for memory.
• Adding a model is an upload. To add a model, you upload it to the S3 bucket and invoke it; adding models to, or deleting them from, a multi-model endpoint does not require updating the endpoint^[6]. Creating a production variant for each new tenant model would use variant-based deployment, a different mechanism, instead of this dynamic loading.
• Files in S3 warm nothing. A model enters memory only when a request invokes it, so uploading more model files to the bucket does not reduce anyone's latency.

A multi-model endpoint trades predictable latency for density; a model that cannot accept an occasional load delay belongs on its own endpoint.

A model trained on a laptop, in another cloud or on an on-premises cluster is still a set of artifacts plus code that knows how to serve them, so moving it to AWS is a question of which runtime accepts that package. The AWS MLA-C02 exam guide names the skill 'Deploy models that were built outside of AWS into AWS environments (for example, Amazon SageMaker AI, Amazon Bedrock Custom Model Import)'^[2], and the deployment target follows from what you hold:

What you hold	Deployment target
Framework model artifacts, for example a PyTorch or TensorFlow model	A SageMaker AI model that uses an AWS-provided framework container
Customized weights of a supported foundation model architecture, to be served by Amazon Bedrock	Amazon Bedrock Custom Model Import
ML services already packaged as container images on Kubernetes	Amazon Elastic Kubernetes Service (Amazon EKS), with the images in Amazon Elastic Container Registry (Amazon ECR)

SageMaker AI hosting for an externally trained model

SageMaker AI can take a framework model that you trained outside of SageMaker AI and deploy it on any of its hosting options^[1]; a SageMaker AI training job is not a prerequisite, so retraining a compatible model just to obtain a training-job record adds nothing. You package the artifacts as a model.tar.gz file in the directory structure that the framework container expects^[1]. Before deploying, note the AWS Region of the S3 bucket, the S3 URI of the model artifacts, the IAM role for SageMaker AI, and the framework and version of an AWS-provided image^[12]. That IAM role is the execution role: the identity SageMaker AI uses to read the artifacts and pull the image.

One regional rule trips many migrations: the Amazon S3 bucket where the model artifacts are stored must be in the same AWS Region as the model that you are creating^[12]. Copying the artifacts to a bucket in the deployment Region fixes it. Granting broader S3 permissions does not, because the rule concerns where the bucket is, not who may read it.

Amazon Bedrock Custom Model Import

Custom Model Import creates a custom model in Amazon Bedrock from a foundation model that you customized in another environment, such as SageMaker AI^[15], and serves it with on-demand throughput through the InvokeModel or InvokeModelWithResponseStream operation^[15]. It imports weights; it is not a container host:

• The model must use one of the supported architectures^[15], which include Mistral, Mixtral, Flan, several Llama versions, GPTBigCode, Qwen families and GPT-OSS. Custom Model Import does not support embedding models^[15].
• From Amazon S3, you supply the model files in the Hugging Face weights format^[16], the file layout the Hugging Face transformers library produces. At minimum that means the weights in .safetensors files, config.json, tokenizer_config.json, tokenizer.json and tokenizer.model^[15].
• The import job runs as a service role that must be able to read those files, and a cross-account S3 bucket or AWS Key Management Service (AWS KMS) key needs access granted to the import^[16]. A correct file format does not substitute for permission to read or decrypt the source, and a role that can only invoke models cannot read it either.
• An imported model cannot be used with Bedrock batch inference or AWS CloudFormation^[15].

Uploading an inference Docker image does not work here, because the import expects model files of a supported architecture, not a serving container.

Amazon EKS for existing containerized services

When an existing Kubernetes deployment already packages each ML service as a standalone container image, the move to AWS is a lift-and-shift (moving the workload without redesigning it): push the images to Amazon ECR and deploy them to the Amazon EKS cluster. Pods must reference the full registry/repository:tag image name^[17], and the nodes need pull permissions: on managed or self-managed nodes the worker node IAM role needs Amazon ECR permissions, and on AWS Fargate the Fargate pod execution role provides them^[17]. Moving an image does not change the trained model inside it, so neither retraining nor a rewrite into another ML orchestration framework is part of the move.

Match the artifact you hold to the runtime that accepts it: framework artifacts to a SageMaker AI model whose artifact bucket is in the same Region^[12], supported foundation model weights to Custom Model Import^[15], and existing container images to Amazon EKS, pulled from Amazon ECR^[17].

For Amazon Bedrock's serverless model inference, you do not choose an instance; the decisions are how capacity is allocated and where a request may be processed. Bedrock serves a foundation model (FM, a large pre-trained model used through an API) through several capacity and routing options, among them those in the table below, and a custom model has its own path to on-demand serving.

Option	What you get	Fits when
On-demand inference	Usage-based access with no capacity to set up	Variable or moderate traffic
Provisioned Throughput	Dedicated capacity in model units at an hourly price	Consistent, high throughput for a supported model
Batch inference	An asynchronous job that reads prompt files from Amazon S3 and writes output files to S3	Large sets of independent prompts that need no immediate answer
Cross-Region inference profile	Bedrock picks a Region from the profile's set for each request	Higher throughput across Regions, where data location rules allow

Provisioned Throughput

Provisioned Throughput lets you provision a higher level of throughput for a model at a fixed cost, billed hourly^[18]. Capacity is bought in model units (MUs), and each MU delivers a specific number of input and output tokens per minute for the model^[18]. You choose no commitment, 1 month or 6 months; the longer the commitment, the more discounted the hourly price, and billing continues until you delete the Provisioned Throughput^[18]. Support differs by model and Region, so confirm the model is on the Provisioned Throughput supported list for your Region before planning on it. On-demand inference is the alternative whenever usage-based access without a capacity commitment meets the requirement.

Custom models: Provisioned Throughput or a custom model deployment

The Provisioned Throughput page states that if you customized a model, you must purchase Provisioned Throughput to use it^[18]. Models brought in with Custom Model Import (previous section) follow their own path: you can use a model that you import with on demand throughput^[15]. For a model created with a model customization job^[19], the Provisioned Throughput rule has one documented alternative: a custom model deployment sets up on-demand inference for a custom model, so you pay only for what you use^[19]. It requires a model customized on or after July 16, 2025 from a supported base model (Amazon Nova Lite, Nova 2 Lite, Nova Micro or Nova Pro in US East (N. Virginia), or Meta Llama 3.3 70B Instruct in US West (Oregon))^[19]. You invoke it by passing the deployment's Amazon Resource Name (ARN) as the modelId^[19].

The deployment and the model are separate resources. Deleting a custom model deployment ends its on-demand inference but does not delete the underlying custom model; to use the model on demand again, you create a new deployment^[19]. Deleting the custom model itself to stop one deployment removes the retained model, not just its serving resource.

Batch inference

Batch inference lets you submit multiple prompts and generate responses asynchronously: you upload input files to Amazon S3, submit a job that names the bucket, and retrieve the output files from S3^[20]. Each input record is processed independently without multi-turn interaction, so tool calling is not supported^[20]: an agent conversation that needs back-and-forth tool exchanges cannot run as a batch job. Batch inference isn't supported for provisioned models^[20], and it never reserves capacity for synchronous application requests.

Cross-Region inference profiles

An inference profile defines a foundation model and the AWS Regions to which requests can be routed^[21]. The system-defined cross-Region profiles come in two scopes: with a geographic profile Bedrock selects a commercial Region within that geography (such as US or EU), and with a global profile it selects any supported commercial Region^[21]. The application specifies the inference profile's ARN when calling InvokeModel, InvokeModelWithResponseStream, Converse or ConverseStream^[22] instead of managing separate regional calls, and there is no additional routing cost; the price is calculated from the Region in which you call the profile^[21]. Choose the geographic scope when data must stay within a geography. A single-Region model ID never produces cross-Region routing, and inference profiles currently don't support Provisioned Throughput^[21], so the two are separate ways to raise throughput, not a combination.

On Bedrock, the options covered here are pay per use, reserve units, batch the work, or widen the Regions; custom models add the deployment-versus-model distinction.

Some requirements need the opposite of Bedrock's serverless approach: control over the instances that serve a foundation model. AWS's decision guide states the split: Amazon Bedrock is serverless, meaning you do not have to manage infrastructure^[23], while SageMaker AI is the choice when you need to access the container or orchestration layer to directly manage cost-latency-throughput tradeoffs^[23]. Two paths put an FM on SageMaker AI hosting.

Amazon SageMaker JumpStart

SageMaker JumpStart is a model hub that offers built-in publicly available and proprietary foundation models^[23]. Deploying a JumpStart foundation model follows the SageMaker Studio model deployment steps to a SageMaker AI endpoint^[24], where you select the hosting compute and deployment settings yourself; JumpStart optimized deployments also offer preset configurations for a target use case with a cost, throughput, latency or balanced optimization^[24]. Some models require you to accept an end-user license agreement before deployment^[24].

Amazon Bedrock Marketplace

Bedrock Marketplace offers over 100 popular, emerging and specialized FMs in addition to Bedrock's own selection^[25], and it mixes the two platforms. After you subscribe to a catalog model, you deploy it to a SageMaker AI endpoint, providing the endpoint name, the number of instances and the instance type; the model is hosted by SageMaker AI^[26]. You then call it through Bedrock: the Converse or InvokeModel operation, with the endpoint ARN as the model ID^[27], with Converse available only for compatible models^[27]. Because the model is reached through Bedrock's APIs, it works with Bedrock tools such as Agents, Knowledge Bases and Guardrails^[25]. So a Marketplace model is not ordinary serverless Bedrock access: you pick and pay for its instances.

Marketplace and Custom Model Import are easy to mix up because both end with a non-native model reachable from Bedrock. A Marketplace subscription deploys a catalog model to SageMaker AI hosting; Custom Model Import brings in weights that you customized elsewhere. Neither activates the other.

When the requirement is to invoke an FM through an API without selecting serving infrastructure, choose Amazon Bedrock; when it names instance types or deployment settings, choose a SageMaker AI endpoint through JumpStart; when the model is offered only through Bedrock Marketplace, subscribe and deploy it to its SageMaker AI endpoint.

A team that moves its chatbot from one FM to another finds that each model's native request body, the format that InvokeModel takes, is different; the parameters for that body are listed per model in the inference parameters documentation^[28]. The Converse API exists to remove that rework.

Converse, and ConverseStream for streaming responses, provide a consistent API that works with all Amazon Bedrock models that support messages, so you can write code once and use it with different models^[29]. A request carries messages (the role and content of each turn), an optional system prompt, and inferenceConfig for the inference parameters common to all models^[29]; parameters unique to one model go in additionalModelRequestFields^[29]. The shared interface standardizes the request shape, not the models: a capability that one model lacks does not appear because the interface is shared. The same request can also carry toolConfig for tool use, described in the next section, and guardrailConfig to apply a guardrail.

Streaming and model independence are separate properties. InvokeModelWithResponseStream streams output too, but its request body still follows the selected model's own inference format; when an application needs both streaming and one message format across models, ConverseStream is the operation. Permissions follow the base operations: calling Converse requires permission for bedrock:InvokeModel, and ConverseStream requires bedrock:InvokeModelWithResponseStream^[29].

Use Converse for portable conversation code, ConverseStream when that code must stream, and keep model-specific options in additionalModelRequestFields.

A model sometimes needs information it does not have, such as today's most-played song on a radio station; tool use settles who fetches it. In Amazon Bedrock, the model doesn't directly call the tool^[30]: you supply definitions of the tools with the request, the model decides when one is needed, and something else runs it. Bedrock documents three modes of tool use^[30], which differ in who runs the tool:

Mode	Who runs the tool
Client-side tool use	Your application code, after the model returns a tool request; available with the Converse, InvokeModel, Responses and Chat Completions APIs
Server-side tool use	Amazon Bedrock, which invokes a registered Lambda function or AgentCore Gateway on the model's behalf; currently available on the Responses API
Anthropic Claude tool use	Your application code, using Anthropic-defined tool types and the Anthropic Messages request format

Bedrock also provides the OpenAI Responses API^[31] and the OpenAI Chat Completions API^[32]. AgentCore Gateway, which exposes existing backends as tools, is described under AgentCore Gateway: existing backends as tools.

With Converse, client-side tool use is a loop of five steps, which the figure below traces as an exchange between your application and the model:

1. The application sends the messages plus tool definitions (name, description, input schema) in toolConfig.
2. The model stops with the stop reason tool_use and returns a toolUse block naming the tool, its inputs and a toolUseId.
3. The application runs the tool itself.
4. The application sends a toolResult block that echoes the same toolUseId, so the model can match the result to the request^[28].
5. The model continues and generates its answer using the result.

The Converse example in the Bedrock documentation^[28] follows exactly this pattern: it checks stopReason for tool_use, calls the tool, and appends a user message whose content is the toolResult. Execution ownership is the point: in client-side mode your code runs the tool, and in a successful tool-use loop you send its result back so the model can continue its answer. The optional toolChoice field defaults to auto, in which the model decides whether to request a tool; any and tool force a tool request and are supported only by some models^[28].

This loop is a property of a direct model call. Configuring an agent's action groups, covered in the next section, is a different layer, in which the agent orchestrates the steps.

Amazon Bedrock Agents Classic is the current name of Amazon Bedrock Agents (launched November 2023); it is no longer open to new customers from July 30, 2026, and AWS recommends migrating its workloads to Amazon Bedrock AgentCore^[33]. Existing agents keep running, so their configuration still matters. This section covers the three capabilities you configure on an Agents Classic agent: action groups, knowledge base associations and code interpretation. One rule applies to every change: prepare the agent after changing its configuration, before you test it^[34]. Preparing, with Prepare in the console or the PrepareAgent operation, packages the agent with your latest changes to its working draft^[35].

Action groups: the definition is separate from the execution

An action group tells the agent which actions exist and how each one is carried out. You define the actions with an OpenAPI schema (a standard machine-readable API description) that describes each action as an API operation, or with function details that list the parameters the agent must elicit from the user^[36]. When the agent decides to invoke an action, it elicits the parameters it needs^[36] and hands them to the fulfillment option you configured, chiefly one of two:

• Lambda function: the agent passes the action and parameters to a Lambda function that you create to hold the business logic^[37].
• Return control: the agent skips Lambda and returns the information and parameters in the InvokeAgent response; your systems produce the result, which can go back in the session state of another InvokeAgent request^[37].

The schema describes the operation; the fulfillment option supplies its implementation. Return control and client-side tool use (previous section) both leave execution to your application, but at different layers: return control is a setting of an agent's action group, while client-side tool use belongs to a direct model call with toolConfig. An AgentCore Gateway tool definition is not an action group configuration either; Gateway exposes backends as MCP (Model Context Protocol) tools, as described under AgentCore Gateway: existing backends as tools.

Knowledge base association

Creating and syncing a knowledge base connects it to no agent. You associate the knowledge base with the agent, during or after agent creation, and provide instructions that tell the agent when and how to query it^[34]; the agent includes these instructions in its prompt context to decide whether to retrieve information from the knowledge base for a given user request^[34]. An action group schema cannot stand in for this association: it describes operations and their inputs, not a knowledge base.

Code interpretation

Code interpretation enables the agent to generate, run and troubleshoot code in a secure test environment^[38], so it can analyze and visualize data and extract information from files the user uploads^[38] and answer from the results of actual execution rather than from code text alone. You enable Code Interpreter in the agent builder, then save and prepare the agent to apply the change before testing^[39]. Return control is not a managed code executor: it sends the selected action and inputs back to your application to fulfill.

On an Agents Classic agent, the schema says what an action is, the fulfillment option says who runs it, a knowledge base needs an association plus instructions, and code interpretation is the managed way to execute generated code; prepare after each change.

A team has a working agent written with LangGraph that calls a model from another provider, and wants managed hosting without rewriting it. Amazon Bedrock AgentCore Runtime is built for this case: it is framework agnostic, working with frameworks such as LangGraph, Strands and CrewAI and with custom agents that use no specific framework^[40], and it works with any large language model, such as models offered by Amazon Bedrock, Anthropic Claude, Google Gemini and OpenAI^[40]. Rewriting a working agent into another framework is not a prerequisite for using it.

The service contract

What Runtime does require is its service contract: the standardized communication protocol that your agent application must implement to integrate with the hosting infrastructure^[41]. You select a protocol for the runtime, and the container must serve that protocol's endpoints and message format. The protocol comparison^[41] lists four:

Protocol	Port	Mount path	Message format	Discovery	Use case
HTTP	8080	/invocations, /ws	REST JSON or SSE, WebSocket	None	Direct API calls, real-time streaming
MCP	8000	/mcp	JSON-RPC	Tool listing	Tool servers
A2A	9000	/ (root)	JSON-RPC 2.0	Agent Cards	Agent-to-agent communication
AG-UI	8080	/invocations, /ws	Event streams (SSE or WebSocket)	None	Interactive UI experiences

Server-sent events (SSE) deliver real-time streaming responses^[42], while the WebSocket endpoint supports real-time bidirectional streaming^[42]; the MCP contract uses a JSON-RPC based request/response format^[43]. A container that merely starts is not enough. An application that exposes only an MCP server, deployed under the HTTP protocol, does not serve the /invocations interface the runtime then expects; configure the runtime protocol and the application's server consistently.

MCP or A2A

The two protocols that connect an agent to other software serve different roles, and Runtime lets agents communicate with tools and other agents through either^[40]:

• MCP (Model Context Protocol) is how a service exposes tools to agent clients. Its discovery mechanism is a tool listing.
• A2A (Agent-to-Agent protocol) is for multi-agent communication and discovery^[41], with Agent Cards as its discovery mechanism.

A tool server and a collaborating agent are different integration roles: when another agent must discover and work with yours, A2A is the contract, and an MCP tool listing does not substitute for it.

AgentCore Runtime accepts the agent you already have, whatever its framework and model, provided the container implements the selected protocol's contract.

An agent sits between two parties: the callers that invoke it and the tools that it invokes. Each side has its own authentication path, and the figure below shows both, with AgentCore Gateway, which converts APIs, Lambda functions and existing services into MCP-compatible tools^[44], on the tool side.

Inbound authentication: who may invoke the agent

AgentCore Runtime authenticates callers with IAM SigV4 (AWS Signature Version 4) by default, or with JWT (JSON Web Token) bearer tokens checked against an authorizer configuration: a discovery URL plus the allowed audiences, clients or scopes^[45]. That lets end users authenticate through a corporate identity provider such as Okta, Microsoft Entra ID or Amazon Cognito into only the agents they may use^[40]. A runtime supports either SigV4 or JWT inbound authentication, not both at the same time^[45].

Outbound authentication: what the agent may call

Outbound authentication lets the agent access third-party services such as Slack, Zoom and GitHub, either on behalf of users or autonomously, using OAuth or API keys^[40]. For delegated user access, the agent retrieves OAuth tokens on behalf of end users through the authorization code grant^[45]. Successful inbound authentication alone does not automatically authorize a downstream request; the agent still needs an appropriate credential for each tool that requires authentication^[46].

Inbound authentication answers who may call the agent; outbound authentication answers what the agent may call.

AgentCore Gateway converts APIs, Lambda functions and existing services into MCP-compatible tools^[44] and supports OpenAPI, Smithy and Lambda as tool input types^[44]. Its translation capability converts agent requests that use MCP into API requests and Lambda invocations^[44], so the backend keeps its business logic and the gateway only adapts access to it. Gateway also manages both inbound authentication (verifying agent identity) and outbound authentication (connecting to tools), including OAuth flows, token refresh and credential storage^[44]. Direction is named from each component's side: Gateway's inbound authentication checks the agent that calls it, which is that agent's outbound call in the Runtime terms of the previous section. Importing the API's documentation into a knowledge base is no substitute: it makes the descriptions retrievable but never executes the action.

Gateway turns existing backends into tools: with IAM or JWT inbound authentication configured, the agent authenticates to Gateway with its outbound credential, which Gateway's inbound authentication validates^[47], and Gateway then authenticates outbound to each backend (a Gateway target) that requires it^[46].

When an application asks an Amazon Bedrock knowledge base a question, it makes two choices: which operation to call, and how retrieval behaves for that query. These query-time choices fall under the AWS MLA-C02 exam guide skill 'Apply Retrieval Augmented Generation (RAG) system configurations (for example, retrieval strategies, reranking)'^[2]; RAG means adding retrieved passages to a model's prompt at request time.

Retrieve or RetrieveAndGenerate

• Retrieve queries the knowledge base and returns source chunks that are relevant to the query^[48]. Your application decides what to do with them, for example passing them to a model with its own prompt.
• RetrieveAndGenerate retrieves chunks and generates a response, returned with citations to the original source data^[49].

The choice decides whether the managed call owns only retrieval or the whole retrieve-and-generate path. Calling Retrieve and expecting a synthesized answer with citations asks the retrieval-only operation for a generation step it does not perform.

Both operations accept a retrievalConfiguration whose vectorSearchConfiguration holds the retrieval settings below^[50]. Those settings apply to a custom knowledge base, one where you configure and manage some of the underlying infrastructure^[51]. A fully managed knowledge base, where Amazon Bedrock manages the ingestion, storage, indexing, and retrieval infrastructure, including embedding and reranking with service-managed models by default^[51], uses managedSearchConfiguration instead, always uses hybrid search, and uses a service-managed reranker by default only with managed embedding^[48], meaning the service-managed embedding model rather than your own Bedrock embedding model chosen at creation time^[51].

Listing: a retrievalConfiguration fragment for a Retrieve request to a custom knowledge base

"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "numberOfResults": 10,
    "overrideSearchType": "HYBRID",
    "filter": { "equals": { "key": "department", "value": "finance" } },
    "rerankingConfiguration": { ... }
  }
}

The ... marks the omitted reranker settings (the reranker model's ARN among them). The keys are the fields of the vector search configuration^[52], each explained in the next subsection: numberOfResults caps the results, overrideSearchType sets the search type, filter restricts which documents may match, and rerankingConfiguration adds a reranker.

The settings, stage by stage

• Query decomposition. In RetrieveAndGenerate, query decomposition breaks a complex query into smaller subqueries, which may run multiple queries against the knowledge base^[50] before the final answer. It suits a question that combines several information needs. Raising the result limit is no substitute, because more results for a compound query still come from searching with the original query.
• Search type (overrideSearchType). Hybrid search combines searching vector embeddings (semantic search) with searching the raw text; semantic search only searches vector embeddings; if you set nothing, Bedrock decides^[50]. Hybrid search is only supported for Amazon RDS, Amazon OpenSearch Serverless and MongoDB vector stores that contain a filterable text field; otherwise the query uses semantic search^[50]. Verify the existing store before relying on raw-text matching.
• Explicit metadata filters (filter). A filter restricts which documents may satisfy the query, for example one department, a date range or a category. The metadata must exist and be supported: for an S3 data source the bucket needs at least one fileName.extension.metadata.json file for its documents, and an OpenSearch Serverless index must use the faiss engine^[50]. Telling the generation prompt to use only one department's documents does not constrain what retrieval returns.
• Implicit filtering. Bedrock can generate and apply a retrieval filter from the user query and a metadata schema^[50]. In implicitFilterConfiguration you supply metadataAttributes, schemas giving each attribute's key, type and description, plus the ARN of the model that maps user intent to those fields; implicit filtering is supported by Anthropic Claude models^[50]. Explicit filters remain the choice when the application already knows the filter values.
• Number of results (numberOfResults). A query returns up to five results by default, and numberOfResults sets the maximum^[50]. With hierarchical chunking (documents split into large parent chunks that contain smaller child chunks), the limit counts the child chunks retrieved, and child chunks that share a parent are returned as that parent chunk, so fewer results than requested can come back^[50]. That is retrieval assembling its response, not an ingestion failure.
• Reranking (rerankingConfiguration). A reranker model calculates the relevance of chunks to a query and reorders the results^[53], and its ranking overrides the default ranking of the knowledge base^[53]. It works only on text^[53] and only reorders candidates that search returned, so it cannot recover a keyword match the search missed; lowering generation temperature changes response sampling, not ranking.
• Guardrail. For RetrieveAndGenerate, set guardrailConfiguration with a guardrailId and guardrailVersion inside generationConfiguration^[50]. Guardrails apply to the input and the generated response, not to the references retrieved from the knowledge base^[49]. Safety wording in the prompt does not enable the managed guardrail controls; confirm the chosen model supports the guardrail features you need.

The figure below orders these settings along the query path; query decomposition, like generation and its guardrail, applies only to RetrieveAndGenerate^[50].

Each setting owns one stage: decomposition shapes the query, search type and filters decide what can match, the result limit caps how much comes back, reranking orders it, and the guardrail checks the generated answer.

The AWS MLA-C02 exam guide lists 'Task 3.1: Manage deployment infrastructure for ML and AI model types.'^[2], with skills that include 'Select deployment orchestrators and multi-model or multi-container deployment strategies' and 'Evaluate and select appropriate foundation model (FM) deployment options'; the patterns below fall under that task, each with its section named in italics; for each requirement, the feature to choose is the one whose mechanism meets it. Task 3.1 also lists the skills 'Select model inference strategies (for example, real-time and batch processing)', 'Deploy models that were built outside of AWS into AWS environments (for example, Amazon SageMaker AI, Amazon Bedrock Custom Model Import)', 'Deploy and configure agents for specific tasks, integration with other services and tools, and agent communication protocols' and 'Apply Retrieval Augmented Generation (RAG) system configurations (for example, retrieval strategies, reranking)' (AWS MLA-C02 exam guide, domain 3^[2]).

Inference options and shared endpoints

• Requirement: interactive request; proposed: batch transform. Batch transform returns output files in Amazon S3 when its job finishes, so it cannot answer a transaction synchronously; use real-time inference (Four inference options).
• Requirement: intermittent traffic with large payloads; proposed: Serverless Inference. Serverless Inference does not provide the S3 request-and-result workflow and caps payloads at 4 MB; large inputs that can wait need Asynchronous Inference (Asynchronous Inference).
• Requirement: bursty traffic plus a GPU or VPC configuration; proposed: Serverless Inference. Serverless Inference excludes GPUs and VPC configuration, and Provisioned Concurrency only warms serverless capacity; use real-time inference (Four inference options).
• Requirement: independent tenant models; proposed: inference pipeline. An inference pipeline chains stages for every request; per-tenant models on one framework can use a multi-model endpoint (Sharing one endpoint).
• Requirement: preprocessing chain; proposed: direct invocation. Direct invocation runs only the container a request names; a chain needs an inference pipeline (Sharing one endpoint).
• Requirement: independent models in separate framework-specific containers; proposed: multi-model endpoint. A multi-model endpoint serves every model from one shared serving container^[6], so models that each need their own framework container fit a multi-container endpoint with direct invocation; a GPU-backed MME on Triton^[54] is the multi-framework exception (Sharing one endpoint).
• Requirement: GPU memory per model; proposed: traffic weights. Production-variant weights route traffic; inference components declare each model's accelerator and memory needs (Sharing one endpoint).
• Requirement: new tenant model; proposed: new production variant. A multi-model endpoint loads a compatible model uploaded under its S3 prefix on first invocation, with no endpoint update (How a multi-model endpoint loads).

External models and foundation models

• Requirement: deploy artifacts from a cross-Region bucket; proposed: broader S3 permissions. The artifacts must sit in the deployment Region; permissions do not change location (Hosting a model trained outside AWS).
• Requirement: Custom Model Import; proposed: a Docker image as its input. The import takes Hugging Face format weights of a supported architecture, not containers (Hosting a model trained outside AWS).
• Requirement: a role for the import job; proposed: invoke permissions only. Invocation permission does not let the job read or decrypt the source files (Hosting a model trained outside AWS).
• Requirement: reserved capacity for synchronous traffic; proposed: Bedrock batch inference. Batch inference creates asynchronous S3 jobs; dedicated online capacity is Provisioned Throughput (Running foundation models on Amazon Bedrock).
• Requirement: multi-Region routing; proposed: a plain model ID. Only a cross-Region inference profile routes across Regions (Running foundation models on Amazon Bedrock).
• Requirement: deploy a Marketplace model; proposed: Custom Model Import. A catalog subscription deploys to a SageMaker AI endpoint; import is for weights customized elsewhere (Foundation models on SageMaker AI hosting).
• Requirement: portable streaming chat code; proposed: InvokeModelWithResponseStream. It streams, but its body follows each model's own format; use ConverseStream (Converse).

Agents and retrieval

• Requirement: run an existing agent on AgentCore Runtime; proposed: rewrite the agent. Runtime is framework agnostic; implement the protocol contract instead (Hosting agent code on AgentCore Runtime).
• Requirement: deploy an MCP-only server; proposed: the HTTP protocol. The runtime protocol and the server must match (Hosting agent code on AgentCore Runtime).
• Requirement: agent discovery; proposed: an MCP tool listing. Agent discovery and communication is A2A, with Agent Cards (Hosting agent code on AgentCore Runtime).
• Requirement: tool authorization; proposed: the inbound session. The agent needs outbound credentials for tools that require authentication (Inbound and outbound authentication).
• Requirement: an API as an agent tool; proposed: its documentation in a knowledge base. Retrieval returns text; AgentCore Gateway executes the backend as an MCP tool (AgentCore Gateway).
• Requirement: an agent that consults a knowledge base; proposed: creating the knowledge base alone. It needs an association plus instructions, then a prepared agent (Configuring an Agents Classic agent).
• Requirement: a code executor; proposed: return control. Return control hands execution to your application; code interpretation runs generated code in a secure environment (Configuring an Agents Classic agent).
• Requirement: recover missing keyword matches; proposed: a reranker. A reranker reorders only what search returned; enable hybrid search on a supported store (Configuring a knowledge base query).
• Requirement: department restriction; proposed: writing it into the prompt. A metadata filter restricts which documents may satisfy the query (Configuring a knowledge base query).
• Requirement: compound question; proposed: higher result limit. More results still come from the original query; enable query decomposition (Configuring a knowledge base query).
• Requirement: explain fewer results than numberOfResults with hierarchical chunking; proposed: an ingestion failure. Child chunks that share a parent come back as that parent; fewer results alone do not indicate an ingestion failure (Configuring a knowledge base query).

Every Task 3.1 skill^[2] begins with 'Select', 'Evaluate and select', 'Deploy', 'Configure' or 'Apply', and across all three groups the feature to choose is the one whose mechanism meets the stated requirement.