Choosing Foundry Services for Generative AI and Agents

A requirement lands on your desk: "summarize 40,000 support tickets a night, extract the product code from each one into a field a reviewer can spot-check, and let an agent answer follow-up questions from the internal knowledge base." Nothing in that sentence tells you how to provision a resource or wire a pipeline. It tells you which building blocks to pick, and picking them is what this objective tests. The sibling pages own the rest of the lifecycle: Setting Up AI Solutions in Foundry provisions the resource, projects, and deployments; Managing, Monitoring, and Securing AI Systems runs them; this page decides what goes inside them.

Selection breaks into four decisions, and the figure below shows them in the order a question usually walks. First, which model class fits the task. Second, whether a prebuilt Foundry Tool replaces the model call entirely; Foundry Tools are the ready-made AI capabilities such as Content Understanding and Content Safety, and the fourth section below covers when one beats a model. Third, how content is indexed and retrieved. Fourth, which tools and memory an agent attaches. Microsoft Foundry is the platform all four decisions happen inside, and one Foundry resource fronts the model catalog^[1], the prebuilt Foundry Tools^[2], and Foundry Agent Service.

The order matters because each decision narrows the next. A question that has already told you the output must be schema-bound JSON has settled decision two, which makes half the model-class options irrelevant. Read the requirement for which decision it is actually asking about, and the distractors usually belong to a different one.

Fix the class before you compare individual models. The Foundry catalog carries thousands of entries and the leaderboard only helps once you know which shelf you are shopping on, so the useful first question is what shape the task has, not which model scores highest. The figure below lays out the five classes an AI-103 question realistically chooses between and the signal that selects each one.

A chat model is the general-purpose default for assistant tasks, drafting, and question answering. A reasoning model generates intermediate reasoning tokens and summarizes them before it returns the first response token, so a streaming request can sit silent while the model thinks and client-side timeouts have to be set far higher than for a chat model; you control how much of that work happens with the reasoning effort parameter^[3], not with sampling parameters. That last point is worth pinning down because it is a predictable misread: temperature and top_p change how randomly the model picks its next token and nothing else. Raising them does not add a reasoning pass, and no amount of sampling tuning turns a chat model into a reasoning model.

A small language model (SLM) is a compact generative model, typically from under 1 billion to around 14 billion parameters, where Microsoft's Phi family is the canonical example, against large language models with hundreds of billions. Compared with an LLM it needs far fewer resources^[4], running on a single GPU or even a CPU, gives lower latency and higher throughput per device, and can run on-premises or on-device, while trading away some breadth of general knowledge. That profile makes an SLM the right selection for narrow, well-specified, high-volume work: classification, intent routing, field extraction, short summarization. A frontier LLM is the right selection when the task needs broad world knowledge or long multi-step reasoning.

A multimodal model is the pick when the ask is open-ended interpretation of an image or document: captioning, describing, answering questions about what is in the picture. And code models are a class of their own: the catalog carries Codex-family reasoning models^[5] tuned for agentic software work such as reading and refactoring files across a repository, writing tests, and opening pull requests. Selecting one of those is a decision about which model authors the code, which is a different axis from the Code Interpreter tool that runs code a model already wrote.

Once the class is fixed, two mechanisms rank and route within it. Model leaderboards^[1] in the Foundry portal compare curated catalog models on quality (reasoning, knowledge, question answering, math, and coding), safety, performance measured as latency and throughput, and cost, with scenario leaderboards for specific use cases such as coding or math. Microsoft is explicit that these are standardized comparisons over public datasets and directs you to re-evaluate shortlisted models against your own data, so a leaderboard rank is a shortlist, not a fitness proof. Model router^[6] is the other mechanism: it is itself a deployed model that analyzes each prompt and routes it to one of its supported underlying models, giving you a single deployment and chat surface instead of many. Its routing mode sets the trade-off, with Balanced (the default) picking the cheapest model within a narrow quality band of the best model for that prompt, Cost widening that band for more savings, and Quality always taking the highest-rated model. Model router is a trained routing model inside one deployment, not a gateway or load balancer across resources, and its effective context window is capped by the smallest model in its subset.

So the working order is: read the requirement for a shape signal (volume and latency, multi-step logic, an image, source code), pick the class that signal names, and only then argue about which specific model.

Three mechanisms change what a model produces, and Microsoft documents them as complementary rather than competing^[7]. Prompt engineering carries per-request instructions such as language or support tier. Retrieval-augmented generation (RAG) injects retrieved passages at inference time, so refreshed content is reflected as soon as the index updates and nothing is retrained. Fine-tuning adapts behavior, style, task performance, and output format. A single scenario often needs all three, so a question asking you to choose one is really asking which mechanism owns the thing that has to change.

The split is easiest to hold as a question about what changes and how often, which is the pairing the figure below sets out for each mechanism. Volatile facts change per transaction, and weights cannot be updated per transaction, so fine-tuning a model on a daily-changing catalog produces a model that states stale detail confidently while you pay repeatedly for training and hosting. Behavior and format are stable properties of the task, which is exactly what training can encode. Instructions vary per conversation, which is what a system prompt is for.

Picking the fine-tuning technique

When fine-tuning genuinely is the answer, the training data you can actually supply selects the technique. Enumerate the three before reaching for any one of them:

Technique	Training data you must supply	Fits
Supervised fine-tuning (SFT)[8]	Labeled prompt and completion (or conversational) examples	Task specialization, output format, instruction following; problems with finite correct solutions
Direct preference optimization (DPO)[9]	Preferred and non-preferred responses supplied as pairs	Subjective qualities: tone, style, content preference, with no separate reward model to fit
Reinforcement fine-tuning (RFT)[10]	A grader that scores each response, instead of example outputs	Complex reasoning with many solution paths but checkable answers

Foundry also documents stacking the techniques, running supervised fine-tuning first and DPO afterwards, since preference fine-tuning works on base models and on models already fine-tuned with SFT. Whichever you use, the result is a separate custom model that has to be deployed to its own endpoint before any client can call it, and Microsoft is explicit that fine-tuning carries upfront costs for training the model plus additional hourly costs for hosting the custom model once it is deployed, on top of the usual per-token inference. That standing cost is often the deciding factor when retrieval would have solved the problem instead.

Match the mechanism to what has to change, and the answer falls out: facts that move point at retrieval, behavior and format point at fine-tuning, and per-conversation instructions belong in the prompt.

Some requirements name an output a general model cannot produce, and those are the ones that point at a prebuilt tool instead. Foundry Tools^[2] are the prebuilt AI capabilities, including Content Understanding, Speech, Translator, Document Intelligence, and Content Safety, and Microsoft documents them as available as part of the Microsoft Foundry resource^[11] rather than as separate accounts you provision, connect, and secure one by one. Consolidating them behind that one resource is why provisioning a separate account per capability, and then wiring per-service keys, is the wrong answer to a "reduce the identity and networking work" requirement.

Three signals in a stem reliably select a tool over a model call.

Named fields with a confidence score. A vision-capable chat model is right for open-ended interpretation. When the output has to be named business fields in structured JSON with a numeric per-field confidence you can threshold for human review, the requirement points at a schema-driven analyzer. Microsoft's own tool-selection guidance^[12] makes the gap explicit: the managed services provide confidence and grounding, while a build-your-own solution on Foundry models does not, so "with no confidence scores, you either accept all results or review all results." A chat completion returns no calibrated per-field confidence, so there is nothing to compare against a 0.80 review threshold.

Deterministic, repeatable output at volume. Azure Translator^[13] is neural machine translation across a large language set through a single managed call, producing consistent output run to run at a low per-character cost, which suits high-volume plain copy. An LLM translation flow is the right pick only when tone, domain context, or nuance must be carried, because it costs more per token, adds latency, and can render the same source differently between runs.

Moderation of content the model never sees. This is the sharpest boundary on the page, and the figure below traces it. Foundry guardrails apply at model and agent intervention points^[14], so they can only inspect prompts, tool traffic, and completions that flow through a Foundry deployment. Content that must be screened before it ever reaches a model, such as a user upload being stored or routed to a human reviewer, requires a direct call to the Azure AI Content Safety^[15] analyze APIs, which return per-category severity you compare against your own threshold. If the payload does not traverse an intervention point, no guardrail fires, however carefully it was configured.

Four signals, then, each select a Foundry Tool: schema-bound fields, a confidence threshold, deterministic output, or moderation off the model path. Open-ended interpretation is the one that selects a model.

Before an agent can ground on your content, that content has to become reachable, and there are three sanctioned ways to make it so. Two of them build a search index and one deliberately does not. The figure below shows all three as parallel lanes, because they are alternatives rather than stages of one pipeline.

The pull model attaches an indexer to a supported data source object and automates crawling, change detection, and skillset execution on a schedule^[16]. The smallest interval a schedule allows is five minutes, which means index freshness is bounded by that run interval: content written 30 seconds after a run waits for the next one. In exchange you get the enrichment machinery, because a skillset, the chain of AI enrichment steps that run over each document as it is indexed, attaches to an indexer rather than standing on its own.

The push model calls the Index Documents REST API or an SDK client from your own code, submitting per-document upload, merge, mergeOrUpload, and delete actions. Microsoft documents no restriction on data source type and no restriction on execution frequency for this path, which makes push the only option when content originates outside the supported sources (an on-premises line-of-business system, for instance) or when the index has to stay in sync faster than any schedule allows. The price is that skillsets do not run independently of an indexer, so AI enrichment and integrated vectorization are unavailable on the push path and your code has to produce the chunks, the vectors, and the deletes itself. Those two facts together are how a question is usually built: an unsupported origin or a near-real-time freshness requirement forces push, and the correct answer accepts the extra work.

The third lane skips the index. A knowledge source^[17] defines the content a retrieval pipeline draws on, and it comes in two kinds. An indexed knowledge source is backed by a search index you host on the Azure AI Search service, so content must be ingested first through pull or push. A remote knowledge source retrieves content from an external platform at query time, which avoids an ingestion pipeline altogether but makes freshness and permissions the source system's responsibility. Not every knowledge source needs its own indexer, because a remote source is queried live and never populates a local index.

Read a stem for the origin and the freshness bound, then: supported source plus a tolerable schedule points at pull; an unsupported source or sub-schedule freshness points at push and its manual chunking and vectors; content you would rather query in place points at a remote knowledge source.

Ingestion decides what can be found; query planning decides how hard the system works to find it. Classic RAG sends one query to the index and leaves planning and the handoff to the language model to your application code. Agentic retrieval^[18] moves that orchestration into the service: your application calls a knowledge base with a retrieve action, and a language model decomposes the request, using the conversation history, into focused subqueries that execute simultaneously against the knowledge sources, are each semantically reranked, and are merged into unified grounding data with optional references and an activity log. The figure below walks that pipeline left to right.

The knob that decides whether any of that planning happens is retrieval reasoning effort^[19], configured on the knowledge base and defaulting to low. Two different settings on this page share the words reasoning effort, so keep them apart: the one in the model-class section is a per-request parameter on a reasoning model that governs how much the model thinks, while this one is a property of an Azure AI Search knowledge base that governs how much query planning the retrieval pipeline does before it searches. At low and medium effort the knowledge base sends the query and the conversation history to a language model to generate subqueries. At minimal effort that planning step is skipped entirely and queries go straight to the knowledge sources, which is the bypass drawn under the pipeline in the figure. Lowering effort therefore cuts token spend and latency at the cost of decomposition quality, and it is easy to underestimate what is lost: with planning bypassed there is no query rewriting and no history-aware decomposition, so layered follow-up questions such as "and what about the enterprise tier?" degrade badly.

One piece of vocabulary is worth debunking at first contact, because it looks like a third option and is not. Iterative search is not a selectable pattern alongside agentic retrieval and classic RAG; it is an internal follow-up pass inside agentic retrieval at medium reasoning effort, and because it is sequential it adds latency rather than the parallelism that makes agentic retrieval fast.

Multi-turn conversations with compound questions justify agentic retrieval at low or medium effort; single-shot lookups where your code already owns orchestration do not, and minimal effort is the setting you choose when you have decided decomposition is not worth paying for.

Four grounding options compete for the same slot in an agent definition, and the deciding question is almost never which retrieval algorithm is better. It is who owns the corpus and whose identity reads it. The comparison table in the orientation above lays the four out criterion by criterion and the decision tree walks them as a sequence of questions; this section explains why each criterion decides what it decides, which is why it carries no third figure of its own.

File Search^[20] augments an agent with knowledge from files that developers or end users upload, backed by vector stores, the embedding indexes the agent builds from those uploads and searches at query time. That is exactly right when users attach their own documents to a conversation and exactly wrong when the corpus has to stay under your governance, because index schema, enrichment, and refresh are not yours to control when the agent owns the store.

The Azure AI Search tool grounds the agent on an existing index on your own search service. Reach for it when the corpus already has an ingestion pipeline, enrichment skills, a semantic configuration, and a permission model, because all of that survives; the agent becomes one more consumer of an index you already run.

The SharePoint tool^[21] and the Microsoft Fabric data agent tool^[22] are the pair that answers a permission requirement. SharePoint grounds an agent on documents in a connected site or folder using the Microsoft 365 retrieval stack, so you export nothing, build no semantic index, and manage no refresh logic. The Fabric tool answers questions over governed structured data by calling a published Fabric data agent over its lakehouse, warehouse, KQL database, or Power BI semantic model sources. Both run under identity passthrough, on-behalf-of the signed-in user: the retrieval or query executes as that user, so every end user must already hold read access to the SharePoint site or to the Fabric data agent and its underlying sources, app-only or service-principal authentication is unsupported, and the data source and the Foundry project must be in the same tenant. Building your own File Search store or private index over content that already lives in SharePoint or Fabric duplicates an ingestion pipeline and throws away the per-user trimming these tools give for free.

Web grounding is the fourth option and the one with the hardest boundary. The Web search tool and the Grounding with Bing Search tools bring real-time public web results with citations into an answer, which fits questions about current public information. Grounding with Bing Search retrieves from the web at large rather than from specific domains; to narrow results you use Grounding with Bing Custom Search, or Web Search's own allow and block lists. All of them reach only public, Bing-indexed content, and they carry no tenant permission model, so an internal catalog, policy library, or ticket history cannot be answered this way at all.

Reduced to four signals: uploaded files point at File Search, an index you already run points at the Azure AI Search tool, per-user permission trimming over tenant content points at SharePoint or Fabric, and current public facts point at web grounding. Nothing else about the four is a tie-breaker.

An agent's tools sort cleanly by one question: who runs the code. Foundry Agent Service documents two categories^[20], built-in and custom, and function calling sits deliberately outside both. The figure below groups them by that execution locus, and reading a stem for who executes is usually enough to eliminate two distractors.

Built-in tools are preconfigured capabilities the service executes for you after basic configuration, with no external hosting and no custom code. Web search, Code Interpreter, File Search, Azure AI Search, and Azure Functions are the commonly-used ones, among others in the catalog such as Image Generation and the SharePoint and Fabric tools discussed above. Code Interpreter deserves one clarification because its name invites a misread: it gives whatever model you deployed a sandbox in which to run Python that model has written, for data analysis, computation, and charts over attached files. It does not change which model authors the code, so a "raise the quality of generated code" or "refactor across the repository" requirement is answered by selecting a code model, not by attaching Code Interpreter to a general chat model.

Custom tools point the agent at something you supply. The Model Context Protocol (MCP) tool connects an agent to tools hosted on an MCP server endpoint, which Microsoft documents as the fit when the tools are shared across multiple agents or maintained by a different team. The OpenAPI tool connects the agent to one external HTTP API described by an OpenAPI 3.0 or 3.1 specification, with anonymous, API key, and managed identity authentication defined inside the tool definition. Agent-to-Agent endpoints connect an agent to other agents. A toolbox sits alongside these as an aggregator rather than a fourth kind: it is a curated bundle of tools configured once and exposed as a single MCP-compatible endpoint, so any MCP-capable runtime can consume it instead of every agent definition attaching each tool. It handles credential injection, token refresh, and policy enforcement centrally with Microsoft Entra ID and OAuth, and it supports versions, with agents bound to the consumer endpoint picking up a promoted default without code changes.

Function calling is the third column of the figure because your application, not the service, executes it: the agent proposes the call, your code runs the function and returns the result. That is the distinction behind a recurring trap. The Azure Functions tool has the service invoke a deployed Azure Function; function calling has your own process run the code. Same word, different executor.

Three executors, then: the service runs built-in tools, you host or describe custom tools, and your application runs function calls. A toolbox does not change who executes anything, it changes how many endpoints an agent has to know about.

"Remember that I prefer German and that I always order for the Berlin office" is a requirement about time, and it separates two different stores. A conversation is the raw transcript layer: it is a durable object that stores items rather than only chat messages, capturing messages, tool calls, and tool outputs, so the next turn can reuse that context verbatim. Memory in Foundry Agent Service^[23] is a separate managed long-term store that extracts, consolidates, and later retrieves durable knowledge so an agent stays continuous across sessions, devices, and workflows. The figure below shows the flow from session items through extraction into the store, and then out again at retrieval time.

Memory is not raw transcript storage, and expecting it to be is the most common misreading. The service uses a model to extract and consolidate, merging duplicates and resolving conflicting facts, so what comes back is distilled rather than verbatim. It arrives in three types, each with its own retrieval moment: user profile memory holds durable preferences and personal context such as language or accessibility needs and should be retrieved near the start of a conversation to establish personalization; chat summary memory holds distilled summaries of prior topics and is retrieved per turn for continuity; procedural memory holds reusable how-to routines inferred from past interactions and is retrieved when the user asks for a recurring workflow.

There are two ways to reach the store. Attaching the memory search tool^[24] to a prompt agent lets it read from and write to the memory store during conversations, and that is the recommended default. The low-level Memory Store APIs are for advanced cases that need direct control of individual memory records, a store-level default time to live, and explicit remember-or-forget behavior. The trade is scope handling: automatic scope resolution, bound to the user ID, is supported through the memory search tool, while the low-level APIs require you to set scope explicitly on every request rather than inferring it from the caller.

One related distinction belongs here because it shares the word session. Reusing the same agent session identifier does not replay prior turns. With the Responses protocol, continuity comes from previous_response_id or a conversation identifier; the session identifier alone does not resend earlier messages to the model.

Within a single exchange, conversation items carry the context verbatim; across sessions, devices, and workflows, the distilled memory store does, and the memory search tool is how you use it unless you specifically need record-level lifecycle control.

Most questions on this objective hide one distinction in the stem. Find it, and the answer follows.

• "Thousands of calls per hour", "on the device", "single GPU", "classification" points at a small language model. The distractor is the largest frontier model, justified by "quality matters"; on a focused task the small model wins on cost and latency without losing the task.
• "Multi-step logic", "plan then verify", "hard math or code" points at a reasoning model with its reasoning effort set. The distractor raises temperature or top_p, which only changes token-selection randomness.
• "Mixed prompt difficulty on one endpoint" points at model router. The distractor calls it a gateway or load balancer across resources; it is a trained routing model inside one deployment, and its effective context window is capped by the smallest model in its subset.
• "Named fields", "structured JSON", "route below 0.80 for human review" points at a schema-driven analyzer. The distractor is a multimodal model, which returns no calibrated per-field confidence to threshold.
• "Refactor across the repository", "improve the generated code" points at selecting a code model. The distractor attaches Code Interpreter, which only executes Python the deployed model already wrote.
• "Screen the upload before it is stored or sent to a reviewer" points at a direct Content Safety analyze call. The distractor is a deployment guardrail, which never sees content that does not cross an intervention point.
• "Catalog changes daily" points at retrieval. The distractor fine-tunes, which cannot update weights per transaction.
• "Match a house tone" or "preferred versus rejected answers" points at direct preference optimization; "many valid solution paths but a checkable answer" points at reinforcement fine-tuning with a grader; everything else formatting- or task-shaped points at supervised fine-tuning.
• "Source is an on-premises system" or "index must reflect changes within seconds" points at the push path with the Index Documents API, and the correct answer accepts that your code produces the chunks, the vectors, and the deletes.
• "Compound follow-up questions over several turns" points at agentic retrieval. If the stem instead says token spend must drop and decomposition is expendable, it points at minimal reasoning effort.
• "Each user must only see what they already have access to", with content in SharePoint or Fabric points at those tools and their on-behalf-of passthrough. The distractor builds a private index, which discards the permission trimming and cannot use app-only authentication anyway.
• "Current public information with citations" points at web grounding; the same stem with an internal corpus points anywhere but web grounding, which reaches only public, Bing-indexed content.
• "Tools maintained by another team" or "shared across agents" points at the MCP tool; "one external HTTP API with a specification" points at the OpenAPI tool; "our own process must run the code" points at function calling rather than the Azure Functions tool.
• "Remember the user next week" points at the long-term memory store through the memory search tool. The distractor reuses a session identifier, which does not replay prior turns.