Domain 2 of 5 · Chapter 3 of 3

Optimize and Operationalize Generative AI Systems

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing AI-103 premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • What operationalizing adds to a working prototype
  • Prompt structure: durable contract, data, request
  • Inference parameters and the two model families
  • Evaluators that isolate the failing stage
  • Calibrating an LLM judge against human labels
  • Automating the loop with the agent optimizer
  • Three ways traces start flowing
  • Reading spans and the Observability dashboard
  • Two caches, and what each one calls a match
  • Streaming and where the content filter sits
  • Model router: one deployment, disclosed choice
  • Detecting drift that adaptive alerts absorb
  • Deciding what the model must not decide
  • Exam-pattern recognition

The three lever families, and what each one actually moves

QuestionPrompt leversInference leversServing-path levers
What you changeInstructions, delimiters around untrusted content, few-shot examples, the abstention ruletemperature or top_p, reasoning_effort, verbosity, max output tokens, streaming modePrompt-cache prefix layout, semantic cache threshold, model router deployment, rules-engine split
Symptom it addressesFabrication, ignored constraints, unstable output shapeAnswers too long or too slow, truncation, non-deterministic extractionCost per request, repeated identical work, one premium model serving every prompt
Where it is configuredThe agent definition and the message arrayRequest parameters, per callThe deployment, the gateway policy, or your application code
Evidence that it workedGroundedness and Relevance scores on the same fixed datasetToken counts and latency percentiles read from tracescached_tokens in the response, per-model split in Azure Monitor, cost analysis
What it cannot fixA retrieval defect, which Document Retrieval measures separatelyAn ambiguous rubric or an unaligned judgeA quality regression caused by routing to a weaker model

Decision tree

Symptom measured on afixed dataset?Measure firstevaluators, traces, dashboardnoIs the complaint aboutanswer quality?yesDo you have retrievalrelevance labels?qualityDo requests share abyte-identical prefix?cost or latencyDocument Retrievallabels, no judge modelyesGroundedness gateadd the abstention rulenoPrompt cachingstable prefix firstyesSimilar in meaning,not in bytes?noSemantic cachetighten the thresholdyesOne model servingevery prompt?noModel routerBalanced, Cost, QualityyesInference leverseffort, verbositynoAlways: change one lever, then re-score the same dataset
Cheat sheet

Cheat sheet

  • Untrusted document or user content must be delimited and declared as data, never concatenated with instructions
  • System instructions hold the durable role and output contract while the user turn carries the request
  • Explicit step-by-step prompting is for non-reasoning models; reasoning models are steered with reasoning_effort
  • Instructing the model to answer only from supplied context and otherwise abstain raises measured groundedness
  • temperature and top_p both widen sampling, so tune one and leave the other at its default
  • reasoning_effort accepts minimal, low, medium, and high, trading latency and tokens for reasoning depth
  • Parallel tool calls are not supported when reasoning_effort is set to minimal
  • The verbosity parameter controls answer length independently of reasoning depth
  • Reasoning tokens count against the output token budget even though they are never returned
  • The Rubric evaluator scores against custom weighted criteria and returns a normalized score with per-dimension reasoning
  • A judge that correlates weakly with human labels needs rubric repair, not acceptance of the offset
  • A self-critique pass buys lower fabrication at the cost of tokens and latency, and is not an independent check
  • The agent optimizer baselines a hosted agent, generates candidate instructions, and re-scores them on the same dataset
  • Connecting an Application Insights resource to the project turns on server-side tracing with no code change
  • Client-side GenAI tracing is opt-in through environment variables, and message-content capture is a development-only setting
  • Distributed traces break a run into model, tool, and decision spans so latency can be attributed to a stage
  • Hosted agent containers receive an injected Application Insights connection string and emit traces by default
  • Foundry tracing covers LangChain, LangGraph, the OpenAI Agents SDK, and Microsoft Agent Framework
  • Prompt caching needs at least 1,024 tokens and a byte-identical leading prefix to hit
  • Dynamic-threshold alerts adapt to gradual drift and never fire, so drift needs a rolling baseline comparison
  • A semantic cache threshold set too low serves one stored answer to queries with different intent
  • The Foundry Observability dashboard reports token consumption, latency, error rate, and quality scores per agent
  • Streaming improves time-to-first-token, not total tokens or duration, and the default content filter still buffers a streamed response into vetted chunks
  • Model router deploys like any model and discloses which underlying model served each request
  • Routing mode is Balanced, Cost, or Quality, and each widens or ignores the quality band differently
  • A router deployment's effective context window is that of its smallest underlying model
  • Automatic failover is on by default, and on a custom deployment the model subset is also the fallback set
  • Deterministic policy steps belong in a rules engine, with the model handling only the ambiguous language

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Observability in Generative AI
  2. Prompt engineering techniques
  3. Azure OpenAI reasoning models: reasoning effort, verbosity, and reasoning tokens
  4. Structured outputs with Azure OpenAI
  5. Set the retrieval reasoning effort (Azure AI Search agentic retrieval)
  6. Retrieval-Augmented Generation (RAG) evaluators for generative AI
  7. Rubric evaluators (preview)
  8. Agent evaluators reference
  9. AI agent orchestration patterns Well-Architected
  10. Agent optimizer in Foundry Agent Service overview (preview)
  11. Optimize agent instructions, skills, tools, and models (preview)
  12. Set up tracing for AI agents in Microsoft Foundry
  13. Add client-side tracing to Foundry agents
  14. Export hosted agent telemetry by using OpenTelemetry
  15. Configure tracing for AI agent frameworks
  16. Prompt caching with Azure OpenAI in Microsoft Foundry Models
  17. API Management policy reference: llm-semantic-cache-lookup
  18. Enable semantic caching for LLM APIs in Azure API Management
  19. Content streaming in Azure OpenAI: default and Asynchronous Filter
  20. Model router for Microsoft Foundry concepts
  21. How to use model router for Microsoft Foundry
  22. Azure Monitor metric alerts with dynamic thresholds
  23. AI agent adoption guidance for organizations (Cloud Adoption Framework) Well-Architected