Implement generative AI quality assurance and observability

A classical machine learning model outputs a number you can grade against a label; a generative application outputs open-ended text, so 'is it any good?' and 'is it safe?' stop being obvious and become measurement problems of their own. At 10 to 15 percent this is a comparatively light domain, and it hangs on one model: Microsoft Foundry frames generative AI observability as three pillars that share a single telemetry backend, Azure Monitor Application Insights.Observability in Microsoft Foundry The pillars are evaluation (score output quality and safety against a reference you choose), monitoring (production dashboards and alerts on both answer quality and operational health), and tracing (the step-by-step record of one request's execution). Name the pillar and the tool follows; the classic exam trap is matching a symptom to the wrong one, since 'responses are slow' is an operational-monitoring signal (latency, throughput, error rate), 'responses are confidently wrong or unsafe' is an evaluation score, and 'why did this one request fail?' is a tracing question.

Read this page as a map, then follow the two subtopics in order; the three pillars split across them, with evaluation on its own page and the two production-runtime pillars, monitoring and tracing, sharing the other. Evaluation and Validation for Generative AI owns the measurement toolkit you reach for before and during a release: the four built-in evaluator families (a large language model (LLM) acting as judge for quality, math-based textual-similarity metrics, hosted risk-and-safety scorers, and agentic evaluators), the evaluate() batch harness that runs them over one dataset in a single pass, the Simulator and adversarial red-teaming tools that manufacture test data when you have none, and how to wire a pass or fail quality gate into a continuous integration and delivery (CI/CD) pipeline.Evaluate a generative AI application with the Evaluation SDK Observability for Generative AI Applications (this page reuses the word 'observability' for only the two runtime pillars, though evaluation is the third observability pillar) owns what you watch once it is live: continuous and scheduled monitoring with Azure Monitor alerts, cost and performance telemetry (token consumption, latency, throughput), and OpenTelemetry (the open standard for distributed tracing) spans, the individual recorded steps of a request, that only appear after you associate an Application Insights resource with the project.Trace generative AI applications Each subtopic carries the exact evaluator names, score scales, region limits, and span attributes this page only points to.

Across both subtopics Foundry rewards the same instinct: replace human eyeballing with an automated, telemetry-backed measurement. In practice that means preferring a built-in evaluator inside an evaluate() run wired into CI/CD over a manual spot check, continuous evaluation on sampled live traffic over waiting for a user to complain, and an Azure Monitor alert that reaches a person over a dashboard nobody is watching.Monitor generative AI applications Two corollaries the exam leans on: pick the evaluator by the reference you hold (a fixed reference answer points to a textual-similarity metric, grounding documents but no reference to groundedness or relevance, neither to a language-model judge, and harm screening to the hosted safety evaluators), and build new work on the current stack, the azure-ai-evaluation software development kit (SDK) plus Foundry evaluations, because the older prompt flow evaluation path retires on April 20, 2027.Prompt flow retirement One scale-direction trap rides along: the quality metrics reward higher scores while the safety-severity scale punishes them, so a high number is good news for quality and bad news for safety.