Perform evaluation, error analysis, and tuning

An agent keeps pushing changes that fail continuous integration (CI) checks. Is the task description missing something, is a tool unavailable, or did the agent misread a requirement it was actually given? This domain treats evaluation as one loop of three stages, each answering its own question from evidence: define what success looks like and which signal measures it, diagnose a shortfall by naming its failure layer (one of the five listed in the next section) from the record of what the agent was given and what it did, and tune the surface that owns that cause. The classic trap is reasoning from the outcome instead: a pull request exists, so the work must be right, or the result is wrong, so the prompt must need rewriting.

The define stage comes first, in Define agent success criteria and evaluation signals, because criteria written after the run cannot steer it: put complete acceptance criteria in the task, then match each requirement to the signal that actually measures it, such as CodeQL findings for vulnerable code patterns or executed test output for whether tests passed. The diagnose stage, in Analyze agent failures and identify their root causes, takes over when a signal shows a shortfall: open the record of what happened, such as a Copilot cloud agent session log or the Chat Debug view in Visual Studio Code (VS Code), and name the failure layer before choosing a fix. The five layers are a configuration gate (a platform setting that stopped an action or input by design), tool availability, the tool call, context, and reasoning. The tune stage, in Tune agent behavior from evaluation results, closes the loop: change the surface that owns the diagnosed cause, for example instructions, a custom agent's Markdown prompt, the tools list, Copilot cloud agent setup steps, Copilot CLI session controls, or Copilot Memory curation, then rerun against the same acceptance criteria. The decision tree on this page routes a question to the right stage.

When two answers both look reasonable, the exam rewards the one grounded in execution or enforcement. Executed test output outweighs the generated test source, a session log outweighs a completion summary, and a check result becomes a merge gate only when branch protection or a ruleset makes it a required status check: it then blocks merging, for collaborators subject to the rule, until the check passes, whereas a configuration gate decides whether an action or input happens at all. Tuning follows the same instinct: instructions and prompts shape what an agent attempts, while controls such as tools lists, Copilot CLI permission rules, and hooks (Copilot agent hooks, not Git hooks) decide what it can actually do, so a behavior that must be guaranteed moves out of prose and into configuration (the tuning page explains how a hook behaves when it fails). Even then, a tuning change stays a hypothesis until a new run shows the expected result.