Define agent success criteria and evaluation signals

Unlock the complete study guide.

Bundled into the existing Developing in Agentic AI Systems premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

Scoping a delegated task: which work fits, what counts as done
What validation evidence actually shows about an agent's change
Reading the change itself at the source level
Verifying rendered behavior in the running application
Which security or quality check reports which finding
Quantitative runtime signals and what each one measures
What still stands between a finished agent task and a merge

Which signal answers which acceptance question

Acceptance question	Signal to read	What it establishes	What it cannot establish
Did the agent write a vulnerable code pattern?	Copilot cloud agent's built-in CodeQL analysis	Code security issues in the code the agent generated	Whether an added package carries an advisory — that is the built-in dependency analysis
Does a package the agent added carry a malware or high-severity advisory?	The built-in dependency analysis, which checks newly introduced dependencies against the GitHub Advisory Database	Malware advisories and CVSS High or Critical vulnerabilities for packages the agent introduced	Anything about dependencies that were already in the repository
Did the generated code expose a credential?	Copilot cloud agent's built-in secret scanning result	API keys, tokens, and other secrets in the code the agent generated	Whether a secret-bearing push was stopped; that is push protection
Is the code well structured and maintainable?	Copilot code review, the second opinion Copilot cloud agent requests by default	Quality feedback on the changes, with suggested fixes	A dependency advisory verdict or a credential detection
Does the repository accept the dependency change?	The dependency review action's check on the pull request	Vulnerable versions introduced by package-version changes, and a blocking failure when the check is required	That Copilot cloud agent's built-in analysis was wrong or redundant
Was a secret stopped before it reached the repository?	The push protection block reported during the push attempt	That the push was rejected and why, preventively	That no credential exists elsewhere in the repository's history
Did the tests pass?	The output of the executed run	Which cases ran and which failed, including failures an agent running the tests in VS Code repaired and reran	Which behaviors the suite was asked to cover
How much of the change did the tests reach?	The coverage report uploaded as a workflow artifact	Coverage of the change, inspectable after the run	Whether the tests are isolated, deterministic, or meaningful
What exactly did the agent change?	The Copilot CLI diff of the current directory	The files and edits waiting to be committed	An interpretation of them — that is the CLI's /review command, not Copilot code review
Which tool keeps failing?	Copilot CLI's github.copilot.tool.call.count	Invocations split by tool name and by success	How long any of those calls took
Is the tool slow, or are the LLM calls and agent invocations slow?	Copilot CLI's github.copilot.tool.call.duration against gen_ai.client.operation.duration	Tool execution latency by tool name, versus LLM call and agent invocation duration	How many tokens either component consumed
Is the MCP server reachable, or is its tool broken?	Copilot CLI's github.copilot.mcp.server.connection.count	Completed connection attempts by transport and outcome	Whether a reachable server's tool logic is correct
Will a failing result stop the merge?	Whether that check is a required status check on the protected branch	That a required check must be successful, skipped, or neutral before collaborators subject to the rule can change the branch, so bypass configuration matters too	Whether a human has approved — the agent cannot approve or merge

Decision tree

Cheat sheet

A well-scoped agent task states the problem, acceptance criteria, and intended file scope
Research and planning tasks need not produce a pull request to succeed
Broad, ambiguous, deep business-logic, and production-critical tasks require more cautious delegation
Completing cloud-agent work is separate from human approval and merging
A testing agent can be evaluated on test quality as well as the existence of tests
Generating tests and executing tests provide different evidence
Requested test types and edge cases determine the validation scope
Repository validation can require several different checks
Coverage reports provide a distinct inspection surface for test coverage
The CLI diff command exposes the actual change set for review
The CLI review command requests agent analysis of the changes
CodeQL findings identify security issues in agent-generated code
Dependency analysis checks newly introduced packages for specified advisory risks
Secret scanning detects sensitive credentials in generated code
Copilot code review provides a second opinion on the agent's code
Tool-call counts can be separated by tool and success outcome
Tool latency and model or agent duration use different runtime metrics
Token-use signals distinguish input consumption from generated output
MCP connection outcomes are distinct from tool execution outcomes
A reported check becomes an enforced merge gate only when it is required
Repository dependency review supplies a separate acceptance signal for agent changes
Push protection can reject secret-bearing changes before they reach the repository
Browser interaction evidence verifies rendered behavior beyond source inspection
Interactive browser verification does not replace repeatable repository tests

Unlock with Premium — includes all practice exams and the complete study guide.