Domain 2 of 6 · Chapter 3 of 3

Tune agent behavior from evaluation results

Unlock the complete study guide.

Bundled into the existing Developing in Agentic AI Systems premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Reading an evaluation finding as a tuning decision
  • Instruction files: persisting a correction across tasks
  • Tuning a custom agent profile
  • Which tools a custom agent actually gets
  • Keeping MCP tool errors visible to the agent
  • Copilot CLI tool permissions
  • Hooks: stopping a risky tool call before it runs
  • Hooks: holding completion until the work is done
  • Requiring accountable review of tuning changes
  • Preinstalling the cloud agent's environment
  • Correcting context in a Copilot CLI session
  • Copilot Memory: curating what persists
  • Making a broad request verifiable
  • Batching pull request feedback for Copilot
  • Revalidating a tuning change
  • Reasoning through exam scenarios

Matching an evaluation finding to the tuning surface

Evaluation findingTune this surfaceTempting but wrong
Cloud agent changes keep failing CI on build or test conventionsRepository-wide instructions in .github/copilot-instructions.mdRestating the conventions in every task prompt
Specialized rules matter only for one kind of fileA path-specific .instructions.md file with an applyTo globAdding those rules to the repository-wide instructions
The agent loses time rediscovering and installing dependenciesPreinstalled tools in copilot-setup-steps.ymlMore instructions telling the agent how to install them
A custom agent needs one lookup tool from an MCP serverserver-name/tool-name in the profile's tools listserver-name/* plus a prompt sentence forbidding the other tools
A profile is told to run tests but never runs a commandThe execute alias in the tools listMore validation prose in the Markdown prompt
One dangerous CLI command must never run automaticallyA --deny-tool rule, which takes precedence over allowsA narrower allow rule added beside --allow-all
The agent declares completion before validation has passedAn agentStop hook returning decision: block with a reasonTrying to continue the work from sessionEnd
A personal convention appears in one user's suggestionsDeleting that user's Copilot Memory preferenceAdding a repository-wide instruction that contradicts it

Decision tree

Must it be prevented, not discouraged?YesEnforcement control--deny-tool rule or agentStop hookNoUnreliable environment?YesSetup stepscopilot-setup-steps.ymlNoMissing capability?YesTools listsingle MCP tool or execute aliasNoNoisy or stale context?YesSession controlsfresh conversation, @file and #numberNoUnwanted persistent knowledge?YesMemory curationdelete the Copilot Memory preferenceNoRole-specific behavior problem?YesCustom agent's Markdown promptspecialist role's behaviorNoInstructionsrepository-wide or path-specific
Cheat sheet

Cheat sheet

  • Repository-wide instructions make recurring validation guidance reusable
  • Path-specific instructions constrain guidance to the files that need it
  • A custom agent's Markdown prompt defines its specialized behavior
  • The lowest-level matching custom-agent profile takes precedence
  • An existing custom-agent pull request retains its selected profile version
  • Batching review feedback lets Copilot address the whole review together
  • Preinstalling dependencies reduces trial-and-error environment setup
  • Required code-owner review protects critical agent configuration changes
  • Compaction checkpoints let reviewers inspect what survived summarization
  • A fresh conversation is appropriate when old context is no longer useful
  • Large tool-output spilling trades immediate detail for context capacity
  • Explicit context references supply the particular material an agent needs
  • The instructions command provides direct control over loaded instruction sources
  • Repository owners can curate persistent facts directly
  • Personal preferences and repository facts have different memory scopes
  • Copilot code review uses repository facts rather than personal preference memory
  • Repository-fact creation depends on the initiating user's access and memory setting
  • Enable one MCP tool, not a whole server, when a narrow capability suffices
  • Unrecognized custom-agent tool names are ignored rather than resolved by intent
  • Shell-based validation requires the execute capability rather than only file editing
  • Repository MCP settings are processed after custom-agent MCP settings
  • CLI deny rules take precedence over tool allows
  • CLI MCP permission patterns can authorize an individual tool
  • An individual MCP server can be disabled without disabling every server
  • Actionable tool-execution errors support corrected retries
  • A preToolUse hook can reject a risky tool invocation before it executes
  • An agentStop hook can require another turn before completion
  • An agentStop hook should detect when it already forced continuation
  • A command-hook crash and a timeout do not enforce the same outcome
  • HTTP preToolUse service failures fall through rather than reliably blocking calls
  • Native hook matcher regular expressions match the whole target value
  • A tuning intervention needs a new validation result linked to the change
  • Always-loaded instructions should prioritize information the agent cannot infer
  • Decomposing a broad request into verifiable steps makes failures easier to catch early

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. https://code.visualstudio.com/docs/copilot/customization/custom-instructions
  2. Agent hooks in Visual Studio Code
  3. https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/customize-the-agent-environment