Choose an evaluation method

A returns agent is asked How long do I have to send something back? and answers You can return most items within 30 days of delivery. Did it pass? There is no answer to that question until you say which scorer is judging, because the same response can come back as a pass, as a failure, or as Invalid, the outcome a case gets when it never declared what right looks like, and all three verdicts are defensible.

The scorer has a name. A test method is Microsoft's term for one way of scoring a test case; the documentation also calls the same thing a grader, an alias worth recognising once because it appears in older wording, and then setting aside. This guide says test method everywhere after this sentence.

You arrive here with the artifact already built. Build agent test sets covered the four decisions that produce a test set: its shape (Single response or Conversation), where its test cases came from, what expectation each case declares, and the user profile that the evaluation run, one execution of the whole set against the agent, authenticates as. It deliberately stopped before the standard those cases are judged against. Reading the numbers that come back, calibrating thresholds against risk, and comparing one run with the next belong to Interpret agent evaluation results. The slice in between, which method expresses which acceptance criterion, is this page. The work is a translation: a requirement written in prose ("the refund answer must state the 30-day window, and it must have gone through the authenticated refunds tool") becomes a named method, an expectation you owe on every case, and, sometimes, a pass score you have to justify.

Microsoft describes every method on the same four axes, and reading a method along them in order is the practical selection procedure. Measures says which dimension of correctness the method inspects. Test set type says which shapes the method is offered for, which makes it a hard filter rather than a preference. Scoring says whether a percentage comes back or a bare verdict, covered in Percentages you threshold, verdicts you don't below. Configurations says what the choice costs you in authoring, from None at one end up to a name, evaluation instructions and a set of labels at the other, all three of which are the Custom method's fields and are explained in Custom methods encode a standard of your own below. The figure below runs those four axes as the filter they are.

Methods attach to the test set, not to individual cases. When creating^[1] or editing^[2] a test set you select Add test method, select all the methods you want to test with, and select OK; you can add multiple methods. Some of them then ask for a pass score, and some ask for more criteria. Selecting Save stores the change. Selecting a method already on the set reopens it so you can edit its criteria or delete it. Because a set can carry several methods at once, the real decision is rarely "which one" and usually "which combination, and what does each one add that the others cannot see".

One caution belongs at the top rather than buried in a traps list, because it changes how you read the documentation while you work. Do not carry away a count of the available methods. Change the details of a test set^[2] opens by telling you that you have a choice of three test methods, also referred to as graders, and then prints a method table further down that same page listing seven, the identical table published on Choose evaluation methods^[3]. Both statements sit on one page and they cannot both be current. Work from the per-method table, which is consistent across the articles that reproduce it, and name the methods you mean.

Under the method names sit only three kinds of machinery, and knowing which kind you picked predicts more about the result than the name does.

Deterministic methods compute a comparison between the agent's answer and text you wrote. Exact match, Keyword match and Text similarity are all of this kind: given the same pair of strings they return the same verdict every time, because string equality, token presence and a cosine similarity metric are calculations rather than opinions. Model-judged methods hand the answer to a large language model and ask it to assess something. General quality, Compare meaning and Custom work this way. Execution inspection is the odd one out: Tool use never looks at the wording at all, and instead asks which tools or topics the case actually used. The figure below groups the named methods under those three headings.

Those three groupings are this guide's shorthand for organising Microsoft's methods, not labels you will find in the product. The line under them is Microsoft's own, though: its agent architecture guidance separates code-based graders^[4], whose examples include binary pass-fail and calculated vector similarity, from using a language model as judge, and it notes the tradeoff directly. Code-based graders make deterministic checks easy to verify but struggle with nuance and qualitative aspects such as tone and accuracy. A language model as judge enables scenario-based testing at scale and encodes a wide array of user preferences, at the cost of bringing entropy into the evaluation process when you lean on it alone.

Two practical consequences follow, and they are the reason this distinction leads the page rather than trailing it.

The first is reproducibility. A deterministic method that failed yesterday and passes today has told you something changed in the agent. A model-judged method that moves by a few points between two runs of an unchanged agent has told you considerably less, because the nondeterminism of the language model doing the judging is part of what you are reading rather than a signal about the agent. That interpretation is the next page's subject; what matters here is that choosing a model-judged method is choosing to live with it.

The second is what each kind can express. None of Copilot Studio's deterministic methods can tell you the answer was rude, off-topic or unsupported by the retrieved knowledge, because none of those are properties of the characters in the string. No model-judged method can guarantee that a mandated sentence survived word for word, because a paraphrase is exactly what it is built to forgive. When an acceptance criterion contains both halves, that is not a tie to be broken by preference. It is a signal to attach one method of each kind to the same test set and let each answer the half it can see.

Decide what you are allowed to use before you decide what you would like to use. Three documented conditions remove options, and none of them is negotiable at authoring time. The figure below runs them as an ordered filter over the methods in Microsoft's table.

The first is the test set's shape, the Single response or Conversation choice made at New evaluation and fixed thereafter. Microsoft's method table carries a Test set type cell for every row. General quality, Keyword match and Custom read Single response or conversation. Compare meaning, Text similarity, Exact match and Tool use read Single response. So the moment you built a conversational set to prove context retention or a multistep task, the wording-sensitive methods stopped being available to you, and the conversational test set article^[5] reflects that in its own step list.

One detail on that same page is contested and should not be memorised in either direction. Its step for adding methods says that for conversation test sets you can add the General quality, Keyword match, Tools use or the Custom test methods, while the method table printed immediately below it on that page lists Tool use as Single response, and Choose evaluation methods^[3] opens its Tool use section with "Available for single response test sets". Two statements on one page, and the table is the version that agrees with every other article that reproduces it. Design as though orchestration checks belong in a single response set: if you need to prove that a specific tool was invoked, put that proof where the documentation is consistent, and treat any conversation-set availability you find in the product as a bonus rather than as the plan.

The second condition is the environment. In Government Community Cloud (GCC) environments, agent evaluation is documented with two limitations^[6]: makers cannot add a user profile to their test sets, although they can still run evaluations without one, and makers cannot use the similarity test method, with Microsoft stating that all other test methods are available. The link in that limitation points at the Text similarity section of the method article. The design consequence is concrete: an acceptance criterion that depends on prescribed wording cannot be automated the same way in GCC as in a commercial environment, so it has to be expressed some other way — Keyword match, which is available but only checks that terms are present, or human review, or a check outside the evaluation entirely.

The third condition is the agent itself. Agent evaluation does not currently support Fabric data agents, which is a whole-feature exclusion rather than a method-level one. There is no method to choose for one, and the evaluation strategy for that part of a solution has to be built somewhere else.

What survives all three filters is your actual menu. Only then is it worth asking which method expresses the requirement best, and the remaining sections take the methods in the order the table lists them.

Start with the method that is already switched on. All test sets start with General quality by default, it is available for single response and conversation sets, its Configurations cell reads None, and you do not need to add expected answers to test cases to complete a general quality evaluation. That combination makes it the only method that will score a set you have not finished authoring, which is both its convenience and the reason it gets over-trusted.

What it does is hand the interaction to a large language model and ask how effectively the agent answered, applying a consistent prompt to guide scoring across the set. Microsoft publishes the key criteria it uses^[3], and they are worth knowing individually because each one fails for a different reason:

• Relevance: to what extent the response addresses the question, for example whether it stays on the subject and directly answers what was asked.
• Groundedness: to what extent the response is based on the provided context, rather than introducing unrelated or unsupported information.
• Completeness: to what extent the response provides all necessary information, covering all aspects of the question with sufficient detail.
• Abstention: whether the agent attempted to answer the question at all.

The combination rule is the part that surprises people, and it is the reason a set of plausible-looking answers can score lower than expected. To be considered high quality, a response must meet all these key criteria. If one criterion is not met, the response is flagged for improvement. Only responses that are both complete and well supported receive top marks, while answers that are incomplete or lack supporting evidence receive lower scores. The figure below traces one response through the four criteria and that all-or-nothing gate. In practice this means a beautifully grounded answer that omits half the question cannot reach high quality on the strength of the half it answered well.

General quality is the right instrument when there is no exact answer expected, which is most of what a generative agent produces. An explanation of a returns policy, a summary of a knowledge article, a recommendation drawn from several sources: all of these are correct in many phrasings, and demanding a specific string from them is the single most common mis-selection on this page. Reaching for Exact match because an open-ended grounded explanation "should say the right thing" produces a wall of failures that tell you nothing about the agent.

One caveat is published on the method itself and is easy to trip over during remediation. Reducing the number of knowledge sources for the agent is not guaranteed to improve general quality grading, because the retrieved knowledge, meaning the knowledge the model thinks is relevant to a specific test case, might be too large. If a general quality score disappoints you, pruning sources is a plausible-sounding fix that Microsoft explicitly declines to promise, so treat it as a hypothesis to test rather than a remedy to apply.

The takeaway is narrow and worth holding: General quality answers was this a good answer, on four named criteria combined with AND, and it answers it without an answer key. Everything else on this page answers did this meet what I wrote down — an expected answer, expected keywords, expected capabilities, or your own instructions and labels — and charges you that expectation for the privilege.

Put one expected response and one agent response side by side and you will find that "does it match?" is not one question. Take the returns case from the top of this page, with an expected response of Returns are accepted within 30 days of delivery. against the agent's You can return most items within 30 days of delivery. Four methods read that pair, and each reads a different property of it. The figure below shows the same pair going into all four.

Compare meaning judges intent similarity: it compares the ideas and meaning behind the words rather than the words themselves, to decide how closely the response aligns with what you expected. It is offered for single response sets, its score is a percentage, and you set a passing score threshold to decide what counts as a pass. The default passing score is 50, which is a starting value rather than a recommendation, and leaving it untouched is a decision you should be able to defend. Compare meaning is the right choice when an answer can be phrased in different correct ways but the overall meaning or intent still has to come through, which describes most policy explanations, most summaries, and most of what a grounded agent says.

Text similarity judges wording. A cosine similarity metric assesses how similar the agent's answer is to the wording of the expected response and determines a score, and you set a passing score threshold on top of it. Do not memorise the unit that score comes back in; the documentation gives two, as Percentages you threshold, verdicts you don't below sets out. Microsoft's guidance for it is specific: use it when a correct answer needs to match the expected response exactly, or almost exactly, in both wording and sentence structure, for example when a legal document is generated. It also names its own companion, saying the method is typically used alongside Compare meaning, which ensures similarity of meaning but does not ensure similarity of wording. That pairing is not belt and braces. Each half rejects something the other accepts: Compare meaning passes a correct restatement that destroys the mandated phrasing, and Text similarity passes a close paraphrase of the right shape that has quietly changed the obligation.

Keyword match judges token presence and nothing else. It checks whether the answer contains some or all of the words or phrases you define, and you choose whether a pass requires Any of them, meaning at least one word or phrase matches, or All of them, meaning every expected word or phrase must match. Microsoft distinguishes it from Text similarity explicitly: keyword match ensures the presence of certain terms but does not ensure similarity of construction. It is the one wording-adjacent method also offered for conversation sets, which makes it the usual fallback when a multi-turn requirement has a lexical component.

Exact match judges identity. The answer either matches the expected response character for character and word for word, or it fails; anything different fails. It suits short, precise answers such as numbers, codes, or fixed phrases, and Microsoft says plainly that it does not suit answers people can phrase in multiple correct ways. The documentation gives you the boundary between it and Text similarity in one line: if the whole answer must exactly match the whole expected answer, use Exact match instead.

All four are the reason the expectation rule exists. All test methods except general quality require expected responses or keywords, and a case that does not carry the field its method reads produces an Invalid result for that method rather than a fail. Compare meaning, Text similarity and Exact match each read the case's expected response; Keyword match reads its expected keywords or phrases. Invalid is an authoring defect, not an agent defect, and a set full of them looks at a glance exactly like a broken agent.

The traps here run in both directions, which is why naming the property you actually care about matters more than knowing the method list. Reaching for Text similarity because two semantically equivalent policy answers happen to use different sentence structures scores the wrong property: their structures genuinely differ, and the method will say so. Reaching for Text similarity for open-ended answers on the theory that its score tolerates meaning-preserving rewrites is the same error with the opposite reasoning, because tolerating rewrites is precisely what a wording metric does not do. And reaching for Compare meaning alone when the requirement is that a generated notice preserves prescribed sentence structure leaves the obligation unchecked, because meaning is what survives a rewrite of the structure.

An agent can produce a convincing refund answer without ever calling the refunds tool. It can summarise a policy from a knowledge source when the requirement was to look up this customer's actual order. Every method above would call that a pass, because every method above reads the answer. Tool use is the one that reads the run.

What it tests is whether the agent used specific tools or topics to generate an answer: if it did, the case passes, and if it did not, the case fails. There is no score and no threshold, only the binary outcome, and its Configurations cell reads Expected capabilities. The word capabilities is doing real work there, because the field covers both tools and topics: the same check can assert that an authenticated connector operation ran, or that a particular authored topic handled the turn. The figure below follows one case from the question to that comparison.

Configuring it has a per-case path and a bulk path, and knowing the bulk one is the difference between a usable set and an afternoon of clicking. When adding or editing test methods, select Tool use, then add the expected tools or topics. On an individual case you select the case, choose the topics or tools you expect the agent to use in the Select tools panel, select OK, and select Save. To apply the same expected tools and topics to every case in the set, select the Edit icon in the Tools column heading instead of visiting each row. Use the bulk path when the whole set exercises one capability, and the per-case panel when a case's expected capabilities genuinely differ from its neighbours. As with the other configured methods, a case left without the expectation this method reads produces an Invalid result for it.

The reason to reach for it is orthogonality. The methods in the previous section can all be satisfied by an answer that came from the wrong place, and General quality only partly closes the gap: its groundedness criterion asks whether the response is based on the provided context, not whether a named tool ran, so an answer assembled from a knowledge article instead of this customer's actual order can still read as well grounded. So when the acceptance condition is that a refund action must invoke a specific authenticated tool, General quality alone is the wrong instrument no matter how high it scores. Attach Tool use beside whichever answer-scoring method you already chose, and let each one fail for its own reason.

The seam to the rest of the guide is worth stating, because two similar-sounding surfaces live on different pages. Tool use scores an evaluation run's cases against expected capabilities. Watching what an individual execution actually did, step by step, is the job of the activity and run-history surfaces described in Monitor agent flows, and deciding where a tool sits so that generative orchestration, the mode in which the agent picks its own tools rather than following an authored path, can select it at all is Add tools to topics. Tool use will happily tell you that the expected tool did not run; it will not tell you why, and a tool the agent was never able to select is a placement problem rather than a scoring one.

Everything so far scores a property Microsoft chose. Custom is the method for a criterion that is yours: it lets you test and label agent answers using your own criteria, for example a compliance test for an HR agent that labels answers as either compliant or noncompliant with your description of HR compliance. It is offered for single response and conversation sets, and its Scoring cell reads pass/fail, passing when an answer matches a label you marked as passing.

A custom test has two components to configure, and they do different jobs. Evaluation instructions describe the goal you want the test to accomplish, in other words what you want the test to find out about your agent's answers. Microsoft's guidance for writing them is short: be goal oriented, use only the allowed characters, and use bullet points and headings for organisation. Labels describe the result assigned to each answer, and each label carries a Pass or Fail assignment that counts toward the test set pass rate, the aggregate score Copilot Studio reports for the set, for this method. A label has a name and a description, and a good description is concise and contains the attributes you are looking for in matching answers. One workable strategy is two labels, one for answers that fulfil the criteria and one for answers that do not. Label titles can only use letters, numbers, space, hyphen -, underscore _, forward slash /, ampersand &, plus sign +, and period ., so a name like Compliant (2026 policy) will be rejected on its parentheses.

Microsoft's worked evaluation instructions for an HR compliance test. This is the complete example as published, nothing removed, and it shows the shape the guidance above is asking for: a one-line goal statement, then a heading, then the specific things the model should look for.

Evaluate the agent's response for HR policy compliance.

What to check:
- Determine whether the answer protects privacy and avoids revealing or requesting sensitive data.
- Avoids discrimination, bias, or inappropriate judgments.
- Provides safe, neutral, HR-aligned guidance.
- Does not give legal advice or make definitive claims.

Notice what the instructions do not contain: a threshold, a score, or a verdict. The instructions tell the model what to look for, the labels turn that judgment into an outcome, and only the label's Pass or Fail assignment reaches the pass rate. That division is the thing to hold on to, because it is where custom tests are usually mis-built. Writing "fail the answer if it reveals sensitive data" inside the instructions and then defining a single label leaves the method with nothing to decide between.

The authoring sequence follows directly. When adding or editing test methods, select Custom, enter a name for the custom test, add the evaluation instructions, and add two or more labels, each with a name and a description, using Add label for the ones after the second. Set the Pass or Fail result for each label, then select OK. The figure below traces one response through that machinery, from what you authored to what lands in the pass rate.

Use Custom when the criterion is genuinely organisational rather than linguistic: policy compliance, house tone, a required disclaimer, a regulatory boundary the built-in methods have no vocabulary for. Do not reach for it as a general-purpose replacement for the named methods. A custom test is a model-judged method with all the reproducibility properties of one, and where a deterministic method can express the requirement, it will express it more cheaply and more repeatably.

Choosing a method is also choosing what kind of number you will have to defend later. Microsoft's Scoring column has two shapes. General quality, Compare meaning and Text similarity are Scored out of 100%, and the Text similarity cell carries a caveat this section returns to below. Tool use, Keyword match and Exact match are Pass/fail. Custom sits in the second group with a qualifier, Pass/fail (passes the defined label criteria), because its verdict comes from a label you defined rather than from a comparison.

A scored method gives you a dial and obliges you to set it. The pass score is configured when the method is attached: some methods require one, the pass score determines what score results in a pass or a failure, and you set it and select OK as part of adding the method. Compare meaning ships with a default passing score of 50. Text similarity has no published default and expects the same act: run the cosine similarity comparison, then convert the resulting number into a pass or a failure at the line you draw.

Do not memorise the unit of that field. The method table's Scoring cell for Text similarity says Scored out of 100%, while the method's own section on the same article describes a score that ranges from 0 to 1, where 1 means the answer closely matches and 0 means it does not. The two descriptions are of one number on different scales and the documentation does not reconcile them, so the safe knowledge is the behaviour, not the units: there is a continuous similarity score, and there is a threshold you configure against what the field in front of you displays.

A pass/fail method gives you no dial, and that is a feature rather than a gap. There is no meaningful "70% of an exact match", and the absence of a threshold is what makes Exact match, Keyword match and Tool use easy to reason about in a release conversation. It also means that loosening such a check means editing the expectation itself — widening Keyword match's keyword list, or switching its requirement from All to Any — which is a visible edit to the test set rather than a quiet slider move.

Where the number should sit is deliberately not this page's question. Calibrating a threshold against the consequence of a failure, the size of the audience and the presence of a fallback, and deciding how many runs it takes before a model-judged score is a baseline at all, is Interpret agent evaluation results. What belongs here is the selection consequence: attaching a scored method commits you to owning a threshold and to justifying it, so if nobody on the project is willing to defend a number, that is a reason to express the requirement with a pass/fail method instead.

If an agent is configured to support more than one language, the evaluation gains a setting and one method gains a failure mode.

The setting comes first: for a multilingual agent you can select the language you want for a given evaluation^[6]. Microsoft names three evaluation components that behave specifically for such an agent, and they are worth separating because they fail at different points. Query generation is where the language choice lands: the default language is the one you primarily use when entering evaluation inputs, you select the language you want to run the evaluation in, and later evaluations produce output in the same language as the input. Grader execution is where a mismatch becomes a result. Explainability output is the reasoning attached to a result: in multi-turn conversations with a multilingual agent, the agent provides reasoning in the same language it uses in its responses, so a set run in one language produces its explanations in that language too.

The failure mode belongs to Compare meaning specifically. When you evaluate a multilingual agent, Compare meaning compares the expected response with the agent response, and if an issue related to multiple languages arises, the system detects the mismatch, the compare meaning grader fails the response, and the response is marked failed due to language inconsistency. Read that carefully, because it is a different verdict from the one it looks like. The response was not judged to mean the wrong thing. It was judged to be in the wrong language relative to the expectation you authored, which is usually a symptom of an expected response written in the maker's language while the agent answered in the user's.

The practical rule is a triage rule. Before you rewrite an instruction, reword an expected response or lower a pass score in response to a wave of Compare meaning failures on a multilingual agent, check which language the evaluation ran in and which language the expected responses were authored in. A language-inconsistency failure is a setup defect that no amount of agent tuning will fix, and it is the one Compare meaning result that says nothing at all about whether the agent understood the question.

The last thing to know about method selection is what no selection buys you.

Microsoft states the boundary without hedging: agent evaluation measures correctness and performance, not AI ethics or safety problems^[6]. An agent might pass all evaluation tests and still, for example, produce an inappropriate answer to a question. Responsible AI reviews and content safety filters remain in place, and evaluations do not replace those reviews and filters.

It is worth seeing why no method on this page closes that gap, because each one looks as though it might. General quality's groundedness criterion asks whether a response is based on the provided context, which is a question about sourcing rather than about harm; a well-grounded answer can still be one the organisation should never have given. Exact match and Keyword match confirm that specific strings are present or absent in the cases you thought to write, and say nothing about the enormous space of questions you did not write. Tool use confirms the right tool ran, not that running it was appropriate for that user. Custom comes closest, because you can label answers against a policy of your own, and it is genuinely useful for that, but it is still a model judging a criterion you thought of in advance, applied to a test set of finite size.

That finite size is the crux. Every method here scores the cases in the set, and a pass rate is a statement about those cases, not about the agent's behaviour in general. The controls that operate on the general case are a different family: content moderation settings, the ungrounded-response and knowledge-scope choices, and the review practices covered in Plan a responsible AI strategy.

So the closing rule of the page is a two-part one. Choose the method that expresses your acceptance criterion most precisely, combining methods where the criterion has more than one property, and set thresholds you can defend. Then present the resulting pass rate as what it is: evidence that the agent answered a known set of questions to a stated standard, offered alongside responsible AI review rather than in place of it.