Create a test set

You changed the agent's instructions on Tuesday and it feels better in the Test your agent panel. The release board will ask whether it is better, and a feeling collected one question at a time is not an answer. Agent evaluation exists to replace that feeling with a number you can reproduce: Microsoft describes it as automated, structured testing that covers more questions and conversations faster than manual, case-by-case testing, and that gives an objective standard against which to measure and compare changes in performance (About agent evaluation^[1]).

Three nouns carry the whole feature, and this guide uses them in exactly one sense each. A test case is a single interaction that simulates how a user would interact with your agent; the interaction can be a single question or an entire conversation, and it can also carry the answer you expect the agent to reply with. A test set is a named group of test cases. An evaluation run is one execution of a test set against the agent. The qualifier on that last term is deliberate: an unqualified "run" is already bound to one execution of an agent flow on Monitor agent flows, and the two are unrelated objects with unrelated histories. Everything on this page happens on the Evaluation page, the agent-level surface where test sets and recent results live, which is qualified for the same reason as the agent Activity page and the agent Analytics page.

Creating a test set looks like a form, but it is really four decisions, and Microsoft's own creation procedure walks them in that order. First the shape: selecting New evaluation makes you choose Single response or Conversation before anything else, and the choice fixes how many cases the set can hold. Second the origin: where the cases come from, which is a menu of generators, importers and capture paths rather than a single blank grid. Third the expectation: what each case declares as the correct outcome, which is the field almost every test method reads, a test method being Microsoft's name for one way of scoring a case. Fourth the identity: the user profile the evaluation authenticates as, which decides what the agent can actually reach while the set runs. The figure below shows those four feeding one test set and, from it, one evaluation run.

That ordering is not decoration. Shape caps the case count and rules some test methods out. Origin decides whether the expectation field arrives prefilled or as homework. Identity is chosen last in the interface and is the one people forget, which is why a set that passes for the maker fails for the audience it was written for.

This page stops at the artifact. Choosing which test method to attach to those cases is Select agent evaluation methods, and reading the scores that come back is Interpret agent evaluation results. What you should be able to do by the time this page ends is sit in front of an agent you already know how to build, with knowledge sources and tools already attached, and produce a test set whose shape matches the behavior under test, whose cases came from a defensible source, whose expectations are complete enough to score, and whose identity matches the person the answer is for.

One more boundary belongs here, because it is the most common misreading of the feature. Agent evaluation does not replace the Test your agent panel. Microsoft frames them as complements: the test chat receives and responds to one question at a time and is hard to repeat, but gives you a full interactive session; agent evaluation runs many cases at once and repeats exactly, but gives you less control over the conversation. The documentation's own recommendation is to use both for a full picture of your agent. Treat the panel as the place you discover behavior and the test set as the place you pin it down.

The first thing New evaluation asks is which of two shapes you want, and the answer is dictated by the behavior under test rather than by taste.

A single response evaluation tests your agent on one unconnected question at a time, instead of a full conversation. Microsoft's own illustration is a customer service agent asked What are your business hours?, whose answer is recorded, after which the evaluation starts fresh with How do I find my order history? (Create a single response test set^[2]). Nothing carries over. That isolation is the point: it makes each case an independent probe of how the agent answers a specific question, which capabilities it calls, and the exact wording it uses. A set of this shape holds up to 100 test cases.

A conversational evaluation assesses the agent's behavior over a longer interaction, where each response depends on previous context within an ongoing conversation. Microsoft names the three things it is for: determining whether an agent can maintain context, ask for clarifications, and complete multi-step tasks (Create a conversational test set^[3]). The ceilings are tighter and they come in a pair. A conversational test set supports up to 20 test cases, and each test case supports up to 12 total messages, which the documentation spells out as 6 pairs of questions and answers. The figure below contrasts the two internal structures and the ceilings that go with them.

Those numbers are worth internalizing because they are the constraint that most often forces a redesign. A six-turn service workflow — six user questions and six agent answers — fits a conversational case exactly, with nothing left over: 6 pairs of questions and answers is the documented maximum for one case. A workflow that genuinely needs eight such pairs does not fit, and the honest response is to split the scenario at a natural checkpoint rather than to compress two pairs into one artificial question. Equally, a regression suite of 60 independent questions cannot be a conversational set at all, because 20 cases is the ceiling; it is a single response set.

The inverse mistake is the one the exam likes. When every question must begin without context from the previous question, the answer is a single response set with independent cases, not one long test chat transcript converted into a test case. A transcript is a conversation by construction: turn four in it was answered with turns one through three in scope, so replaying it proves nothing about how the agent handles turn four cold. If you want to know whether the agent answers How do I find my order history? correctly to a first-time asker, the case has to start there.

Shape also gates which test methods are available, and this is the seam where the next page picks up. Some test methods are documented as available for single response test sets only, some for both shapes; the per-method table on Choose evaluation methods^[4] carries a Test set type column for exactly that reason. So the practical rule is to treat shape as a commitment rather than a toggle: if the test method you need is documented for single response sets only, a conversational set cannot carry it. Decide the shape from the behavior, confirm the test method exists for it on Select agent evaluation methods, and only then start filling cases.

You have picked a shape and you are looking at an empty set. Copilot Studio offers six ways to fill it, listed as a flat menu, and a flat menu of six is hard to reason about. They sort cleanly into three families, and the family is what you actually choose between: cases can be generated by Copilot Studio, authored by you, or captured from an interaction that already happened. The figure below groups the six that way.

The generated family is Copilot Studio writing the questions. Quick question set creates test cases automatically from the agent's description, instructions, and capabilities, producing 10 questions for running small, fast evaluations or to start building a larger test set. Full question set generates cases from a knowledge source or from topics, and lets you choose how many. Neither requires you to have any material prepared, which is exactly why they are the fastest path to a first signal and the weakest evidence of coverage, a limit the next section pins down.

The authored family is you supplying the questions. You can select Or, write some questions yourself and build the set by hand, or you can import test cases from a file by dragging it into the designated area, selecting Browse, or using one of the other upload options. Hand-authoring is the fallback with no prerequisite beyond the agent itself, and the origin best suited to a requirement nobody has written down yet; the file import is the same act at scale, and it has a strict contract covered two sections down. It is also the one origin that supplies no questions of its own, which is why this page's overview compares the other five against each other and leaves hand-authoring out of that table rather than giving it a column of blanks.

The captured family reuses interactions that already exist. Use your test chat conversation automatically populates the test set with the questions you provided in your test chat, using questions from the latest test chat; you can also start an evaluation directly from the test chat using the evaluate icon. The other capture path is production traffic: on the agent's Analytics page, hovering a theme reveals an Evaluate button that builds a set from real user questions grouped under that theme. Both are captures, but they capture very different things, and the difference is who wrote the question. In a test chat capture the questions are yours, so they inherit your assumptions about how users phrase things. In a theme capture the questions are the users', misspellings and all.

Origins are not exclusive to each other over the life of a set. On the Evaluation page you can select an existing test set and choose Add questions or Add conversations, then select an option to generate or manually add more test cases (Modify test cases within a test set^[5]). The realistic build is layered: seed with a quick generated set to shake out the obvious, add hand-authored cases for the requirements that matter to the business, then top up from production once the agent has been live long enough to have themes worth mining.

What the family tells you is what the resulting set can be used to argue. Generated cases argue that the agent handles material it already has. Authored cases argue that the agent meets a requirement you stated. Captured cases argue that the agent handles what somebody actually did. A release conversation usually needs at least two of those three, and no amount of volume from one family substitutes for the other two.

Of the six origins, the generated pair carries the most configuration and the sharpest trap, so it earns its own section.

Quick question set has no configuration at all. It creates test cases automatically based on your agent's description, instructions, and capabilities, and generates 10 questions for running small, fast evaluations or to start building a larger test set. The number is fixed. When a scenario says a maker needs a representative handful of answers checked before a demo in two hours, and the agent has a description and instructions but the maker has no prepared material, Quick question set is the answer precisely because there is nothing to select.

Full question set is the configured sibling, and its first prompt is the one that matters: after selecting it you choose either Knowledge or Topics. Microsoft ties each side to an orchestration mode. Knowledge works best for agents that use generative orchestration, and creates questions by using one of your agent's knowledge sources. Topics works best for agents that use classic orchestration, and creates questions by using your agent's topics. Microsoft frames both as best-fit guidance rather than a hard rule, and the fit is not mysterious: a generatively orchestrated agent leans on knowledge, so questions derived from knowledge exercise the path it usually takes, while a classically orchestrated agent answers through authored topics, so questions derived from topics do the same for it. The figure below traces that fork. Choosing Knowledge for a classic agent tends to generate questions the agent has no authored path to answer, so the resulting failures say more about the generator choice than about the agent.

With Knowledge you then select the single knowledge source to generate from, and for both Knowledge and Topics you select and drag a slider to choose the number of questions to generate. The generator accepts a bounded set of source types: Text, Microsoft Word, Microsoft Excel, PDF files, and SharePoint content, with files up to 5 MB usable to generate test questions. A 40 MB policy manual is not a candidate; either point the generator at a smaller extract or supply the questions from another origin.

Conversational sets have the same pair with different labels. Quick conversation set automatically generates 10 short conversations based on the agent's description, instructions, and capabilities. Full conversation set generates conversations using the agent's knowledge or defined topics, and in that option you can select creating short or long conversations. The generated-versus-captured distinction from the previous section still applies inside the conversational shape: when the latest authored interaction itself is the scenario worth preserving, Use your test chat converts that conversation into a test case rather than asking a generator to invent a comparable one.

The limitation to carry away from all of this is the one Microsoft states outright about generation from existing sources: it is good for testing how your agent uses a knowledge source or topics it already has, but it isn't good for testing for information gaps. The mechanism behind the sentence is worth spelling out, because the sentence alone sounds like a preference. The generator reads the selected source and writes questions about what is in it. Content that is absent produces no question, so it produces no failing case, so the report comes back clean. If leadership asks which employee questions the agent currently cannot answer, generating from the handbook you already attached is structurally incapable of telling you: every question it invents is drawn from content the handbook already contains. That job belongs to production questions, whose wording is independent of the handbook you attached, or to cases you write from the requirement rather than from the content.

Importing is the origin with a written contract, and the contract is short enough to state in full.

Instead of building your test cases directly in Copilot Studio, you can create a spreadsheet file with all your test cases and import it. Microsoft states the bounds as an Important block: the file can contain up to 100 questions, each question can be up to 1,000 characters including spaces, and the file must be in comma separated values (CSV) or text format. Save it as a .csv or .txt file and import it from the same New evaluation screen the other origins live on.

The structure is two columns with fixed names in a fixed order. Add the following headings, in this order, in the first row: Question, then Expected response. Enter your test questions in the Question column and the expected responses in the Expected response column. There is no need to invent the layout: you can download a CSV template under Data source after selecting New evaluation, which is the safest way to get the header row exactly right.

A minimal import file (complete; nothing elided)

Question,Expected response
What are your business hours?,"We're open from 9 a.m. to 5 p.m. from Monday to Friday."
How do I find my order history?,"Sign in, then open Orders from the account menu."
What is the return window?,"Returns are accepted within 30 days of delivery."

Two things in that listing are worth naming explicitly. The header row uses the literal strings Question and Expected response, in that order, because the order is part of the contract and not a convention you can vary. And the answers are quoted because they contain commas, which is ordinary CSV quoting rather than anything Copilot Studio requires; the file is a plain CSV and follows plain CSV rules.

One clause of the contract bites later rather than at import. Expected responses are optional for importing a test set. However, you need expected responses to run match, similarity, and compare meaning test cases. So an import with an empty second column succeeds, the set appears, the case count is right, and the set is nonetheless half-finished: the moment a test method that reads the expected response is attached, those cases have nothing to compare against. The failure surfaces as an Invalid result for that method on that case, which is the subject of the expectation section below. Treat a blank Expected response column as a deferred obligation, not as an optional field.

The last practical note is about the ceiling. The file cap of 100 questions matches the single response set cap of 100 test cases, which is not a licence to plan a 100-case set as the default. A hundred cases is a lot of expected responses to author and maintain, and the value of the set comes from rerunning it unchanged. A tight set of twenty questions that genuinely represent the requirement is better regression evidence than a hundred questions nobody will keep current.

The one origin whose questions you did not write, and did not have Copilot Studio invent, is production traffic. It arrives through themes, which are groupings of related user questions that triggered generative answers, surfaced on the agent's Analytics page (Analyze user questions by theme (preview)^[6]). Themes is a preview feature, so treat it as subject to change rather than as a stable contract.

The path is short. On the agent's Analytics page, go to the Themes list, hover over a theme, and select Evaluate; you can also select See all to see more themes first. Then select Create and open, and you land in the same editor the other origins produce, where you name the set, attach test methods, and fill in expectations. Microsoft states what the resulting set contains: selecting Evaluate for a theme creates a new test set at the theme level that includes all questions classified to that theme, up to 100 questions.

There is a narrower route, and it is the one to reach for when the whole theme is too broad. Drill down on the theme first, apply filtering to the question list, and then select Evaluate from the question list page. Same destination, smaller and more deliberate set. The figure below shows both routes converging on one test set.

What makes theme-derived sets distinctive is scoping. Theme-based evaluations test specific parts of your agent's scope, so a customer service agent can track answer quality for billing and payments questions separately from other use cases like troubleshooting. That is a different unit of comparison from a whole-agent score: it lets you track billing quality on its own instead of averaging it into everything else, which is usually what a release conversation needs to hear.

The prerequisites are easy to miss, because they are not on the Evaluation page at all. Before creating test sets from themes you need access to themes in analytics, which requires the Bot Transcript Viewer security role to view the list and its metrics, granted by an admin during agent sharing, and the environment-level settings for storing conversation transcripts in Dataverse turned on. The name of that role is one of the vendor artifacts that still carries the older "bot" wording; it stays spelled exactly as Microsoft spells it, while the thing being tested is an agent everywhere else in this guide. There is also a data threshold: at least 50 questions with generative answers must be logged in the past seven days before Copilot Studio generates suggested themes. A brand-new agent has no themes, and no amount of Evaluation-page work will produce them.

One consequence of how themes are built is easy to misread. Only questions that trigger a generative AI answer can be classified into a theme, and Microsoft is explicit that triggering a call to generative answers isn't the same as having an answer to a user query. So a theme legitimately contains unanswered questions, and a test set built from that theme legitimately contains questions your agent could not answer in production. That is a feature. Those are precisely the cases worth an expected response and a rerun after you fix the knowledge behind them.

A case with no expectation can be run but not scored against anything specific, and this is the decision most half-finished sets are missing.

Microsoft's rule is one sentence and it is worth memorizing in its exact shape: all test methods, except general quality, require expected responses or keywords. General quality is the exception because it asks an LLM how well the agent answered rather than whether it matched something you wrote, which is why you don't need to add expected answers to test cases to complete a general quality evaluation. Every other test method reads a field you author on the case. Three such fields exist, and which one applies depends on the test method: an expected response (the answer text you expect), expected keywords (words or phrases the answer should contain), and expected capabilities (the tools or topics you expect the agent to use). The figure below lays out a case's anatomy with all three.

The failure mode when the field is empty is specific and it is not a failed test. Any test case without expected answers, or without expected keywords for the keyword method, produces an Invalid result for that method. Invalid is not the same as fail: fail means the agent answered wrongly, invalid means you did not tell the evaluation what right looks like. A set full of Invalid results is an authoring defect that will read, at a glance on the results page, like an agent problem.

Editing is done on the case, and Copilot Studio gives you a bulk path. During or after creating a test set, you can edit the test cases by changing the wording of questions or modifying the expected responses, keywords, or tools as needed, and you can select multiple test cases to edit them in bulk by selecting the checkboxes beside each test case (Modify test cases within a test set^[5]). Within one case you edit the text in the Question field, add the expectation fields the attached test methods require, and select Apply to save changes to that case. When the whole set is right, Save stores it without running, and Evaluate runs it immediately. Those two buttons appear at the end of every creation path in the documentation. While expectations are still incomplete, Save is the one to reach for, because running a set whose expectation fields are empty returns Invalid rather than a score.

Conversational cases have their own editor and their own name for the expectation. Selecting the edit icon on a conversational case opens the conversation editor, where you see the user messages, which you can edit, and the agent's reference responses, which are how you expect the agent to respond in the context of the conversation. You can select a user question or a reference response to change it, select Add a user question to add more questions and responses within the 12-message ceiling, and select Done to close the editor. The vocabulary shift is real and worth noting once: in a single response case the field is the expected response, and in a conversational case the per-turn equivalent is the reference response.

Which test method to attach, and therefore which of the three expectation fields you owe, is the next page's subject rather than this one's. One caution belongs here anyway, because it affects how you read the documentation while authoring. Do not rely on a count of the available test methods. Microsoft's Change the details of a test set^[7] page opens by saying you have a choice of three test methods, also referred to as graders, and then prints a table further down the same page listing seven of them, the same seven-row table that appears on Choose evaluation methods^[4]. The two statements sit on one page and cannot both be current. Work from the per-method table, which names each method, what it measures, which test set type it supports and what configuration it needs, and never from a headline number.

The last decision is the one furthest down the form and the one that most often invalidates an otherwise careful set. Selecting User profile lets you select or add the account you want to use for this test set, or continue without authentication, and the evaluation uses this account to connect to knowledge sources and tools during testing.

The consequence is stated plainly by Microsoft: automated testing uses the authentication of the selected test account, so if your agent has knowledge sources or connections that require specific authentication, you select the appropriate account for your testing. A connection in that sentence is the same object Configure actions and connectors covers: a stored authentication credential for a connector, not a new concept introduced by evaluation. What evaluation adds is a way to choose whose connections are in play for a run.

That is what makes profiles a persona tool rather than a plumbing detail. You can use profiles to simulate specific user experiences: Microsoft's own example is that a director's user profile can access different knowledge sources than an intern's user profile, and the agent returns different results (Change the details of a test set^[7]). If a scenario says a SharePoint library and a connector tool return different content depending on who asks, and the requirement is to prove the restricted employee sees the restricted answer, running the set under the maker's broad production connections proves the opposite of what was asked.

Adding one is a short sequence with a review step in the middle, traced in the figure below. Open a test set and select Manage profile, select the User dropdown and either pick an existing account or select Add an account and sign in. Copilot Studio then authenticates that user profile and fetches the user's existing connections, and you can configure, review, and save these connections before you run an evaluation. Connections with a warning are selectable: change the existing connection to a working one, select Reconnect to try to reestablish it, or select Create new connection. Working connections are marked with a green check. When all connections are working select Save, and once the Manage profile and connections pop up disappears select Save again to update the test set. Results then show which user profile was used, so the identity travels with the evidence.

Three bounds ride along with this decision. Using an account other than the account with connector authentication is in public preview, so plan for change. Agent evaluations that use user authentication require access through the Microsoft Copilot Studio connector, and if an admin turns that connection off you can't run tests using the evaluation tool at all, which makes an environment's data policy a genuine prerequisite for this feature rather than a background concern. And in Government Community Cloud environments makers can't add a user profile to their test sets, though they can still run evaluations without one, and the text similarity method is unavailable there while all other test methods are available (About agent evaluation^[1]).

The security consequence deserves its own sentence because it is the least intuitive part of the feature. When Copilot Studio generates test cases or runs an evaluation, it uses the authentication credentials of a connected account to access your agent's knowledge sources and tools, so generated test cases can include sensitive data that the connected account can access, and any maker with access to the agent can also view the test sets linked to that agent. Generation is not a sandbox. Pointing a generator at a confidential source under a privileged account writes what it found into an artifact that every maker on the agent can read.

Four failure modes recur on this surface, and each has a documented cause rather than a folklore one.

The first is generation refusing to produce anything. Test case generation fails if one or more questions violate your agent's content moderation settings. Microsoft lists three possible reasons: the agent's instructions or topics lead the model to generate content that the system flags, the connected knowledge source includes sensitive or restricted content, or the agent's content moderation settings are overly strict. The resolution is to try different actions such as adjusting knowledge sources, updating instructions, or modifying moderation settings. Read that list as an instruction about where to look, because none of the three fixes live on the Evaluation page. Regenerating without changing one of the three inputs is likely to fail the same way, and the input at fault is usually identifiable from the agent's own configuration: a procedures document full of restricted clinical content is a knowledge-source problem, not a generator bug.

The second is the Invalid result already covered, arriving in bulk because a set was imported with an empty Expected response column and a matching test method was attached afterwards. The cure is authoring, not rerunning.

The third is a mismatch between the shape you chose and the evidence you wanted. Twelve messages per conversational case, six question-and-answer pairs, and twenty cases per conversational set are the ceilings; a scenario that needs more has to become more than one case. In the other direction, converting one long test chat into a single case when the requirement was independent questions produces a set that can never demonstrate cold-start behavior.

The fourth is scope. Agent evaluation currently doesn't support Fabric data agents, so an agent whose value comes through a Fabric data agent needs a different validation route for that part of its behavior; the seam is described on Integrate a Fabric data agent. More broadly, agent evaluation measures correctness and performance, not AI ethics or safety problems: an agent might pass all evaluation tests and still produce an inappropriate answer, so responsible AI reviews and content safety filters remain in place alongside evaluation rather than being replaced by it.

Two bounds are worth writing on the plan rather than discovering later. Test results are available in Copilot Studio for 89 days; to keep them longer you export the results to a CSV file. That is a retention window on results, not on the test set itself, so a quarterly comparison needs exports even though the set persists. And for a multilingual agent, you select the language you want for a given evaluation, with the default being the language you primarily use when entering evaluation inputs; when you run more evaluations the output is in the same language as the input. A set authored in English and run against an agent answering in German is not a neutral test, and Microsoft documents the specific consequence for one test method: the compare meaning grader detects the mismatch and marks the response failed due to language inconsistency.

None of these are exotic. They are the four places where an artifact that looks finished on screen is not yet evidence.

A test set earns its keep on the second run, not the first, and everything about how you maintain it follows from that.

The argument for a stable set is stated by Microsoft as one of the things a test set is for: you can test changes to your agents by using the same test set, so you have an objective standard to measure and compare changes in performance. The word doing the work is same. If the questions change between runs, a score difference has two candidate causes and you cannot separate them. Replacing all cases after every agent edit and then reading the resulting pass rates as a regression comparison is the specific antipattern, and it is seductive because regenerating a set is one click while curating one takes an afternoon.

The workable discipline is to treat a set as two layers. The stable core is the cases that encode requirements, and it changes only when a requirement changes. Around it sit exploratory sets, freshly generated or freshly mined from a theme, which you run once, learn from, and either promote into the core or discard. Copilot Studio supports that shape directly: you can quickly create new test sets or modify existing ones to cover changing agent capabilities or requirements, and the Add questions path lets a core set grow without being rebuilt. Deleting is equally deliberate. On the Evaluation page a test set is deleted from the three dots on its card or from the three dots in the Configure test set pane, and a result is deleted separately from the Recent results section, so removing a stale result does not cost you the set that produced it.

Once the core set is stable, running it stops being something a person should do. Agent evaluation supports automation so makers can run evaluations without manual intervention: you run evaluations and see results using the Copilot Studio interface, through Power Platform REST APIs, or by adding actions in tools, flows, or Power Automate. The connector path is the low-code one, and its actions are named: Evaluate Agent runs an evaluation for an agent using a specific test set, Get Agent Test Run Details gets status and results for a specific test run, Get Agent Test Runs lists evaluation runs for an agent, Get Agent Test Set Details gets metadata for a specific test set, and Get Agent Test Sets lists available test sets for an agent (Trigger agent evaluations with connectors^[8]). Microsoft's worked pattern is a Recurrence trigger that calls Evaluate Agent for a selected test set, polls Get Agent Test Run Details until the run state is Completed, and then sends the results to a database, an email alert, or a Power BI dashboard that measures agent quality over time. The figure below traces those four steps and the polling wait that sits in the middle of them.

That is the bridge to release management. Using REST APIs or Power Platform connectors you can programmatically trigger evaluation runs and integrate testing into automated workflows such as continuous integration and continuous deployment pipelines, which lets you validate agent behavior as changes are introduced without manual execution in Copilot Studio. The pipeline mechanics themselves belong to Automate agent deployments; what this page contributes to that pipeline is the artifact it gates on.

So the through-line from the first section holds all the way to production. Shape decided what the set could express, origin decided what it can argue, expectation decided whether it can be scored, and identity decided whose view of the agent it measured. A set that got all four right is a stable, reusable, automatable quality signal. A set that got any one of them wrong still produces a number, which is exactly what makes the four decisions worth making deliberately.