Skip to content

Write and run tests

A Test is a conversation you describe in advance, with a verdict attached: who the person is, what the agent already knows about them, what they say, and what must be true of the agent's replies. You write it in English, and the platform turns the description into an executable case. You collect cases into suites and run a suite whenever you want to know whether the agent still behaves the way you intend: after editing a Playbook, after switching models, before going live.

Every case runs against the real agent, through the same code that handles customers, in Sandbox: the person is a fresh copy of a persona, and every booking, notification, email, integration call and send is captured and shown rather than performed. Nothing a Test does reaches anyone.

Find Tests in the agent's Test tier, beside the emulator.

What a Test is for

The emulator answers "what does the agent say when I say this?" one conversation at a time. A Test answers a different question: "does the agent still do the things I care about?" for every situation you have written down, in one click, with a record you can compare against last week's.

Cases accumulate. When a customer conversation goes wrong, turn it into a case with Make a Test, watch it fail, fix the Behavior document, and watch it pass. From then on the case is a guard: an edit that reintroduces the problem shows up as a newly failing case before a customer sees it.

Suites

A suite is a named group of cases with shared defaults. Group cases by what they protect: Booking behavior, Refusals and guardrails, Regressions. Suite defaults set the channel and the number of samples per case for every case that does not say otherwise.

The suites list shows each suite's case count, the verdict of its last run as a stoplight (pass rate and how many cases are newly failing), a Run button per suite, the progress of any run in flight at the top, and how many case units the agent has left today.

Examples as Tests

Every Playbook that has Examples gets a suite of its own, marked From Examples and named after the Playbook. Each Example is one case: the person's lines from the Example are the script, the case starts in that Playbook, the checks describe the shape of the Example's reply (text, chips, a picker, a card), and one judge criterion asks whether the agent's reply says in substance what the Example's reply says. The platform keeps the suite in step with the Playbook: edit an Example and its case follows; remove the Examples and the suite goes.

Run and read these suites like any other. They cannot be edited here; to change a case, edit the Example on the Playbooks page, or choose Copy to a suite on the case to get an editable copy in one of your own suites. The Playbooks page shows each Playbook's pass rate from the last run of its Examples suite (Tests: 3 of 4 passed), linked to the suite.

Make a Test from a conversation

Any conversation, live or sandbox, can become a case. Open the conversation's turn inspector on the Conversations page and choose Make a Test on the turn to stop at. The person's messages up to that turn become the script, and the checks start from what the agent's reply actually did (a picker shown, chips offered, the actions called, the Playbook it ended in). The case opens in the editor in the Recorded from conversations suite, with a note reminding you to edit the checks into what should have happened. The person in the case is a stand-in (first name, company, product interest, status), never the real phone number or email.

Run on a schedule

A suite can run on its own: in Edit suite, set Run on a schedule to every day or every week, with the time, the weekday, and the time zone. The suites list shows the schedule and the next run; the run appears in the suite's history as started by schedule. A scheduled run counts against the same daily units as any other run and waits when another run is active. When a scheduled run fails, or cannot start, the team is notified through the agent's Notifications capability destination.

Describe the situation

The fastest way to a suite is to describe what you want covered and let the platform write the cases.

  1. Click New suite and choose Generate cases.
  2. Describe the situation in a sentence or two, the way you would brief a colleague: "A prospect with a booked demo messages to say they can't make it." Say what matters to you if it is not obvious ("they must never be asked which appointment").
  3. Set roughly how many cases to write (1 to 15, 5 by default) and an optional tag for all of them.
  4. Click Generate. The dialog shows Writing cases… while the platform works; a suite of five takes about a minute.

The suite opens with the new cases listed. Each one covers the situation from a different angle (the plain path, a terse or hesitant person, a digression, a refusal, an edge in what the agent knows) and tests one thing. The cases are written from your description and from the agent's own Behavior document, so they name the agent's real Playbooks, representatives, templates, and People fields, and they are checked against the agent before they are saved. A case the platform could not make valid is not saved; a notice on the suite page says what was dropped and why.

Generate more on a suite page runs the same dialog against an existing suite. New cases are written to differ from the ones already there.

Write a test in English

Open a case, or click New case in a suite. The editor opens on two text areas.

  • Scenario: who the person is, what the agent already knows about them, how the conversation starts, and what they say. "Dr. Mira Chen, a qualified lead assigned to Jordan with a demo 15 minutes from now, replies to the appointment reminder on RCS: sorry, something came up, she can't make it today."
  • Expected outcome: what must be true of the agent's behavior. "The agent cancels the appointment without asking which one, offers to rebook before she has to ask, and shows a time picker."

Below them a summary line says what the case will execute: "Person: Dr. Mira Chen, RCS, 3 turns, 5 checks, 2 criteria", or Not compiled yet for a case that has a description and nothing else.

Click Update from description to compile the description into the structure the runner executes: the person fixture, the channel and start mode, the person's turns with a typed fallback on every tap, exact checks wherever a fact can be checked, and criteria for what only a reader can judge. Compilation uses the agent's Behavior document, so the result references only Playbooks, representatives, and fields the agent has. Lint findings appear inline; when the description asks for something a case cannot express (a specific calendar date, a simulated caller), the editor shows a one-line caveat and compiles the rest.

Click Save to keep the case, and Run this case to try it on its own. You can save a case that has not been compiled; it is a draft, the suite table flags it, and a run skips it as errored until you compile it.

Describe only what the agent can see. The agent reads the person's name, contact details, company, title, product interest, People fields, assigned representative, and appointments; it does not read notes, tags, or consent status, and an expectation that depends on those tests something the agent cannot see. Use relative dates ("next Tuesday") because cases run at real time.

Editing the description later and clicking Update from description again changes only what the new description requires. Edits you made under Advanced survive unless the description contradicts them.

Advanced

Expand Advanced to see and edit the structure behind the description: a form for the common fields and a YAML tab that shows the whole case as a document. Both edit the same thing, and what you see on the YAML tab is exactly what the API stores. The expander opens on its own for cases that have structure but no description. Lint findings appear inline as you edit: an error blocks saving (a Playbook that does not exist on this agent, a misspelled check), a warning tells you the case will run but may not mean what you think (a People field the agent does not declare, a time-picker or booking check on an agent whose scheduling is not configured), and a note records a limitation. Warnings also show as a chip on the case's row in the suite, so a case that cannot pass as written is visible before anyone spends a run on it.

Scenario

Person. The fields the agent will resolve on the persona the case runs against: name, phone (which sets the likely time zone, as in production), email, company, title, product interest (matched against your Things catalog), status, and any People fields. Assign a representative by id when the case is about a specific person on your team. Leave the person empty for an anonymous inbound with no record. The copy is made for the case and removed afterwards; it never appears on the People page.

Appointments. Existing appointments for the person, relative to now: "in 15 minutes, 30 minutes long, with Jordan". Reminder cases and duplicate-booking cases need one.

Start. How the conversation opens, one of:

Start The case begins with
Inbound (default) The person's first turn.
Outreach The agent starts the conversation: choose the Outreach and supply the inputs its opener needs.
Reminder The appointment reminder already sent; the person's first turn is their reply to it.
Resume A conversation in progress: supply the prior turns as history and name the active Playbook.

History. Prior turns entering the case, for any start mode.

Clock. You can write a fixed date and time for the case, absolute or "next Tuesday at 14:00 in Chicago". The platform accepts and validates it but does not apply it yet: cases run at real time, and the editor notes this.

Stubs. If the agent has custom integrations, give each one the response it should return during the case (an order status, a stock level). An integration with no stub is captured: the agent is told the call was captured rather than made and carries on as though it succeeded. The result records the call either way, so a check on whether the agent called it holds in both cases.

Script

Each turn is either something the person types or a tap.

Turn Meaning
Text The person sends the text.
Chip Tap the reply chip whose label contains the text, or matches a pattern such as /book|yes/i.
Slot Pick the first, second, or nth time in the last time picker.
Confirm Answer yes or no to the last confirmation prompt.
System event The channel reports an opt-out (STOP), opt-in, subscribe, or unsubscribe.
Action Tap a suggested action: add to calendar, dial, open a link, view location.
Media Send an image or video.

Give every tap a fallback: the text the person types if the chip, slot, or prompt is not there. Without one the case errors when the affordance is missing; with one the case continues and your checks show what was missing. A Simulate turn, which hands the rest of the conversation to an AI-played person with a goal, is part of the format but not available yet; the editor flags it and the case would error at run time.

Expectations

Checks are exact. Pick from the closed list; the editor offers each one and will not save a check it does not know.

Check What it asserts
Final reply has text The last agent turn said something.
Time picker shown A tappable time picker went out on some turn.
Appointment created / Active appointments A booking was made during the case; exactly n non-cancelled appointments exist afterwards.
Outcomes The Flow outcomes recorded, in order.
Active Playbook at end Which Playbook the conversation ended in.
Person fields Values written to the person's record, such as a People field the agent was meant to capture.
Chips on any turn / Chips on last message A chip with this label appeared; the final message offered tappable options.
Text includes / excludes / matches Substrings that must or must not appear anywhere in the agent's text; regular expressions that must match.
Actions taken / not taken How many times the agent used a named action (created an appointment, looked up an order), or that it never did.
No warnings No turn raised a runtime warning.
Model calls per turn / Turns to outcome Upper bounds on model calls in one turn and on turns before a Flow outcome.
Reply count How many messages the agent sends per turn, as a range.
No reply to A system event such as STOP got no reply.

Criteria are sentences a judge grades pass or fail, quoting the line the verdict rests on. Each one names the observable fact and the turn it applies to. "The first reply names Jordan as the representative" is gradable; "the agent is helpful" is not. When a check can say the same thing, use the check. The judge grades your criteria and nothing else.

Samples repeats the case within a run, up to five times. The case passes only if every sample passes, which is how you catch behavior that is right most of the time.

Notes and tags

Use notes to record why the case exists: what was seen, when, and in which conversation. Use tags to run related cases together (scheduling, regression) across suites. A generated case also records the description it was written from.

Preview what the agent will know

Preview on the case editor composes what the agent's first turn would read, without running it: the person block exactly as what the agent reads renders it (with the likely time zone, the matched product interest, and the representative), the intake block and the rendered opening message for an outreach start, the current-date line, the channel line, and the Playbook the conversation would start in. Use it to confirm the scenario says what you meant before spending a run on it, and to see which fixture fields the agent will and will not see.

Running a suite

Click Run on a suite (or Run all on the suites page), Run selected on the cases you have ticked in a suite's case table, or Run tag for a tag. Before the run starts the platform estimates its cost in case units; the figure appears in the confirmation and on the run page. If the estimate exceeds what the agent has left today, the run does not start and the message names the remaining budget, when it resets, and a smaller selection that fits.

One run at a time per agent. A run starts within a minute and shows its progress as a stacked bar (passed, failed, errored, pending) that refreshes while it executes, with the cases currently in flight listed beneath it.

  • Pause stops new cases from starting; a case mid-conversation finishes its current turn and starts over from its scenario on resume. A run that runs out of daily units pauses itself and resumes when the budget resets at midnight UTC.
  • Resume continues.
  • Cancel ends the run; cases not yet started are marked skipped and the results so far are kept.

Reading a run

A finished run shows its counts and pass rate over decided cases (errored cases are reported separately and never counted as failures) and a comparison with the previous run of the same selection: newly failing, newly passing, and still failing. Newly failing is the number to watch after an edit. Pin a run to make it the baseline that later runs are compared against.

The attribution card says what was under test: whether the Behavior document changed since the compared run, the runtime build, the agent's model, and the judge version. A verdict that flips when only the judge version changed is a judge change, not a regression.

Reading a result

Open a case in the run to see its result. The page leads with the case's description, so you can read the verdict against what you asked for.

  • Transcript. The person's side with the directive that produced each line, the agent's replies rendered the way the channel would show them, and system lines for anything the platform recorded instead of doing: [notify: New call booked for Tuesday 2 pm], [integration: lookupOrder → stubbed response]. Nothing in a system line left the platform.
  • Checks. Each check with pass or fail and a one-line detail, such as "chip 'Book a call' never appeared (saw: Tell me more | Not now)", and under each check what it reads. Every check comes from the case's expected outcome: "actually shows bookable times, not just a promise" becomes Time picker shown. If the case as it is written now carries lint warnings, they appear above the checks, because a check that this agent can never satisfy (a time picker on an agent without scheduling) fails for a reason unrelated to conversation quality.
  • Criteria. Each criterion with pass or fail, the judge's reason, and the agent line it quotes as evidence, plus a short summary and the turn the judge thought was weakest.
  • Usage. The units the result cost, and the token counts behind them.
  • Inspect turns opens the agent's turn records for the case, the same records a customer conversation keeps, so a wrong reply can be traced to what the agent read and decided.

An errored result says which stage failed and why (the case was a draft that had not been compiled; the scenario could not be built because a Playbook was deleted; a chip was missing and the turn had no fallback; the judge did not answer; the case exceeded its unit ceiling). Fix the case or the agent and run again.

Case units and the daily budget

Each result is weighted after it runs in case units, from the tokens the agent, the judge, and any simulated person consumed. A short scripted case with a judge call is one unit; a long case with several samples is several. The suites list shows the units left today, and every result shows what it cost, which is also the signal to trim a case that has grown.

The platform sets a daily budget of units per workspace (250 by default), shared by the workspace's agents, with a ceiling per run and per case. Test runs are billed like any other agent usage; conversations in the emulator are not. The budget resets at midnight UTC. Your account contact can raise the workspace limits.

Generating and compiling cases is billed as assistant usage on the agent, like other operator-initiated writing, and does not draw on the daily case-unit budget.

The Copilot as an author

Anychat Copilot can write and repair Tests. Ask it to "write Tests for the booking Playbook" and it describes the cases, compiles them, previews each one to confirm what the agent will know, and saves them for your approval. Ask it to run a suite and it reports the estimate before starting, then reads the results and proposes Behavior edits and case edits together, each behind its own approval. Runs the Copilot starts count against the same daily budget. Ask it why a case failed and it reads the result, names the expectation the failing check came from, and looks at the case's turn records for the cause, the same way it diagnoses a live conversation.

What's next