Skip to content

How a Test runs

A Test case runs the agent as it is deployed. The turn code, the composed turn, the Playbook routing, the Rules, the capabilities, the channel rendering, and the model are the ones a customer gets. What is different is the world around the agent: the person is a persona, and the effects that would otherwise leave the platform are captured instead. This page describes that boundary, so a verdict can be read for what it is.

The production pipeline

Each case is delivered to the agent's conversation service as a sequence of inbound events, the same entry point the messaging gateway calls after it verifies a webhook. Every turn composes what the agent reads, routes to a Playbook, offers the capabilities that Playbook uses, calls the agent's configured model, renders the reply for the scenario's channel, and writes a turn record. The replies come back to the runner instead of going to a channel, and the runner renders them for the transcript the way the channel would show them (text, cards and carousels, chips, a time picker with its slots). See How the transcript shows rich messages.

Taps in the script are delivered the way the gateway delivers them: a time slot or a confirmation arrives as the typed result the channel would send, a reply chip as its label text. Deterministic paths in Flows are therefore exercised exactly as on a phone.

How the transcript shows rich messages

The transcript is text, so each rich element the person saw is one line in square brackets, and a chip is always printed under the thing it belongs to. A carousel lists its cards, and each card's chips are indented beneath that card; the chips under the whole message are a separate [suggested replies: …] line:

Here are the boats we rent at both marinas.
[carousel: 2 cards]
  [card 1: 19-Ft Powerboat — Seats 8] (media)
    [chips: Book at Wahweap | Book at Halls Crossing]
  [card 2: Kayak — Single seat] (media)
    [chips: Book at Wahweap]
[suggested replies: Something else | Rates (link)]

Here the judge can see that "Book at Halls Crossing" is offered for the powerboat only. A chip that does something on the phone says what: (link), (call), (map), (add to calendar). A question with choices shows its [options: …], a time picker lists its slots ([time picker, 3 slots: …]), and separate messages are separated by a blank line. The judge is told how to read this layout, and conversation review and a persona's transcript use the same one.

A fresh copy of a persona

Each case runs against a copy of a persona — a Person record in the sandbox, with a starting point — made for that case alone. The scenario's person fields are the starting point: name, contact details, company, title, product interest, status, People fields, an assigned representative, messaging consent, and any prior turns the case starts from. The runtime resolves the copy the way it resolves a customer, by the conversation identity bound to it, reads it into the turn, and writes to it through the People capability and the scheduling mirror, so a check on a People field or on an appointment reads persisted state, not a mock. Appointments listed in the scenario are created through the scheduling service against that copy.

The copy lives in the sandbox partition. It is invisible on the People page, in segments, in audiences, in exports, to connected outreach and follow-ups, and in the People count on the Overview. Its phone number is profile information that identifies no one, so a case may reuse a real customer's number without touching their record. When the case finishes, the copy and its appointments are removed. A scenario with no person runs the cold-inbound path with the agent's real auto-create setting; a record the agent creates during the case is in the partition too.

The sandbox partition

Conversations a Test creates are sandbox conversations. They are kept beside the live ones and carry the sandbox mark, so Inspect turns on a result opens the same turn records a customer conversation keeps. What the partition never does is count. It is excluded from People, from statistics, from the automatic conversation review, from billing meters on message volume, and from every scheduled action: reminders, follow-ups and Automated Outreach do not run against it. Conversation and message listings leave it out unless you ask for it.

The mark applies to the conversations, not to the spend: the model usage a run causes is billable usage attributed to the agent, as described under case units. Turn records are ordinary records, kept 90 days.

Captured effects

Reads are real; writes are captured. Availability is really looked up against the connected calendar, Knowledge is really searched, a custom integration's data is really needed — but nothing the agent does during a case leaves the platform.

The agent On a live turn In a Test
Books, moves or cancels an appointment The appointment is written, mirrored to the connected calendar, enrolled for the reminder, and notified The booking is captured: visible to the case and its checks, never mirrored to a calendar, never reminded, never notified, hidden from Calls and the Overview
Notifies the team The message is sent to the configured destination Captured and shown in the transcript as a system line; nothing is sent
Sends an email The email is sent Captured, not sent
Sends an outbound message The message goes out on the channel Captured, and shown as the opening of the conversation
Calls a custom integration The HTTP request is made Answered from the scenario's stub for that integration. Without a stub the call is captured and the agent is told it was captured rather than made, and to proceed as if it had succeeded. The call, and which of the two happened, is recorded
Writes to the person The customer's record The case's copy of the persona, which the personFields check reads
Replies The message goes to the channel The message is returned to the runner and rendered for the transcript

Every captured effect is returned with the reply and kept on the turn record, so a person watching the emulator and a script running headless see the same list. Effects appear in the transcript as system lines such as [notify: New call booked for Tuesday 2 pm], which is how a judge criterion like "the team was notified before the confirmation" is graded without anything having been sent.

Timers do not run in the sandbox. A case that needs "the reminder has already fired" starts from that state rather than waiting for it.

How tests are written

A case is described in English (a scenario and an expected outcome) and compiled into the structure the runner executes. The author is the platform's most capable model, given the description and the agent's own Behavior document: its Playbooks and when each applies, message templates, representatives, People fields, Things, scheduling, and outreach settings. The author therefore references ids and names that exist on the agent, writes a typed fallback for every tap, prefers an exact check to a judged criterion whenever a fact can be checked, and writes each criterion so a literal-minded judge can grade it from the transcript alone. Generating a suite from a description works the same way, with the suite's existing cases in context so new ones differ.

Every draft is linted against the agent before it is saved: unknown checks, missing Playbooks or representatives, undeclared People fields, and taps without fallbacks are caught, and a draft with errors gets one repair round. A generated case the author cannot repair is dropped and reported rather than saved. Compilation returns the document and its findings without saving anything; the editor saves. Authoring spend is metered as operator-initiated assist usage on the agent, separate from the case units a run consumes.

Examples as Tests

A Playbook's Examples double as Tests. For each Playbook with Examples the platform keeps a system suite with one case per Example, compiled without a model: the Example's person turns are the script, the case starts in that Playbook, and the expectations come from the shape of the Example's reply as rendered by the same compiler that renders a Message (text present, tappable options, a time picker, a card or carousel), plus one judge criterion that the reply says in substance what the Example says. The suite is regenerated in place whenever the Examples change, so a case keeps its history across edits, and is deleted with them. It is read-only through every surface; a case is edited by copying it into a suite of your own.

A case can also be made from any conversation, live or sandbox: the person's messages up to a chosen turn become the script, and the checks describe what the last reply in that window did, read from its turn record. The person in the case is a stand-in built from the People record (first name, company, product interest, status) and never carries a phone number or email.

A suite with a schedule is started by the platform's once-a-minute tick at the next due time, with requestedBy.via set to schedule. Quota applies as to any run; a scheduled run waits while another run is active on the agent, and one that fails or cannot start notifies the team through the agent's Notifications capability.

What the judge sees

A case's checks are decided by the platform from what reached the wire and what was persisted. Criteria are decided by an LLM judge: the same judge that reviews every closed live conversation, applied here on request with the case's criteria, so there is one opinion of what a good conversation is. The judge grades the criteria the case states and nothing else, and is deliberately limited to what a person in the conversation would see, plus the facts the agent was given:

  • a digest of the agent's Behavior document (what the agent is supposed to do);
  • the scenario as a person would understand it: the goal or persona text, and only the part of the person rendered to the agent, produced by the same function as the case preview, so the judge never holds the agent to a fact it could not see (a note, a tag, a consent status);
  • the rendered transcript, including the captured-effect system lines;
  • the criteria.

The judge does not see turn records, the composed turn, or the model's reasoning; verdicts are about the conversation, and checks cover the mechanism. It is told that the person's side may be scripted and to grade only the agent, and it grades each criterion literally, returning pass or fail, a one-sentence reason, and the agent line the verdict rests on. The judge prompt is versioned; the version is recorded on every result and run, and a verdict that changes between runs with different judge versions is attributed to the judge rather than the agent. The judge model is a platform setting (claude-sonnet-5); a suite or case does not choose it.

Case units and quotas

Tests are the one feature an operator can use to spend model tokens on demand in bulk, so every result is measured and every run is bounded.

Each result is weighted after it runs in case units: the tokens the agent's turns, the judge call, and any simulated-person calls consumed, with output tokens counted at five times an input token and cache reads at a tenth, divided by a fixed unit size (40,000 weighted tokens in the current calibration) and rounded up, with a minimum of one. The weight is measured, never declared: a case counts as several units only when it cost several. Recalibrating the unit size re-weights future results only.

Decided results (passed or failed) are metered as billable usage on the agent, evaluation.cases and evaluation.case_units, alongside the model usage they caused; an errored result records its model usage but no case meters, since no verdict was delivered.

Quotas are expressed in units and enforced before and during a run:

Limit Scope Default
Daily units Per workspace, per UTC day; an agent may carry a lower limit 250
Units per run Per run, from the pre-run estimate 150
Case samples per run Cases × samples 200
Units per case A case that exceeds it is errored, and not billed 8
Samples per case 1, up to 5
Turns per case 20 as a guideline, 40 at most

A run's estimate is the sum of each selected case's last measured units (or a figure derived from its shape); a run whose estimate does not fit is refused with the shortfall and a smaller selection that fits. While a run executes, each finished case's measured units count against the day, and a run that exhausts the budget pauses at the next case boundary and resumes when the day resets. Workspace limits are set by system operators.

What changes a verdict

Every run records what was under test: the Behavior document's content hash and last-edit time, the runtime build, the agent's model, and the judge (and simulator) model and prompt version. When a case flips between two runs, that record says whether the Behavior document changed, the platform changed, or the judge changed, and the run's comparison lists the cases that are newly failing, newly passing, and still failing.

Retention

Results (transcripts, checks, judge output) are kept 90 days, the same as turn records. Run summaries (counts, units, attribution, comparison) are kept one year, so the trend outlives its transcripts. Pinning a run as a baseline does not extend its retention. A case's copy of its persona, its appointments, and its conversation context are removed when the case finishes; a cleanup pass removes anything an interrupted run left behind.

What's next