# Test an agent

## Test an agent before it takes calls

A **test** is a short typed chat between your agent and a pretend caller you
describe, plus **checks** on what the agent said and did. A run plays the chat
and grades the checks. Nothing rings: no phone line, no carrier and no money
are involved.

A run uses **your own AI account**. The agent's brain (its `llm` step) writes
the agent's replies, plays the caller and answers your yes-or-no questions, so
your AI provider bills you for those tokens, and every run reports
`input_tokens` and `output_tokens`. Dvaarik charges nothing for tests.

Tests need a three-part agent (`pipeline_mode: "cascade"`). A realtime
(speech-to-speech) agent has no text step, so a run is refused with a 422 and a
sentence that says so.

In the console, press **Test it** on any agent on the [Agents](/console/agents)
page. Under the agent editor, the same tests can run on your unsaved changes.
**Stop** stops a test that is waiting or running, and **Stop all** stops every
one of the agent's.

## Make one, run it, read the result

```bash
# 1. Save a test (201)
curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/tests \
  -H "X-Api-Key: $DVAARIK_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "name": "Moves a booking",
    "caller_persona": "Ravi, a regular. He wants to move Saturday 4 PM to Monday morning and is in a hurry.",
    "opening_line": "Hi, I need to change my booking.",
    "max_turns": 6,
    "variables": {"company": "Northline Dental"},
    "tool_results": {"book_slot": {"ok": true, "slot": "Monday 10:00"}},
    "checks": [
      {"type": "calls_tool", "tool": "book_slot", "arguments": ["day"]},
      {"type": "says", "text": "Monday"},
      {"type": "never_says", "text": "guarantee"},
      {"type": "judge", "question": "Did the agent read the new time back before ending?"}
    ]
  }'

# 2. Run it (202, queued). The response lists the queued run and its id.
curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/tests/$TEST_ID/run \
  -H "X-Api-Key: $DVAARIK_API_KEY"

# 3. Ask every few seconds until status is passed, failed, error or cancelled
curl -sS https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/test-runs/$RUN_ID \
  -H "X-Api-Key: $DVAARIK_API_KEY"
```

Reading takes `voice:read`; saving, deleting, running and stopping take
`voice:write`. The console door is the same under
`/console/v2/projects/{project_id}/agents/{agent_id}` with your account session.

A run is `queued`, then `running`, then `passed` (every check held), `failed`
(at least one did not), `error` (it could not finish; `error` says why in one
sentence) or `cancelled` (you stopped it). It carries:

- `transcript`: the chat in order. Turn 0 is the agent's first words; each later
  turn is a caller line and the agent's reply. `role` is `agent`, `caller` or
  `tool`. A `tool` line has `name`, `arguments`, `result` and `source`:
  `canned` (the test answered), `webhook` (your server), `engine` (the agent
  hung up), `refused` (the agent has no such tool, or a hand-off could not
  happen), `knowledge` (its knowledge bases), `mcp` (one of its MCP servers) or
  `handoff` (the call moved to another agent). A line your tool webhook
  answered with a failure adds `reason`, a stable code
  ([what each means](/docs/agents#your-tool-webhook)). `truncated` marks a turn
  cut at its length cap.
- `check_results`: each check with `passed` (`true`, `false`, or `null` when
  the run stopped before grading it) and a one-line `detail`.
- `passed_checks` and `total_checks`, `input_tokens` and `output_tokens`, and
  the `provider` and `model` that ran it.
- `ended_by`: how the chat ended, once it has.
- `agent_version` (the saved version it played), `overrides_used`, and
  `cancel_requested_at` (when you asked it to stop).

The last three, with the status and the counts, are on every run in a list
too: `GET /v2/agents/{agent_id}/tests` returns every test with its `last_run`,
and `GET /v2/agents/{agent_id}/test-runs` (optionally `?test_id=`) lists runs,
newest first.

## How a run ended

| `ended_by` | What happened |
| --- | --- |
| `agent_hung_up` | The agent ended the call |
| `caller_hung_up` | The pretend caller hung up |
| `turns_ran_out` | The test's `max_turns` were used up |
| `transferred` | The agent handed the call to a person ([transfers](/docs/agents#transfer-to-a-person)) |
| `error` | The run could not finish; `error` says why. Status `error` always has this |
| `stopped` | You stopped it. Status `cancelled` always has this |

`ended_by` is `null` while a run is queued or running, and on runs made before
7 November 2026.

## Stop a run

```bash
curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/test-runs/$RUN_ID/cancel \
  -H "X-Api-Key: $DVAARIK_API_KEY"
```

The answer (200) is the run as it stands:

- **Queued:** it is `cancelled` at once, with `ended_by: "stopped"`. Nothing was
  asked of your AI.
- **Running:** it still says `running`, now with `cancel_requested_at` set, and
  ends `cancelled` before its next request to your AI. It keeps the chat and
  the tokens so far, and checks are not graded on half a chat (their `passed`
  is `null`); a stop during grading keeps the checks already decided. Asking
  again changes nothing. A run that finishes before its
  next request keeps its real result, with `cancel_requested_at` still set.
- **Finished:** a `409` with a sentence, such as "This test run has already
  finished (passed), so there is nothing to stop."

The machine door counts against your project's requests per minute; the console
door allows 30 stops a minute, enough to stop every waiting test at once.

## When a request is not valid

A request to the test routes that fails validation is a `422` with one item
for each field that is wrong. `field` is the path into your request,
`message` says what is wrong, and `msg` says both:

```json
{
  "detail": [
    {
      "loc": ["body", "checks", 2, "says", "text"],
      "msg": "checks[2].text: text is required",
      "type": "value_error",
      "field": "checks[2].text",
      "message": "text is required"
    }
  ]
}
```

`checks[2]` is the third check. A variable or tool is named after a dot
(`variables.company`, `tool_results.book_slot`), or as a JSON string in
brackets when its name is not a plain identifier. `field` is `null` when the
error is about the body as a whole (it is missing or is not JSON). A value of the
wrong type can give one item for each type it could have been. A request that
reads but is refused for another reason, like a realtime agent, is a `422` whose
`detail` is one sentence.

## Checks

| `type` | Fields | Passes when |
| --- | --- | --- |
| `says` | `text`, up to 200 characters | Any agent reply contains the text, ignoring capitals and spacing |
| `never_says` | `text` | No agent reply contains it |
| `calls_tool` | `tool`, and optionally `arguments` (up to 10 names) | The agent ran the tool, with every listed argument when you list any |
| `never_calls_tool` | `tool` | The agent never ran it |
| `ends_call` | none | The agent hung up itself; needs the agent's `can_end_call` on |
| `judge` | `question`, up to 300 characters | Your own model reads the chat and answers yes, with a one-line reason |

A tool call counts only when it ran, answered by the test or by your server. A
test has up to 10 checks, 5 of them `judge`: each one is one more request on
your AI account.

## Tools in a test

By default a test never reaches your systems. Each tool the agent calls is
answered from the test's `tool_results` (`{"ok": true}` for a tool you gave no
answer), and a tool the agent never declared is refused, as on a call.

Set `use_real_tool_webhook: true` to call the agent's real `tool_webhook_url`
instead. Each request carries the header `X-Dvaarik-Test: true` and
`"test": true` in the signed body, so your server can tell; anything it does
for real, like a booking, really happens. The agent needs a tool webhook URL
and its signing secret, or the run stops with a sentence that says so.

A tool from one of the agent's [MCP servers](/docs/agents#mcp-tools) is answered
the same way, under the name the agent sees (`tool_results["crm__lookup_order"]`,
`{"ok": true}` by default). With `use_real_tool_webhook: true` it is called on the
server for real, with the header `X-Dvaarik-Test: true`; the switch needs a tool
webhook only when the agent has actions of its own. `calls_tool` and
`never_calls_tool` checks use the name the agent sees.

## Hand-offs in a test

With `can_hand_off` on, a run really moves, as a call would
([Hand the call to another agent](/docs/agents#hand-the-call-to-another-agent)):
from the hand-off on, the receiving agent's own model writes the replies, on its
own AI account, with its own prompt and actions (answered by the test, or by its
own tool webhook and MCP servers with `use_real_tool_webhook: true`).

- The hand-off is a `tool` line named `hand_off_to_agent`, with
  `source: "handoff"` when the call moved, or `"refused"` when it could not
  (the `result` says why).
- Each agent line names the agent that spoke it, in `agent`.
- A `calls_tool` check on `hand_off_to_agent` counts only a hand-off that
  moved the call.
- The tokens count every agent's model.

## Test changes before you save them

Send `overrides` with a run to try a different `prompt`, `greeting`,
`variables` or `tools` without saving the agent:

```bash
curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/tests/run-all \
  -H "X-Api-Key: $DVAARIK_API_KEY" -H "Content-Type: application/json" \
  -d '{"overrides": {"prompt": "You book visits for {{company}}. Always read the time back."}}'
```

The agent's own rules apply to the overrides. They are kept on the queued run
only until it finishes and are never saved to the agent, whose version does not
move. `overrides_used` marks such a run.

## Rehearse the backup brain

An agent with a [backup brain](/docs/agents#backup-brain)
(`pipeline_config.llm_fallback`) can rehearse a failure of its main brain: send
`force_primary_failure: true` with a run or with Run all.

```bash
curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/tests/run-all \
  -H "X-Api-Key: $DVAARIK_API_KEY" -H "Content-Type: application/json" \
  -d '{"force_primary_failure": true}'
```

- The agent's main brain is treated as failed and never asked: every agent
  reply comes from the backup brain, on its own AI account. The pretend caller
  and the `judge` questions keep the main brain.
- Each agent line the backup brain answered has `fallback: true` in the
  transcript. A run without the flag can have such lines too, when the main
  brain really failed.
- The run says `force_primary_failure: true`; its `provider` and `model` still
  name the main brain.
- An agent with no backup brain is refused with a `422`. When the backup
  brain's account cannot be used, the run ends `error` with a sentence that
  says so.

In the console, turn on **Rehearse the backup brain** above Run all. It shows
only for an agent with a backup brain, and the lines it answered are marked.

## Limits

| What | Limit |
| --- | --- |
| Tests per agent | 50 |
| Run all | at most 6 tests in one request; with more, name up to 6 in `test_ids` |
| Waiting or running | 12 per project and 24 across your projects; past that a run is refused with a 409 |
| Running at the same moment | 2 per developer; the others wait their turn |
| Back-and-forths (`max_turns`) | 1 to 12, 6 by default |
| Caller description, opening line | 2,000 and 500 characters |
| Canned tool answers | 20 tools, 2,000 characters of JSON each |
| One run | the chat stops after 90 seconds and keeps what was said; the `judge` questions get 30 more |
| Runs kept | 30 days, then deleted automatically |

Deleting a test deletes its runs.

---

Source: https://developers.dvaarik.com/docs/agent-tests · every page as one file: https://developers.dvaarik.com/docs.md