Test an agent

Test an agent before it takes calls#

A test is a short typed chat between your agent and a pretend caller you describe, plus checks on what the agent said and did. A run plays the chat and grades the checks. Nothing rings: no phone line, no carrier and no money are involved.

A run uses your own AI account. The agent's brain (its llm step) writes the agent's replies, plays the caller and answers your yes-or-no questions, so your AI provider bills you for those tokens, and every run reports input_tokens and output_tokens. Dvaarik charges nothing for tests.

Tests need a three-part agent (pipeline_mode: "cascade"). A realtime (speech-to-speech) agent has no text step, so a run is refused with a 422 and a sentence that says so.

In the console, press Test it on any agent on the Agents page. Under the agent editor, the same tests can run on your unsaved changes. Stop stops a test that is waiting or running, and Stop all stops every one of the agent's.

Make one, run it, read the result#

# 1. Save a test (201)
curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/tests \
  -H "X-Api-Key: $DVAARIK_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "name": "Moves a booking",
    "caller_persona": "Ravi, a regular. He wants to move Saturday 4 PM to Monday morning and is in a hurry.",
    "opening_line": "Hi, I need to change my booking.",
    "max_turns": 6,
    "variables": {"company": "Northline Dental"},
    "tool_results": {"book_slot": {"ok": true, "slot": "Monday 10:00"}},
    "checks": [
      {"type": "calls_tool", "tool": "book_slot", "arguments": ["day"]},
      {"type": "says", "text": "Monday"},
      {"type": "never_says", "text": "guarantee"},
      {"type": "judge", "question": "Did the agent read the new time back before ending?"}
    ]
  }'

# 2. Run it (202, queued). The response lists the queued run and its id.
curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/tests/$TEST_ID/run \
  -H "X-Api-Key: $DVAARIK_API_KEY"

# 3. Ask every few seconds until status is passed, failed, error or cancelled
curl -sS https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/test-runs/$RUN_ID \
  -H "X-Api-Key: $DVAARIK_API_KEY"

Reading takes voice:read; saving, deleting, running and stopping take voice:write. The console door is the same under /console/v2/projects/{project_id}/agents/{agent_id} with your account session.

A run is queued, then running, then passed (every check held), failed (at least one did not), error (it could not finish; error says why in one sentence) or cancelled (you stopped it). It carries:

• transcript: the chat in order. Turn 0 is the agent's first words; each later turn is a caller line and the agent's reply. role is agent, caller or tool. A tool line has name, arguments, result and source: canned (the test answered), webhook (your server), engine (the agent hung up), refused (the agent has no such tool, or a hand-off could not happen), knowledge (its knowledge bases), mcp (one of its MCP servers) or handoff (the call moved to another agent). A line your tool webhook answered with a failure adds reason, a stable code (what each means). truncated marks a turn cut at its length cap.
• check_results: each check with passed (true, false, or null when the run stopped before grading it) and a one-line detail.
• passed_checks and total_checks, input_tokens and output_tokens, and the provider and model that ran it.
• ended_by: how the chat ended, once it has.
• agent_version (the saved version it played), overrides_used, and cancel_requested_at (when you asked it to stop).

The last three, with the status and the counts, are on every run in a list too: GET /v2/agents/{agent_id}/tests returns every test with its last_run, and GET /v2/agents/{agent_id}/test-runs (optionally ?test_id=) lists runs, newest first.

How a run ended#

ended_by is null while a run is queued or running, and on runs made before 7 November 2026.

Stop a run#

curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/test-runs/$RUN_ID/cancel \
  -H "X-Api-Key: $DVAARIK_API_KEY"

The answer (200) is the run as it stands:

• Queued: it is cancelled at once, with ended_by: "stopped". Nothing was asked of your AI.
• Running: it still says running, now with cancel_requested_at set, and ends cancelled before its next request to your AI. It keeps the chat and the tokens so far, and checks are not graded on half a chat (their passed is null); a stop during grading keeps the checks already decided. Asking again changes nothing. A run that finishes before its next request keeps its real result, with cancel_requested_at still set.
• Finished: a 409 with a sentence, such as "This test run has already finished (passed), so there is nothing to stop."

The machine door counts against your project's requests per minute; the console door allows 30 stops a minute, enough to stop every waiting test at once.

When a request is not valid#

A request to the test routes that fails validation is a 422 with one item for each field that is wrong. field is the path into your request, message says what is wrong, and msg says both:

{
  "detail": [
    {
      "loc": ["body", "checks", 2, "says", "text"],
      "msg": "checks[2].text: text is required",
      "type": "value_error",
      "field": "checks[2].text",
      "message": "text is required"
    }
  ]
}

checks[2] is the third check. A variable or tool is named after a dot (variables.company, tool_results.book_slot), or as a JSON string in brackets when its name is not a plain identifier. field is null when the error is about the body as a whole (it is missing or is not JSON). A value of the wrong type can give one item for each type it could have been. A request that reads but is refused for another reason, like a realtime agent, is a 422 whose detail is one sentence.

Checks#

A tool call counts only when it ran, answered by the test or by your server. A test has up to 10 checks, 5 of them judge: each one is one more request on your AI account.

Tools in a test#

By default a test never reaches your systems. Each tool the agent calls is answered from the test's tool_results ({"ok": true} for a tool you gave no answer), and a tool the agent never declared is refused, as on a call.

Set use_real_tool_webhook: true to call the agent's real tool_webhook_url instead. Each request carries the header X-Dvaarik-Test: true and "test": true in the signed body, so your server can tell; anything it does for real, like a booking, really happens. The agent needs a tool webhook URL and its signing secret, or the run stops with a sentence that says so.

A tool from one of the agent's MCP servers is answered the same way, under the name the agent sees (tool_results["crm__lookup_order"], {"ok": true} by default). With use_real_tool_webhook: true it is called on the server for real, with the header X-Dvaarik-Test: true; the switch needs a tool webhook only when the agent has actions of its own. calls_tool and never_calls_tool checks use the name the agent sees.

Hand-offs in a test#

With can_hand_off on, a run really moves, as a call would (Hand the call to another agent): from the hand-off on, the receiving agent's own model writes the replies, on its own AI account, with its own prompt and actions (answered by the test, or by its own tool webhook and MCP servers with use_real_tool_webhook: true).

• The hand-off is a tool line named hand_off_to_agent, with source: "handoff" when the call moved, or "refused" when it could not (the result says why).
• Each agent line names the agent that spoke it, in agent.
• A calls_tool check on hand_off_to_agent counts only a hand-off that moved the call.
• The tokens count every agent's model.

Test changes before you save them#

Send overrides with a run to try a different prompt, greeting, variables or tools without saving the agent:

curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/tests/run-all \
  -H "X-Api-Key: $DVAARIK_API_KEY" -H "Content-Type: application/json" \
  -d '{"overrides": {"prompt": "You book visits for {{company}}. Always read the time back."}}'

The agent's own rules apply to the overrides. They are kept on the queued run only until it finishes and are never saved to the agent, whose version does not move. overrides_used marks such a run.

Rehearse the backup brain#

An agent with a backup brain (pipeline_config.llm_fallback) can rehearse a failure of its main brain: send force_primary_failure: true with a run or with Run all.

curl -sS -X POST https://api.developers.dvaarik.com/v2/agents/$AGENT_ID/tests/run-all \
  -H "X-Api-Key: $DVAARIK_API_KEY" -H "Content-Type: application/json" \
  -d '{"force_primary_failure": true}'

• The agent's main brain is treated as failed and never asked: every agent reply comes from the backup brain, on its own AI account. The pretend caller and the judge questions keep the main brain.
• Each agent line the backup brain answered has fallback: true in the transcript. A run without the flag can have such lines too, when the main brain really failed.
• The run says force_primary_failure: true; its provider and model still name the main brain.
• An agent with no backup brain is refused with a 422. When the backup brain's account cannot be used, the run ends error with a sentence that says so.

In the console, turn on Rehearse the backup brain above Run all. It shows only for an agent with a backup brain, and the lines it answered are marked.

Limits#

Deleting a test deletes its runs.