# Testing & Evaluation

> Test a voice agent like you unit-test code: scripted conversations, assertions, an LLM judge, and latency metrics — in CI.

Voice agents fail in ways that text tests never catch. The agent talks over
the caller, answers slowly, or drifts off-script. **Agent evaluation** lets
you pin that behavior down. You script a conversation. You assert what the
agent should do. You get a pass/fail result plus latency and quality metrics,
reproducibly, in CI.

A simulated caller connects to your agent over a WebSocket. No phone number,
no browser, and no SIP are involved. The caller speaks your scripted turns
with text-to-speech. The evaluator transcribes and grades the agent's
replies. The run exits `0` only if every assertion passes. It writes a JSON
report with per-turn metrics.

> **NOTE:**
> Evaluation runs against a real agent, over the same runtime path that a live
>   call uses. It uses the same speech-to-text, language, and speech providers
>   that your agent runs with. Thus a green run reflects real behavior, not a
>   mock.

## Write a scenario

A scenario is JSON. It sets who the caller is, which agent to test, the
scripted turns, and the assertions on each turn and on the call as a whole.

```json
{
  "name": "billing_flow",
  "agent_key": "agent_abc",                      // your agent; "" runs the default pipeline
  "user_speech": { "provider": "elevenlabs", "voice": "" },
  "asr":         { "provider": "deepgram" },     // transcribes the agent's replies
  "judge":       { "provider": "openai", "model": "gpt-4o-mini" },
  "turns": [
    {
      "user": "Hi, I was double charged.",       // scripted text (spoken via TTS) — or "wav": "fixtures/x.wav"
      "expect": [
        { "type": "response_within_ms", "ms": 3000 },
        { "type": "regex",  "pattern": "refund|charge" },
        { "type": "judge",  "criterion": "acknowledges the double charge" }
      ]
    }
  ],
  "expectations": [ { "type": "judge", "criterion": "stayed polite throughout" } ],
  "timeouts": { "call_ms": 120000, "turn_ms": 20000 }
}
```

Set `agent_key` to the agent you built in the console. Or set it to `""` to
smoke-test the default speech-to-text → language → speech pipeline with no
agent configured. A turn is either scripted `user` text (spoken through the
TTS provider) or a `wav` fixture (PCM16, mono or stereo, any sample rate).
Use a `wav` fixture when you need a specific voice, accent, or noise profile.

## Assertion types

Each turn carries `expect` assertions. The call carries call-level
`expectations`, which the judge grades against the whole transcript.

- **`response_within_ms`** (`{ ms: number }`) — Latency gate: the gap from the caller's last speech frame to the agent's   first audio frame must be under `ms`. It catches a slow agent before your   users do.
- **`regex`** (`{ pattern: string }`) — Case-insensitive regex over the agent's transcript for that turn. It is   good for must-say phrases, disclosures, or IDs.
- **`judge`** (`{ criterion: string }`) — LLM-as-judge: the configured judge model returns a strict `{ "pass": bool,   "reason": string }` verdict against your plain-English `criterion`. It   **fails closed**. A malformed or flaky judge reply counts as a fail. Thus a   bad judge can never silently green a run.
- **`function_call`** (`{ name: string, args_contains?: object }`) — Assert that the agent called a   [tool](/modalities/voice/runtime/tool-calling) by the end of the turn. You   can also assert certain argument values.   > **NOTE:** > Preview — pairs with the runtime's tool-call surface.

## Metrics in the report

Every run writes a JSON report with per-turn timings. All timings are
measured client-side:

| Metric | What it is |
| ------ | ---------- |
| `latency_to_first_audio_ms` | Caller's last speech frame → agent's first audio frame (the number your callers feel). |
| `agent_turn_duration_ms` | Wall-clock length of the agent's turn. |
| `agent_audio_ms` | How much audio the agent actually spoke. |
| greeting latency | Connect → the agent's first audio out. |
| call duration | Total scripted call length. |

Pair these metrics with the live-call
[latency breakdown](/modalities/voice/observability/latency-breakdown) and
[MOS / jitter / loss](/modalities/voice/observability/mos-jitter-loss) to
compare test-bench numbers against production.

## Run an evaluation

  1. **Point at an agent**
Build and save the agent in the console, so its config is live. Note its
    `agent_key`. To smoke-test with no agent, use `agent_key: ""`. The default
    pipeline then runs off your provider keys.
  2. **Provide credentials**
The simulated caller reuses the same provider clients that your agent
    uses. Thus it needs keys for the TTS, ASR, and judge providers in your
    scenario (e.g. Deepgram, {' '}ElevenLabs, OpenAI). Set the keys in the
    environment.
  3. **Run the evaluator**
Point the evaluator at your engine and scenario. Run it. The evaluator
    connects, plays the scripted call, grades every assertion, and writes the
    report.

    ```bash
    clutchcall eval run ./scenarios/billing_flow.json --report ./out
    ```
    > **NOTE:**
> Today the evaluator ships as a standalone binary. You run it against a
>       self-hosted or managed engine. The `clutchcall eval` wrapper is
>       the convenience entry point. See
>       [Self-Hosted deployment](/modalities/voice/quickstart/on-prem) to run it
>       against your own engine.
  4. **Read the result**
The process exits `0` only if every assertion passed. The JSON report
    lands in your output directory. The report holds the per-turn metrics
    above and, for each failed assertion, the judge's `reason`.

## Gate your CI on it

The run is a single command with a meaningful exit code. Thus you can drop it
into CI to block a regression before it ships:

```yaml
- name: Evaluate voice agent
  run: clutchcall eval run ./scenarios/*.json --report ./eval-out
  # non-zero exit fails the job; upload ./eval-out for the per-turn report
```

> **TIP:**
> Keep a suite of scenarios for each agent. Write one scenario per critical
>   flow (billing, refund, escalation), plus a latency-only smoke scenario. The
>   `agent_key: ""` default pipeline gives you a zero-config smoke test that
>   needs only provider keys. Thus CI can catch a broken transport or a provider
>   outage without a saved agent.

## Related

  - **[Turn Detection & Barge-in](/modalities/voice/concepts/turn-detection)** — Tune what your latency and barge-in assertions test.
  - **[Tool Calling](/modalities/voice/runtime/tool-calling)** — Learn the tools that `function_call` assertions check for.
  - **[Runtime Configuration](/modalities/voice/runtime/configuration)** — Learn the agent config that an evaluation runs against.
  - **[Latency Breakdown](/modalities/voice/observability/latency-breakdown)** — Compare bench latency to production.