# Turn Detection & Barge-in

> Tune when your voice agent listens, replies, and yields the floor. Covers VAD modes, end-of-turn timing, and hold-and-confirm barge-in.

Turn detection makes a call feel human. Your agent should stop the instant the
caller starts to talk (barge-in). It should reply the instant the caller
finishes (end-of-turn). It should not cut off a caller who only paused to
think. It should not cancel itself when the caller mutters "mhm" to keep it
going.

To get this, pick a detection mode per agent and tune a small set of timings.
The interrupt signal is wired end to end for you across every transport:
PSTN, browser, and the WebRTC fallback leg. The same configuration behaves the
same way on a phone call and in a browser tab.

## Three ways a turn gets detected

Every agent uses exactly one of these. The right choice follows from the shape
of your pipeline.

| Mode | Who decides the turn | Use it when |
|------|----------------------|-------------|
| **On-device VAD** (default) | The runtime, on the caller's audio frames | Cascaded ASR → LLM → TTS agents (for example Deepgram → your LLM → ElevenLabs). Works on any transport. |
| **`server_vad`** | The realtime provider's own endpointing | Realtime speech-to-speech models that already emit speech-start / speech-stop events (OpenAI Realtime, Gemini Live). |
| **Client-commit** | Your app / the runtime's local detector | Self-hosted speech-to-speech models that expose no server-side VAD — turn boundaries are committed explicitly. |

On-device VAD is the default because most agents are cascaded. They need a
local trigger to know when to hand the utterance to the ASR stage. When an
agent's entry node is a realtime provider and you do not set a mode
explicitly, the runtime auto-promotes the default to `server_vad`. The
realtime session owns the audio path end to end, so a local detector would
never actually run.

> **NOTE:**
> Two on-device backends exist: a lightweight built-in energy gate
> (`type: "energy"`) and a higher-accuracy on-device VAD model. You get the VAD
> model by default when you omit `type`. Both feed the identical turn-taking
> and barge-in logic below. A backend switch never changes your timing knobs.

## Configure turn detection per agent

Turn detection lives in your agent config under `turn_detection`. Every field
is optional. Omit the block, and you get the phone-safe defaults.

**Cascaded agent (on-device VAD):**
```json
{
  "entry_node": "ASR",
  "pipeline": { "ASR": { }, "LLM": { }, "TTS": { } },
  "turn_detection": {
    "prefix_padding_ms": 300,
    "silence_duration_ms": 500,
    "params": {
      "barge_confirm_ms": 300,
      "response_min_speech_ms": 600,
      "answer_min_speech_ms": 150
    }
  }
}
```

**Realtime agent (server_vad):**
```json
{
  "entry_node": "REALTIME",
  "pipeline": { "REALTIME": { } }
  // turn_detection omitted → auto-promoted to { "type": "server_vad" }
}
```

You can always override the auto-promotion. Set `"type": "energy"` on a
realtime agent to force local detection instead. An explicit type wins over
the default in every case.

Set this on the agent from your workspace console at `agent.firstbyte.fast`,
or push it through the control-plane API. See
[Runtime configuration](/modalities/voice/runtime/configuration) for the full
agent-config shape.

## Tune when the agent replies

Start with the defaults, then use these fields. Lower values give faster
replies. Higher values help callers who pause to gather their thoughts.

| Field | Default | What it does | When to change |
|-------|:-------:|--------------|----------------|
| `silence_duration_ms` | 500 | Trailing silence before end-of-turn fires | Lower (~300) for faster replies; raise (~800) for deliberate callers |
| `prefix_padding_ms` | 300 | Pre-speech audio kept ahead of the speech-start marker | Raise if the ASR clips first syllables |
| `threshold` | 0.5 | Speech-probability gate for the on-device VAD model | Raise on noisy trunks to reject background chatter |
| `params.answer_min_speech_ms` | 150 | Shortest burst that counts as a real answer after the agent yields | Keep low so terse replies ("yes", a digit) still commit |
| `params.response_min_speech_ms` | 600 | Shortest burst that counts as a real interruption *over* the agent | Raise to suppress more backchannels; lower to barge more eagerly |
| `params.silence_floor` | 5 | Accumulator level treated as "quiet again" | Rarely touched; lower on very quiet lines |

The split between `answer_min_speech_ms` and `response_min_speech_ms` is
deliberate. A short burst means opposite things at different moments. In the
answering window, the agent stopped and waits. There, even a one-syllable
"yes" is a real turn. Over the agent while it talks, that same short burst is
almost always a continuer ("mhm", "ok", "right"). The over-agent floor is
therefore higher, and backchannels never cancel the response.

## Hold-and-confirm barge-in

Barge-in is armed automatically whenever the agent speaks. The mechanism is
**hold-and-confirm**, controlled by `params.barge_confirm_ms` (default `300`):

1. **Onset arms a pending barge**
The instant the caller starts to speak over the agent, the runtime arms a
*pending* barge. It does not cancel the agent yet.
2. **Sustained speech confirms it**
The runtime fires the interrupt and cancels the in-flight response only if
that speech continues past `barge_confirm_ms`.
3. **Backchannels self-cancel**
A short "mhm" or "ok" ends before the window elapses. The runtime drops the
pending barge, and the agent continues to talk. The only cost of a real
interruption is the ~`barge_confirm_ms` of extra latency before the cut.

Set `barge_confirm_ms: 0` to disable the hold. The interrupt then fires on the
very first frame of over-talk. Use `0` only on transports with client-side
echo cancellation (the browser path). There, a fast cut is wanted, and the
agent's own audio cannot bleed back into the mic.

> **WARNING:**
> There is no acoustic echo cancellation in the telephony media path. On PSTN,
> the agent's own TTS would otherwise leak back through the caller's mic and
> self-trigger a barge. To prevent this, the runtime raises the during-speech
> mic gate (`params.margin_while_tts`) to match the quiet-line gate
> (`params.margin`) on phone calls. The agent's voice can then never cancel the
> agent. On the browser path, where the client cancels echo, you can lower
> `margin_while_tts` for a faster barge.

> **NOTE:**
> Backchannel suppression here is **acoustic only**. It is based on how long the
> caller speaks, not on what they say. Lexical gating (to treat "stop"
> differently from "uh-huh") and provider-side semantic endpointing are not
> built today. If you need word-aware interrupts, handle them in your own tool
> logic.

## Where barge-in actually works

Barge-in quality depends on whether the runtime can cancel the downstream
provider *mid-flight*. The on-device VAD always detects the interruption. The
honest question is whether the agent's audio actually stops cleanly. A
"partial" row is a known limitation, not a misconfiguration.

| Provider path | Mid-flight cancel | What happens on barge |
|---------------|-------------------|-----------------------|
| OpenAI Realtime / Gemini Live (`server_vad`) | Clean | Runtime sends the provider's interrupt/truncate; the session stops generating |
| Streaming TTS (ElevenLabs, Deepgram, and similar) | Clean | The runtime flushes the outbound audio buffer within a frame; playback stops immediately |
| Streaming ASR | Clean | Cancel propagates; the runtime discards the partial transcript |
| HTTP (non-streaming) LLM | Partial | The runtime silences playback locally, but the request runs to completion server-side, and the provider still bills it |

And by transport leg, where the caller's audio actually comes from:

| Transport | Interrupt source | Status |
|-----------|------------------|--------|
| SIP / RTP (PSTN trunk) | On-device VAD on the caller's RTP → runtime cancel | End to end |
| Browser (WebTransport / media-over-QUIC) | Caller mic decoded → on-device VAD | Same path as PSTN |
| WebRTC fallback leg | Server-side WebRTC termination → on-device VAD | Same path as PSTN |
| External vendor bridge (LiveKit / Vapi / Twilio) | Depends on the vendor's own interrupt event | Varies by adapter |

When you route media through an external agent-orchestration vendor, not the
built-in pipeline, the barge signal is only as good as that vendor's adapter.
For the tightest interrupt budgets, keep the media on the native SIP/RTP,
browser, or WebRTC-fallback path.

## Idle watchdog

The flip side of barge-in is silence. When both the caller and the agent go
quiet, the idle watchdog can re-engage or end the call. Set it under
`turn_detection.idle`:

- `soft_prompt_s` — after this many idle seconds, the runtime nudges the model
  to speak (`soft_prompt_text`, or a bare re-engage on a realtime model).
- `hangup_s` — after this many idle seconds, the runtime drops the AI leg.

Both default to `0` (off) in the raw schema. But the runtime ships sensible
ceilings: a re-engage prompt, a hangup, a capped response length, and a hard
maximum session duration. A stuck agent therefore cannot burn a call forever.
See [Session lifecycle](/modalities/voice/runtime/session-lifecycle) for those
guardrails.

## Common gotchas

> **WARNING:**
> **`silence_duration_ms` too low (< 300 ms).** End-of-turn fires on the natural
> pauses between words, so the agent replies mid-sentence. Raise the value
> before you blame the ASR.

> **WARNING:**
> **`server_vad` on a cascaded agent.** Only a realtime provider emits
> server-side VAD events. A Deepgram → LLM → ElevenLabs agent set to
> `server_vad` will never see a turn boundary and will sit silent. Use
> on-device VAD (omit `type`) instead.

> **WARNING:**
> **Delayed barge on an HTTP LLM.** Non-streaming LLM calls run to completion
> server-side. The runtime silences playback the moment the caller barges. But
> the provider still finishes, and bills, the full response. For tight barge
> budgets, use a streaming or realtime model.

## Related

- **[Runtime configuration](/modalities/voice/runtime/configuration)** — See the full agent-config shape, including where `turn_detection` lives.
- **[Handoffs](/modalities/voice/concepts/handoffs)** — Learn what a barge-in and end-of-turn feed into when you transfer to a human.
- **[Bring your own speech-to-speech](/modalities/voice/runtime/byo-speech-to-speech)** — Use the client-commit path for self-hosted realtime models with no server VAD.
- **[Debug barge-in](/modalities/voice/observability/barge-in-debugging)** — Trace why an interrupt did or did not fire on a live call.