# Turn Detection & Barge-In

> Tune when your voice agent listens, talks, and yields. This covers VAD, end-of-utterance, and barge-in interrupts.

Tune when your voice agent listens, talks, and yields the floor. Turn
detection makes a call feel natural. The agent stops the instant the
caller starts to talk (barge-in). The agent replies the instant the
caller finishes (end-of-utterance). If the tuning is wrong, the call
feels broken: the agent talks over people, or it stays silent for
awkward seconds while it waits for "real" silence.

You control this with two settings: a detection mode per agent, and a
small set of tuning parameters. Pick a mode. Set the timings. The
interrupt signal is then wired end-to-end for you.

## The two detection modes

```
┌─────────────────────────────────────────────────────────────────┐
│  TurnDetection {                                                │
│    type: "silero" | "server_vad",                               │
│    silence_threshold_ms?, min_speech_ms?, prefix_padding_ms?    │
│  }                                                              │
└─────────────────────────────────────────────────────────────────┘
```

| `type`        | Who runs VAD                              | When to use                                                           |
|---------------|-------------------------------------------|-----------------------------------------------------------------------|
| `silero`      | Gateway, locally (on-device VAD)  | Cascaded ASR → LLM → TTS. Default.                                    |
| `server_vad`  | The AI provider (e.g. OpenAI Realtime)    | Realtime models that already emit `speech_started`/`speech_stopped`.  |

**`silero` is the default** because most pipelines are cascaded
(for example, Deepgram → Anthropic → ElevenLabs) and need a local
trigger. The runtime auto-promotes to `server_vad` only when the agent's
entry node is `REALTIME` AND the operator did not supply an explicit
type.

## Configure turn detection per agent

Put a `TurnDetection` block in your agent config:

```yaml
# agent-config.yml
entry_node: ASR
turn_detection:
  type: silero
  silence_threshold_ms: 500    # min silence before EOU fires
  min_speech_ms: 300           # filter out clicks / noise
  prefix_padding_ms: 200       # include this much pre-speech audio
nodes:
  ASR: { provider: deepgram, ... }
  LLM: { provider: anthropic, ... }
  TTS: { provider: elevenlabs, ... }
```

For a realtime entry node, omit the whole block. The runtime then
defaults to `server_vad`:

```yaml
entry_node: REALTIME
nodes:
  REALTIME: { provider: openai-realtime, model: gpt-4o-realtime-preview }
# turn_detection auto-promotes to {type: server_vad} based on entry_node
```

The runtime always honors an override. If you want gateway-side
detection with a realtime entry node, set `type: silero` explicitly.

## Tune when the agent replies

These are the timings that you will change most often. Start with the
defaults. Lower them for faster replies. Raise them for callers who
pause to think.

| Field                   | Default | What it does                                          | When to change                                              |
|-------------------------|---------|-------------------------------------------------------|-------------------------------------------------------------|
| `silence_threshold_ms`  | 500     | Min trailing silence before EOU fires                 | Lower (300) for snappier replies; higher (800) for thinkers |
| `min_speech_ms`         | 300     | Discard candidate utterances shorter than this        | Raise to 500 if line noise causes false triggers            |
| `prefix_padding_ms`     | 200     | Audio kept before the speech-start marker             | Raise if first syllables are getting clipped on the ASR     |
| `silence_floor_dbfs`    | -45     | Signal level below which is treated as silence        | Quieter trunks need lower (-50); noisy ones higher (-40)    |

## Know where barge-in actually works

This is an honest scorecard. Where it says "no", the agent will continue
to talk over the user. Treat this as a known limitation, not a
configuration error.

### Agent LLM/ASR/TTS providers

| Provider                        | Cancel mid-flight? | Barge trigger                                |
|---------------------------------|:------------------:|----------------------------------------------|
| OpenAI Realtime                 | ✅                 | Local VAD or `speech_started` → `truncate`   |
| OpenAI HTTP / Anthropic / Gemini / Ollama | ❌ (timeout-only) | Gateway silences playback; LLM completes server-side |
| Deepgram ASR streaming          | ✅ (forward cancel)| Cancel propagates; partial transcript discarded |
| Deepgram TTS / ElevenLabs TTS   | ✅ (clear buffer)  | Outbound PCM buffer cleared at PushAudio node |

### WebRTC / streaming vendor adapters

| Vendor      | Path                  | Interrupt signal              | Status                                       |
|-------------|-----------------------|-------------------------------|----------------------------------------------|
| **SIP/RTP** | Native PSTN trunk     | on-device VAD → bridge control      | ✅ End-to-end                                 |
| Twilio      | Media Streams         | `clear` event                 | ⚠️ Adapter doesn't yet emit it on barge       |
| Vapi        | WebSocket transport   | `stop` message                | ⚠️ Adapter doesn't yet emit it on barge       |
| LiveKit     | Room participant      | None (no clean media-plane interrupt) | ❌ Client-side action required        |
| Daily       | Room participant      | None                          | ❌ Client-side action required                |
| Chime       | Meeting attendee      | None                          | ❌ Client-side action required                |
| Browser     | WebTransport          | Caller mic → server VAD       | ✅ Same as SIP path                           |

For `❌` vendors, the AI's TTS continues to stream. FirstByte silences
the playback locally, but the AI is unaware. For latency-sensitive UX,
prefer SIP/RTP or Browser. After the adapter work lands, Twilio/Vapi
will also be good options.

## Common gotchas

- **Realtime entry node + explicit `type: silero`**: this works, but you
  do VAD twice. The OpenAI server-side VAD still fires. It can produce
  `speech_started` events that the runtime ignores. Pick one mode.
- **`silence_threshold_ms` too low (< 300)**: end-of-utterance fires on
  inter-word pauses. The agent interrupts the user mid-sentence.
- **Cascaded pipeline with `server_vad`**: only the realtime model can
  produce server-VAD events. A Deepgram → Anthropic → ElevenLabs pipeline
  with `server_vad` will never emit a turn boundary. This is a
  misconfiguration. Use `silero` instead.
- **HTTP LLM "barge" feels delayed**: Anthropic/Gemini/OpenAI HTTP run
  to completion server-side. FirstByte silences the audio playback,
  but the server still bills you for the full response. For tight
  barge-in budgets, use a Realtime model.

## How it works

Barge-in feels instant because the interrupt travels one control path
end to end. You configure the mode. The plumbing is already wired.

When local VAD trips, this chain fires:

```
caller starts speaking
       │
       ▼
VAD source: on-device VAD on the gateway (SIP/RTP path)
            OR vendor server-VAD via the agent runtime (direct-media path)
       │
       │  control frame: [0xFFFFFFFF sentinel][type=BARGE_IN][session_id]
       ▼
agent_runtime_bridge → runtime
       │
       ▼
DagExecutor::on_cancel(CancelEvent{source: user_vad})
       │
       ├──▶ ASR / LLM / TTS nodes: forward cancel
       ├──▶ PushAudio node: clear outbound buffer
       └──▶ RealtimeNode: emit `conversation.item.truncate` to OpenAI,
                          close session
```

The same chain fires on `source: server_vad` when the AI provider
reports that the user spoke first (e.g. OpenAI Realtime's
`input_audio_buffer.speech_started`). Both sources land in the same
handler. Node code is thus detection-mode-agnostic.

## Where to dig further

- `get_page rpc/audio-frames` — wire format for the PCM/PCMU stream that
  feeds the gateway VAD.
- `get_page rpc/method-ids` — `Barge` method ID for operator-initiated
  interrupts.
- `get_page admin/agent-dags` — full `TurnDetection` schema in the agent
  config.