# Deepgram

> Use Deepgram for streaming speech-to-text in a cascaded voice agent, with low-latency interim transcripts and barge-in.

Deepgram is the default speech-to-text provider for a cascaded FirstByte
Voice agent. When you run an ASR -> LLM -> TTS pipeline, the caller's audio
streams into Deepgram over a persistent socket. Deepgram returns interim and
final transcripts. The runtime sends the transcripts to the language model as
soon as the caller stops talking. This integration is **shipped**. A bare
cascade with no `asr_provider` set already uses Deepgram.

You set one config field. You add a key. You place a call. You do not write
code and you do not rebuild.

## Turn it on

1. **Select Deepgram for the ASR stage**
In your `agent-config.yml`, set `asr_provider: deepgram` and select a model. If
you do not set `asr_provider`, the cascade uses Deepgram anyway. Thus the main
purpose of this step is to select the model.

```yaml agent-config.yml
name: support-agent
pipeline_mode: cascaded

asr_provider: deepgram
asr_model: nova-2          # forwarded onto the ASR socket

llm_provider: openai
llm_model: gpt-4o-mini
tts_provider: elevenlabs
tts_voice: <voice-id>
language: en
```

2. **Add your Deepgram key**
Keys are **per-tenant**. Set `DEEPGRAM_API_KEY` in the FirstByte console at
`agent.firstbyte.fast`. The control plane seals the key at rest and resolves it
at call setup. The key is never stored in the config document. For
single-tenant or local development, the runtime also reads the key from the
environment:

```bash
export DEEPGRAM_API_KEY=...
```

> **WARNING:**
> If the ASR provider has no usable key, the runtime refuses the agent at session
> creation with a clean error. The agent does not start and then fail mid-call.
> Check that the key is present before you route live traffic.

3. **Place a test call**
The control plane hydrates the config into the runtime with no rebuild. Place a
call. Then open the [latency breakdown](/modalities/voice/observability/latency-breakdown)
to see ASR-final timing for each turn.

## What the runtime sends Deepgram

The FirstByte Agent Runtime opens one streaming Deepgram session for each
call on the caller's shard. It forwards each 8 kHz PCM16 audio frame as the
frame arrives off the media plane. The runtime consumes Deepgram's streaming
results: interim hypotheses while the caller speaks, then a final transcript at
the endpoint. The runtime sends the final text to the LLM stage. The runtime
owns endpointing (see below). Thus it drives Deepgram in streaming mode and
does not delegate turn-taking to the transcriber.

The runtime passes the `asr_model` value directly to Deepgram. Any current
Deepgram streaming model id works:

```yaml
asr_provider: deepgram
asr_model: nova-2
```
Deepgram's general-purpose streaming model. This is the recommended default for
English telephony.

```yaml
asr_provider: deepgram
asr_model: nova-3
```
A newer model id. The runtime forwards it verbatim to Deepgram. Check accuracy
on a staging number before you switch production traffic.

```yaml
asr_provider: deepgram
asr_model: nova-2-phonecall   # or another Deepgram model id
```
You can select any streaming model that Deepgram exposes by its id. The runtime
does not keep an allow-list.

## Barge-in

Barge-in works with a Deepgram cascade. The runtime's **local, on-device VAD**
handles endpointing and interruption, not the transcriber. Thus the caller can
talk over the agent, and the agent stops. This behavior does not depend on your
ASR choice.

The runtime uses **hold-and-confirm** barge-in. A short backchannel ("mhm",
"ok") does not cancel the agent. Sustained speech past the confirm window
cancels the agent. Tune the behavior in the turn-detection block:

```yaml agent-config.yml
turn_detection:
  type: local               # on-device VAD owns endpointing + barge-in
  silence_threshold_ms: 500
  min_speech_ms: 300
  barge_confirm_ms: 300     # sustain speech past this to interrupt the agent
  response_min_speech_ms: 600
```

> **NOTE:**
> Backchannel suppression is **acoustic**. It uses utterance duration and the
> hold-and-confirm window, not the words spoken. Lexical gating ("stop" vs "ok")
> is not built. See [turn detection](/modalities/voice/concepts/turn-detection)
> for the full set of controls. See
> [barge-in debugging](/modalities/voice/observability/barge-in-debugging)
> to examine what fired on a live call.

## Deepgram for text-to-speech

Deepgram also ships as a **text-to-speech** provider (the Aura voices). To use
Deepgram on both ends of the cascade, also set the TTS stage:

```yaml agent-config.yml
tts_provider: deepgram
tts_voice: aura-asteria-en
```

See [BYO ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts) for the full
provider matrix across all three stages.

## Related

- **[BYO ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts)** — Assemble the full cascade with every ASR, LLM, and TTS provider in one config file.
- **[Turn detection](/modalities/voice/concepts/turn-detection)** — The endpointing and barge-in model that runs in front of Deepgram.
- **[Runtime configuration](/modalities/voice/runtime/configuration)** — Every agent-config.yml field: providers, turn detection, budgets, codecs.
- **[Latency breakdown](/modalities/voice/observability/latency-breakdown)** — See per-stage ASR / LLM / TTS timing on a live call.
