# BYO ASR / LLM / TTS Models

> Assemble a best-of-breed cascade from one config file. For example, use Deepgram, your LLM, and ElevenLabs or Cartesia.

Run a **cascaded** agent when you want to pick the best model for each stage instead of a single duplex model. The runtime chains three provider sessions per call:

```
caller audio (PCM16, 8 kHz)
   -> ASR   (speech -> text)          e.g. Deepgram
   -> LLM   (text  -> reply text)     e.g. OpenAI / Anthropic / Gemini / Ollama
   -> TTS   (text  -> speech)         e.g. ElevenLabs / Cartesia
   -> caller audio
```

You choose each provider independently in one `agent-config.yml`. You drop in the credentials. The FirstByte Agent Runtime then opens a streaming session with each vendor on the caller's shard. You write no code. The runtime assembles the pipeline from config.

> **NOTE:**
> Do you prefer a single low-latency duplex model? See [BYO speech-to-speech](/modalities/voice/runtime/byo-speech-to-speech). That path sets `pipeline_mode: realtime` and hands the whole turn to one provider (OpenAI Realtime, Gemini Live, and others). A cascade gives you more control and lets you swap each stage. Speech-to-speech gives you the lowest turn latency.

## Set the pipeline

1. **Choose your three providers**
Pick one id from each table below. A bare cascade falls back to the defaults: `deepgram` for ASR, `openai` for LLM, and `elevenlabs` for TTS. Thus you can start and override only the stage you care about.

2. **Add credentials**
Each provider reads a per-tenant key. The runtime resolves the key at call setup (see [Credentials](#credentials)). Set the keys for the providers you selected. The runtime refuses to start an agent that has no usable provider key.

3. **Write agent-config.yml**
Set `pipeline_mode: cascaded` and the per-stage provider/model/voice fields. The full example is below.

4. **Save and place a test call**
The control plane hydrates the config into the runtime. No rebuild is necessary. Place a call. Then watch the [latency breakdown](/modalities/voice/observability/latency-breakdown) to see per-stage timing (ASR final, LLM first token, TTS first audio).

## agent-config.yml

```yaml agent-config.yml
name: support-agent
pipeline_mode: cascaded          # ASR -> LLM -> TTS chain (vs. "realtime")

# --- Speech-to-text ---
asr_provider: deepgram
asr_model: nova-2                # forwarded onto the ASR socket

# --- Language model ---
llm_provider: openai
llm_model: gpt-4o-mini
temperature: 0.4
conversation_memory_messages: 20 # rolling history the runtime replays each turn
llm_max_retries: 2               # retry the primary provider on a failed turn...
llm_fallback_provider: anthropic # ...then fall back once to this provider

# --- Text-to-speech ---
tts_provider: elevenlabs
tts_voice: <voice-id>            # ElevenLabs voice_id / Cartesia UUID / Deepgram Aura voice

# --- Shared ---
language: en                     # ISO 639-1 hint; pins the agent's output language
system_prompt: >
  You are a friendly support agent for Acme. Keep replies short and spoken-friendly.
```

The runtime processes audio as **PCM16 LE at 8 kHz** end to end. The runtime feeds and drains every provider you pick at that rate. It opens Deepgram with `encoding=linear16`. Some providers natively want 16/24 kHz; the runtime resamples for them.

## ASR providers

Set `asr_provider`. You can also set `asr_model`; the runtime forwards it as the provider's model hint.

| `asr_provider` | Vendor | Notes |
| --- | --- | --- |
| `deepgram` *(default)* | Deepgram | Streaming, interim + final transcripts. Default model `nova-2`. |
| `assemblyai` | AssemblyAI | Universal-Streaming v3. |
| `speechmatics` | Speechmatics | Ursa realtime. |
| `soniox` | Soniox | Token-level finality. |
| `gladia` | Gladia | Solaria live — multilingual and code-switching. |
| `gradium` | Gradium | Native 8 kHz telephony input. |
| `sarvam` | Sarvam.ai | Saarika — Indic languages. |
| `azure` | Azure Speech | Continuous Recognition. Region required with the key. |
| `smallest` | smallest.ai | Pulse STT, 36 languages. |
| `cartesia` | Cartesia | Ink streaming STT. |
| `elevenlabs` | ElevenLabs | Streaming ASR surface (shares your ElevenLabs key). |
| `xai` | xAI | Grok STT. |
| `local-whisper` | self-hosted | Whisper on your own `mod_whisper` endpoint. No key. |

Batch ids also exist — `groq`, `openai`, `mistral`, `together`, `fal`, `runpod` — which transcribe after the utterance ends. They cost less, but the caller waits for the whole turn, so prefer a streaming id on a live call.

> **NOTE:**
> Deepgram is the recommended default for English telephony. See [Supported providers](/modalities/voice/integrations/providers) for every id and what each one is good at, and [Self-hosted inference](/modalities/voice/runtime/self-hosted-inference) for running transcription on your own hardware.

## LLM providers

Set `llm_provider` and `llm_model`. Tool calling runs in the LLM stage. See [Tool calling](/modalities/voice/runtime/tool-calling).

```yaml
llm_provider: openai      # default when llm_provider is unset
llm_model: gpt-4o-mini
```

```yaml
llm_provider: anthropic
llm_model: claude-sonnet-4-5
```
This provider honors in-flight cancellation. A barge-in stops the running request, so you do not pay for a discarded turn.

```yaml
llm_provider: gemini
llm_model: gemini-2.5-flash
```

```yaml
llm_provider: ollama
llm_model: llama3.1
```
This provider points at a local Ollama endpoint (`OLLAMA_ENDPOINT`, default `127.0.0.1:11434`). No API key is necessary. Use this option to keep the language model on your own infrastructure while you use cloud ASR/TTS.

Beyond the four above, these ids work the same way: `groq`, `cerebras`, `mistral`, `together`, `fireworks`, `openrouter`, `deepseek`, `perplexity`, `xai`, `mod_inference` (our own gateway), and `openai-compatible` for any endpoint speaking the OpenAI chat API. See [Supported providers](/modalities/voice/integrations/providers).

> **TIP:**
> If the primary provider fails after `llm_max_retries`, `llm_fallback_provider` runs a different provider entirely. Thus a vendor incident degrades one turn and does not drop the call. This is a resilience path, not a load balancer.

## TTS providers

Set `tts_provider`. Put the provider's voice id in `tts_voice`.

| `tts_provider` | Vendor | Voice field |
| --- | --- | --- |
| `elevenlabs` *(default)* | ElevenLabs | `tts_voice` = ElevenLabs `voice_id` (streaming, low first-audio). |
| `cartesia` | Cartesia (Sonic) | `tts_voice` = Cartesia voice UUID (streaming WebSocket). |
| `cartesia-http` | Cartesia | Same voices, one-shot HTTP fallback (no streaming). |
| `deepgram` | Deepgram (Aura) | `tts_voice` = Aura voice, e.g. `aura-asteria-en`. |
| `rime` | Rime | `tts_voice` = Rime voice. Mist / Arcana conversational voices, streaming. |
| `lmnt` | LMNT | Aurora streaming TTS. |
| `smallest` | smallest.ai | Lightning v3.1 — multilingual, very low time-to-first-audio. |
| `sarvam` | Sarvam.ai | Bulbul — Indic voices. |
| `azure` | Azure Speech | Region required alongside the key. |
| `xai` | xAI | Grok TTS, streaming. |
| `openai` | OpenAI | Reuses your OpenAI key; one-shot per utterance. |
| `local-kokoro` | self-hosted | Kokoro through `mod_tts` on loopback. No key. |
| `runpod` | self-hosted | Your own OpenAI-compatible speech endpoint. |

> **NOTE:**
> Streaming TTS providers emit audio as the LLM text arrives. This keeps first-audio low. Sometimes a provider cannot open a streaming session, for example because of a missing voice or a bad key. Then the runtime falls back to a one-shot synthesis for that turn, so the caller still hears a reply.

## Credentials

Provider keys are **per-tenant**. Set them in the FirstByte console at `agent.firstbyte.fast`. The control plane resolves the keys at call setup, seals them at rest, and injects them into the pipeline. The keys are never baked into `agent-config.yml`.

For single-tenant or local development, the runtime also reads keys from the environment as a fallback:

**Cloud providers:**
```bash
export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export GEMINI_API_KEY=...
export DEEPGRAM_API_KEY=...
export ELEVENLABS_API_KEY=...
export CARTESIA_API_KEY=...
```
**Self-hosted LLM:**
```bash
# Ollama needs no key — just the endpoint
export OLLAMA_ENDPOINT=127.0.0.1:11434
```

> **WARNING:**
> If an agent's selected providers have no usable key, the runtime refuses the agent at session creation with a clean error. The agent does not start and then fail mid-call. Check that each stage's key is present before you route live traffic to the agent.

## What runs around the cascade

- **Turn detection.** A cascade uses **local, on-device VAD** by default. The runtime owns endpointing and barge-in; it does not use a cloud provider's server VAD. Tune the thresholds, the barge-in hold window, and the idle watchdog on the [turn detection](/modalities/voice/concepts/turn-detection) page.
- **Language.** The `language` hint prepends a hard directive to the system prompt. The model then answers in your market's language and does not mirror the caller.
- **Session limits.** Connect, idle, and max-session deadlines apply per stage. See [Session lifecycle](/modalities/voice/runtime/session-lifecycle).

> **NOTE:**
> Backchannel suppression in a cascade is **acoustic**. It uses utterance duration plus a hold-and-confirm barge window. Short "mhm"/"ok" replies do not interrupt the agent. Lexical gating ("stop" vs "ok") is not built. Plan around the acoustic behavior. See [turn detection](/modalities/voice/concepts/turn-detection).

## Next steps

- **[Runtime configuration](/modalities/voice/runtime/configuration)** — This reference lists every knob: turn detection, budgets, guardrails, and codecs.
- **[Speech-to-speech](/modalities/voice/runtime/byo-speech-to-speech)** — Trade per-stage control for the lowest turn latency with a single duplex model.
- **[Tool calling](/modalities/voice/runtime/tool-calling)** — Give the LLM stage HTTP, MCP, and client tools.
- **[Latency breakdown](/modalities/voice/observability/latency-breakdown)** — See per-stage ASR / LLM / TTS timing on a live call.