# Runtime Configuration > The agent-config.yml reference: model and provider fields, the turn-detection block, codecs, session and output budgets, and guardrails. Every agent is a single config document. The document names the pipeline shape and the provider and model for each stage. It sets how the runtime decides that the caller finished talking. It sets the budgets that bound a call. You edit the document in the console at `agent.firstbyte.fast` or push it through the control-plane API. The gateway hydrates the document into the runtime at call setup. It hot-reloads tunables **without dropped in-flight calls**. There is no rebuild and no redeploy. Save the document, then place a call. This page is the field-by-field reference. If you wire a pipeline for the first time, start with [BYO ASR/LLM/TTS](/modalities/voice/runtime/byo-asr-llm-tts) (cascade) or [BYO speech-to-speech](/modalities/voice/runtime/byo-speech-to-speech) (duplex). Come back here to tune the knobs. ## The shape of agent-config.yml A cascaded agent is the most common shape. It looks like this. Every field below has a default. Thus a minimal config is only a name, a system prompt, and the providers you want to override. ```yaml agent-config.yml name: support-agent pipeline_mode: cascaded # cascaded (ASR → LLM → TTS) | realtime (duplex) # --- Models & providers --- asr_provider: deepgram asr_model: nova-2 llm_provider: openai llm_model: gpt-4o-mini temperature: 0.4 conversation_memory_messages: 20 llm_max_retries: 2 llm_fallback_provider: anthropic tts_provider: elevenlabs tts_voice: language: en system_prompt: > You are a friendly support agent for Acme. Keep replies short and spoken-friendly. # --- Turn detection --- turn_detection: type: local # local (on-device VAD) | server_vad silence_threshold_ms: 500 min_speech_ms: 300 prefix_padding_ms: 200 barge_confirm_ms: 300 # hold-and-confirm barge-in window response_min_speech_ms: 600 # --- Budgets & guardrails --- max_session: 15m # hard ceiling — the runtime hangs up at this max_output_tokens: 4096 idle: soft_prompt_s: 20 # re-engage prompt after this much silence hangup_s: 60 # then end the call soft_prompt_text: "Are you still there?" ``` A **duplex** agent replaces the cascade fields with a single realtime node. See [Realtime (duplex) agents](#realtime-duplex-agents) below. Everything in the turn-detection and budgets sections applies to both shapes. ## Models & providers Pick the pipeline shape with `pipeline_mode`. Then fill in the stage fields for that shape. The runtime processes audio as **PCM16 LE at 8 kHz** end to end. The runtime feeds and drains every provider at that rate. Some providers natively want 16/24 kHz; the runtime resamples for them. - **``** (`string`, default `cascaded`) — `cascaded` chains three provider sessions (speech-to-text → language model → text-to-speech). `realtime` hands the whole turn to one duplex speech-to-speech model. Choose the cascade for per-stage control and the widest model choice. Choose realtime for the lowest turn latency. - **``** (`string`, required) — A human-readable agent name. The console shows this name. The system stamps it onto call records, so you can tell agents apart in the [dashboards](/modalities/voice/observability/dashboards). - **``** (`string`) — The agent's persona and task. Keep replies "spoken-friendly". Short sentences read better than paragraphs over a phone line. - **``** (`string`, default `en`) — ISO 639-1 hint. It prepends a hard directive to the system prompt. The model then answers in your market's language and does not mirror the caller. ### Cascade fields (`pipeline_mode: cascaded`) - **``** (`string`, default `deepgram`) — Streaming speech-to-text vendor. Supported ids include `deepgram` and `elevenlabs`. The full table is on [BYO ASR/LLM/TTS](/modalities/voice/runtime/byo-asr-llm-tts#asr-providers). - **``** (`string`, default `nova-2`) — The runtime forwards this value to the ASR vendor as its model hint. Deepgram opens the socket with 8 kHz `linear16`. - **``** (`string`, default `openai`) — Language-model vendor: `openai`, `anthropic`, `gemini`, or `ollama` (self-hosted, no key). [Tool calling](/modalities/voice/runtime/tool-calling) runs in this stage. - **``** (`string`) — Vendor model id, e.g. `gpt-4o-mini`, `claude-sonnet-4-5`, `gemini-2.5-flash`, `llama3.1`. - **``** (`number`, default `0.4`) — Sampling temperature for the language model. - **``** (`integer`, default `20`) — Rolling history that the runtime replays into each turn. A larger value keeps more context, but costs prompt size and latency. - **``** (`integer`, default `2`) — The number of retries against the primary language-model provider on a failed turn, before the runtime falls back. - **``** (`string`) — A different provider entirely. The runtime tries it once after `llm_max_retries` is exhausted. This is a resilience path (it degrades one turn and does not drop the call), not a load balancer. - **``** (`string`, default `elevenlabs`) — Text-to-speech vendor: `elevenlabs`, `cartesia`, `deepgram`, or `openai`. Streaming providers emit audio as the LLM text arrives. This keeps first-audio low. - **``** (`string`) — The provider's voice id. This is an ElevenLabs `voice_id`, a Cartesia voice UUID, or a Deepgram Aura voice such as `aura-asteria-en`. ### Realtime (duplex) agents Set `pipeline_mode: realtime`. Describe a single realtime node instead of the three cascade stages. There is no separate ASR or TTS to configure. One model listens and speaks on the same bidirectional session. ```yaml agent-config.yml name: support-agent pipeline_mode: realtime entry_node: REALTIME pipeline: REALTIME: node_type: REALTIME provider: openai model: gpt-realtime voice: shimmer # alloy | shimmer | … instructions: | You are a friendly support agent for Acme. Keep replies short. ``` ```yaml agent-config.yml name: support-agent pipeline_mode: realtime entry_node: REALTIME pipeline: REALTIME: node_type: REALTIME provider: gemini model: gemini-2.0-flash-exp voice: Kore # Aoede | Charon | Fenrir | Kore | Puck instructions: | You are a friendly support agent for Acme. Keep replies short. ``` > **NOTE:** > Cloud duplex providers (OpenAI Realtime, Gemini Live, xAI Grok Voice) are > shipped. Support to serve your **own** open-weights duplex model on your > GPUs is in preview. See > [self-hosted inference](/modalities/voice/runtime/self-hosted-inference). ### Credentials Provider keys are **per-tenant**. They are never written into `agent-config.yml`. Set them in the console at `agent.firstbyte.fast`. The control plane resolves them at call setup, seals them at rest, and injects them into the pipeline. If an agent's selected providers have no usable key, the runtime **refuses the agent at session creation** with a clean error. The agent does not start and then fail mid-call. Environment-variable fallback for single-tenant and local development is covered under [Credentials](/modalities/voice/runtime/byo-asr-llm-tts#credentials). ## The turn_detection block Turn detection makes a call feel natural. The agent stops the instant the caller barges in. The agent replies the instant the caller finishes. Turn detection has two parts: a detection **mode** (who runs voice activity detection) and a small set of **timings**. Start with the defaults and adjust from there. The concepts page, [Turn Detection & Barge-In](/modalities/voice/concepts/turn-detection), explains the behavior end to end. This section is the field reference. - **``** (`string`, default `local`) — `local` runs an **on-device VAD** on the gateway. It is the default and the right choice for cascaded pipelines. `server_vad` defers end-of-turn detection to a cloud realtime provider that already emits speech-start/stop events. The runtime auto-selects `server_vad` for a `realtime` entry node, unless you set `type` explicitly. A cascade with `server_vad` never sees a turn boundary. Use `local` there. - **``** (`integer`, default `500`) — Minimum trailing silence before end-of-utterance fires. Lower the value (300) for snappier replies. Raise the value (800) for callers who pause to think. Below approximately 300, the agent starts to interrupt on inter-word pauses. - **``** (`integer`, default `300`) — The runtime discards candidate utterances shorter than this value. Raise the value toward 500 if line noise or clicks cause false triggers. - **``** (`integer`, default `200`) — The audio kept before the speech-start marker. Raise the value if the ASR input clips the first syllables. - **``** (`integer`, default `-45`) — The signal level (dBFS) below which the runtime treats the frame as silence. Quieter trunks want a lower floor (-50). Noisy trunks want a higher floor (-40). - **``** (`integer`, default `300`) — The **hold-and-confirm** barge-in window. A speech onset arms a *pending* barge. The barge cancels the agent only if speech sustains past this window. Thus a backchannel ("mhm", "ok", "yeah") hits silence first and never cuts the agent off. Set the value to `0` for legacy immediate barge-in. - **``** (`integer`, default `600`) — The minimum speech duration before the caller's turn is allowed to trigger a response. This is a coarser duration filter than `min_speech_ms`. It applies to whole turns, not to candidate frames. - **``** (`number`) — The sensitivity margin on the detection threshold while the agent is **idle**. - **``** (`number`) — The sensitivity margin while the **agent is speaking**. The telephony-safe default is equal to `margin`. The gate is *not* loosened during agent speech. There is no echo cancellation in the media path, so a loosened gate would let the agent's own voice self-trigger a barge. Lower this value only if you have echo-cancelled audio. > **WARNING:** > Backchannel suppression is **acoustic only**. It uses utterance duration > plus the hold-and-confirm window. Lexical gating (it would tell "stop" apart > from "ok") is not built. Plan around the acoustic behavior. See > [Turn Detection & Barge-In](/modalities/voice/concepts/turn-detection). ## Codecs Codec selection is **negotiated per transport at call setup**. It is not set in `agent-config.yml`. The runtime works in a single 8 kHz PCM16 contract internally. Thus the same agent behaves the same, no matter what the caller's leg negotiates. | Caller leg | Wire codec | Inside the runtime | |------------|-----------|--------------------| | Phone (PSTN / SIP trunk) | G.711 µ-law (PCMU) / A-law (PCMA) at 8 kHz | decoded to PCM16 | | Browser / mobile app | Opus over QUIC | decoded and resampled to PCM16 | > **NOTE:** > Opus runs **narrowband** for telephony. The encoder's internal rate is > 8 kHz to match the runtime contract. Its RTP payload still keeps the > standard `opus/48000/2` rtpmap and 48 kHz clock (a 960-tick advance per > 20 ms frame). A 20 ms tick is 160 samples at 8 kHz. Audio is 8 kHz mono end > to end. There is no video codec in the voice path. For details, and for how > to influence carrier negotiation, see > [Codecs](/modalities/voice/transport-telephony/codecs). ## Session & output budgets These budgets bound how long a call can run and how much a single turn can cost. The runtime enforces them. You do not have to build a watchdog. The [Session Lifecycle](/modalities/voice/runtime/session-lifecycle) page covers the full state machine. - **``** (`duration`, default `15m`) — The hard ceiling on total call duration. The runtime ends the session when the call reaches this ceiling. A stuck or runaway agent cannot hold a line open indefinitely. This is a ceiling, not a target. Most calls end well before it. - **``** (`integer`, default `4096`) — The cap on the language model's output per response. It bounds token spend on any single turn (previously unbounded). Lower the cap for terse agents. Keep headroom if the agent reads back long confirmations. ### Idle watchdog The idle watchdog re-engages a silent caller. If the caller is truly gone, the watchdog then hangs up. It is **on by default**. The same knobs apply to cascade and duplex agents. - **``** (`integer`, default `20`) — The silence, in seconds, before the agent emits a re-engage prompt ("Are you still there?"). The prompt fires once per idle stretch. - **``** (`integer`, default `60`) — The silence, in seconds, before the runtime ends the call. The runtime measures it from the start of the idle stretch, so it must exceed `soft_prompt_s`. - **``** (`string`) — The text spoken (or synthesized) as the re-engage prompt. Leave the field unset to get the runtime default. > **NOTE:** > Connect, idle, and per-stage turn timeouts also exist as operator-tunable > runtime settings. They hot-reload without dropped calls. In-flight sessions > keep the timeouts they captured at creation time. Thus a reload never > changes the rules mid-call. If you run self-hosted, these settings are > documented with the > [deployment models](/modalities/voice/concepts/deployment-models). ## How a config change reaches a live call 1. **Edit the config** Change `agent-config.yml` in the console at `agent.firstbyte.fast` or through the control-plane API. 2. **Save — no rebuild** The control plane hydrates the config into the runtime. There is nothing to compile or redeploy. 3. **New calls pick it up immediately** The next call routed to the agent uses the new config. Runtime tunables (timeouts and similar settings) hot-reload for the process. **In-flight calls keep the config they started with.** Thus a mid-call edit cannot destabilize an active conversation. 4. **Verify on a test call** Place a call. Watch the [latency breakdown](/modalities/voice/observability/latency-breakdown) and [call traces](/modalities/voice/observability/call-traces) to check that the new providers, timings, and budgets took effect. ## Related - **[BYO ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts)** — Build a cascade with the provider tables, models, and credentials. - **[BYO speech-to-speech](/modalities/voice/runtime/byo-speech-to-speech)** — Run a single duplex model to get the lowest turn latency. - **[Turn detection & barge-in](/modalities/voice/concepts/turn-detection)** — Learn the behavior behind the turn_detection block. - **[Session lifecycle](/modalities/voice/runtime/session-lifecycle)** — Learn the session from connect to teardown, and the guardrails that bound it. - **[Tool calling](/modalities/voice/runtime/tool-calling)** — Give the LLM stage HTTP, MCP, and client tools. - **[Codecs](/modalities/voice/transport-telephony/codecs)** — Learn what is negotiated on the wire, and how to influence it.