# BYO Speech-to-Speech Models

> Run one duplex speech-to-speech model instead of an ASR → LLM → TTS cascade to get lower turn latency.

Point your agent at a single duplex speech-to-speech model and skip the
cascade entirely. A cascade chains speech-to-text, a language model, and
text-to-speech. That is three network hops, three queues, and three places
that add latency. With a duplex model, the caller's audio streams straight
into one model. The model listens and speaks on the same bidirectional
connection. Fewer hops means the agent starts to reply sooner, and interrupts
feel instant.

You wire a duplex model the same way you wire any other runtime. Name a
provider, a model, and a voice in the agent config. FirstByte Voice opens
the realtime session on the first inbound audio frame. It forwards each PCM
chunk. It streams the model's spoken response back onto the call. It resamples
to and from the provider's expected rate for you.

## Speech-to-speech vs. cascade

> **NOTE:**
> Not every agent should be speech-to-speech. A duplex model is the fastest and
> most natural option. But a cascade (see
> [BYO ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts)) gives you
> independent control of each stage. You can swap the language model and keep
> the voice. You can run a fine-tuned transcriber. You can mix vendors. Choose
> duplex for latency and naturalness. Choose the cascade for control and model
> choice.

## Supported models

| Provider | Config `provider` | Status | Notes |
|----------|-------------------|--------|-------|
| **OpenAI Realtime** | `openai` | Shipped | Server-side VAD, function calling, per-response greeting. |
| **Gemini Live** | `gemini` | Shipped | Prebuilt voices (`Aoede`, `Charon`, `Fenrir`, `Kore`, `Puck`); function calling. |
| **xAI Grok Voice** | `xai` / `grok` | Shipped | OpenAI-Realtime-compatible; server VAD + function calling. |
| **Self-hosted duplex** | `omni` | Preview | Bring your own open-weights duplex model over a realtime WebSocket on your GPUs. See [Self-Hosted Inference](/modalities/voice/runtime/self-hosted-inference). |

> **WARNING:**
> The self-hosted duplex path is wired end-to-end in the runtime. But it is not
> yet verified live against production carriers. Treat it as preview. Validate
> turn-taking and latency on a staging number before you route real traffic.

## Configure a duplex agent

  1. **Set the entry node to REALTIME**
A duplex agent has a single node. Set `entry_node: REALTIME`. Give the
    node a provider, a model, and a voice. There is no separate ASR or TTS
    stage to configure.

    
      
        ```yaml
        # agent-config.yml
        entry_node: REALTIME
        pipeline:
          REALTIME:
            node_type: REALTIME
            provider: openai
            model: gpt-realtime          # sent via the dial URL, not session.update
            voice: shimmer               # alloy | shimmer | …
            instructions: |
              You are a friendly support agent for Acme. Keep replies short.
        ```
      
      
        ```yaml
        entry_node: REALTIME
        pipeline:
          REALTIME:
            node_type: REALTIME
            provider: gemini
            model: gemini-2.0-flash-exp   # "models/" prefix added for you if omitted
            voice: Kore                   # Aoede | Charon | Fenrir | Kore | Puck
            instructions: |
              You are a friendly support agent for Acme. Keep replies short.
        ```
      
      
        ```yaml
        entry_node: REALTIME
        pipeline:
          REALTIME:
            node_type: REALTIME
            provider: xai
            model: grok-voice-latest
            instructions: |
              You are a friendly support agent for Acme. Keep replies short.
        ```
      
      
        ```yaml
        entry_node: REALTIME
        pipeline:
          REALTIME:
            node_type: REALTIME
            provider: omni                # your own duplex model
            model: my-omni-8b
            instructions: |
              You are a friendly support agent for Acme. Keep replies short.
        ```
      
    

  2. **Supply per-tenant credentials**
Each workspace supplies its own keys. The keys are stored per tenant and
    never shared across workspaces. Set them in the Voice AI console at `voiceai.firstbyte.fast`, or through the control-plane API:

    | Provider | Credential |
    |----------|------------|
    | `openai` | OpenAI API key |
    | `gemini` | Gemini API key |
    | `xai` / `grok` | xAI API key |
    | `omni` | Endpoint host (`host[:port]`, default port 443) + optional bearer token |

  3. **Deploy and test**
Save the agent. Point a number or trunk at the agent. Place a call. The
    runtime opens the realtime session on the first inbound frame. If the
    provider refuses the connection, the runtime logs the refusal. The call
    then falls back to a silent passthrough and does not drop.

## Turn-taking on a duplex model

For a `REALTIME` entry node, FirstByte Voice defaults to **`server_vad`**.
The provider does its own endpointing and emits `speech_started` /
`speech_stopped`. The runtime wires these events to barge-in. Thus in-flight
audio is cut the instant the caller talks. You do not add a local turn
detector.

> **NOTE:**
> Some self-hosted duplex models have **no** server-side VAD. In that case, the
> runtime owns the turn boundary with a local detector. It commits the input
> buffer when the caller finishes. This also prevents backchannels ("ok",
> "mhm") from waking the model. See
> [Turn Detection & Barge-In](/modalities/voice/concepts/turn-detection) for the
> knobs.

## Greeting on inbound calls

Some carrier SBCs do not send you RTP until they hear yours. On an inbound
call, the runtime asks the model to speak first, so the carrier's media path
opens. The model greets from your `instructions`. You can also pin an exact
opening line. This behavior is on by default for inbound direct-media calls.
No extra config is required.

## Tools mid-conversation

The shipped realtime adapters advertise your tools in the session bootstrap.
Thus the model can call HTTP and MCP tools during a spoken turn and speak the
result. Configure the tools exactly as you would for a cascade. See
[Tool Calling](/modalities/voice/runtime/tool-calling).

## Session budgets

Every realtime session is bounded, so a stuck provider cannot run forever:

- **Connect timeout** — the handshake gets 5 seconds. After that, the runtime
  gives up and falls back.
- **Max session** — the runtime closes a session at a **15-minute** ceiling,
  even if the provider did not close it. Tune the lifetime, idle re-engage,
  and hangup budgets on
  [Session Lifecycle](/modalities/voice/runtime/session-lifecycle).

## Next steps

  - **[Cascade instead](/modalities/voice/runtime/byo-asr-llm-tts)** — Wire independent ASR, LLM, and TTS providers when you want per-stage
    control.
  - **[Turn detection](/modalities/voice/concepts/turn-detection)** — Tune barge-in, endpointing, and backchannel handling.
  - **[Self-hosted inference](/modalities/voice/runtime/self-hosted-inference)** — Serve your own duplex model on-prem with the inference control plane.
  - **[All runtime knobs](/modalities/voice/runtime/configuration)** — Find models, codecs, budgets, and guardrails in one reference.