# Self-hosted models

> Serve your own speech-to-speech and speech models on your own GPUs. The FirstByte Inference control plane routes calls to them over QUIC.

Run the models on your own hardware. You do not have to point an agent at a
cloud realtime API. Instead, you can serve an open-weights **duplex
speech-to-speech** model, or the language model behind a cascade, on GPUs in
your own datacenter. FirstByte Voice then routes each call to your models.
The audio never leaves your infrastructure. You pick the exact model and
weights. There is no per-minute vendor bill on the inference leg.

**FirstByte Inference** makes this practical. It is a control plane that
sits between the voice gateway and your fleet of model replicas. You stand up
the replicas. The control plane finds them, watches their health and load,
and steers each new session to the best replica over QUIC. Your agents keep
the same config shape they use for any other provider.

> **NOTE:**
> This page is about models that **you** operate. You may only want to point at
> a hosted realtime endpoint (yours or a vendor's) without a control plane.
> That path is [BYO Speech-to-Speech](/modalities/voice/runtime/byo-speech-to-speech)
> or [BYO ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts). You set
> an endpoint host and a key, and you are done.

## Status: what's production-ready

Be clear-eyed about what is shipped and what is still in hardening, before
you route regulated traffic.

> **WARNING:**
> The **control plane is shipped**. It routes text completions
> (`/v1/chat/completions`-shaped requests) to your replicas over QUIC in
> production. The **duplex speech-to-speech audio path** — a single model that
> takes audio in and speaks audio out — is wired end-to-end in the runtime.
> But it is **not yet verified live against production carriers**. Treat the
> self-hosted duplex model as **preview**. Prove turn-taking and latency on a
> staging number before real calls.

Current status:

| Capability | Status |
|------------|--------|
| Control plane: replica discovery, health/load tracking, QUIC routing | Shipped |
| Self-hosting the **language model** stage of a cascade | Shipped (text path) |
| Self-hosting a **duplex speech-to-speech** model end-to-end on a live call | Preview — not yet carrier-verified |
| Self-hosting a standalone ASR or TTS stage on the control plane | Not yet — use a hosted endpoint for those stages today |

## How it fits together

The control plane does one job. It keeps a live picture of your model fleet
and puts each session on the right replica. It owns four roles. It owns
nothing about your model's weights or serving framework.

  - **Replica discovery** — Watches your serving cluster (for example, a Kubernetes namespace). Keeps
    a live roster of ready model replicas as you scale them up and down.
  - **Health & load awareness** — Scrapes each replica's health and load signals continuously. Thus a
    saturated or unhealthy pod leaves the rotation before it drops a call.
  - **Session routing** — Applies a routing strategy (least-loaded, session-affinity, and similar
    strategies) to pick a replica per session. Keeps a conversation pinned to
    that replica.
  - **QUIC transport** — Carries requests to the chosen replica over QUIC, not over a legacy proxy
    hop. The routing brain is standard model-serving infrastructure. Only the
    transport underneath it is swapped.

For a spoken conversation, the **duplex audio adapter lives in the runtime**,
next to the media path. The adapter opens the realtime session, forwards each
PCM chunk, and streams the model's spoken reply back onto the call. It
resamples for you. The control plane routes. The runtime speaks. That split
keeps interrupts instant: barge-in is wired into the media plane in-process,
not across the control plane.

```
call media ─▶ agent runtime ─▶ FirstByte Inference ─▶ your model replica
  (in the           │  (duplex audio      (discover · health ·      (GPUs you
   voice gateway)   │   adapter)           route · QUIC)             operate)
                    └─ turn detection & barge-in stay in the media path
```

## What you serve

You bring the model. FirstByte routes to it. Two shapes are supported
today, with different maturity:

  
    A **single** open-weights model consumes audio and emits audio on one
    streaming session. There is no separate transcription or synthesis stage.
    This is the lowest-latency and most natural option. It is also the path
    that is still in hardening.

    Serve the model behind a realtime WebSocket on your GPUs. Register the
    endpoint. Set the agent's provider to the self-hosted duplex provider
    (`omni`). See
    [BYO Speech-to-Speech](/modalities/voice/runtime/byo-speech-to-speech)
    for the full agent config.

    > **NOTE:**
> Some open-weights duplex models have **no** server-side VAD. They never
>     tell you when the caller stopped talking. In that case, the runtime owns
>     the turn boundary with a local detector. It commits the input buffer when
>     the caller finishes. This also prevents backchannels ("ok", "mhm") from
>     waking the model. See
>     [Turn Detection & Barge-In](/modalities/voice/concepts/turn-detection).
  
  
    Do you prefer a cascade? Self-host just the **language model** stage. The
    control plane serves text completions over QUIC. Thus you point the
    cascade's LLM stage at your own endpoint. You keep a hosted transcriber
    and speech synthesiser, or self-host those stages later.

    Wire it exactly like any other LLM provider in
    [BYO ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts). The
    endpoint just happens to be yours.
  

## Wire it up

  1. **Stand up your model replicas**
Serve your chosen model on your GPUs. Put it behind a realtime WebSocket
    (duplex) or an OpenAI-compatible completions endpoint (cascade LLM). Run
    as many replicas as your call volume needs. The control plane handles the
    replicas as a pool. To scale out, you only add replicas.
  2. **Let the control plane discover the fleet**
Point FirstByte Inference at your serving cluster. It picks up ready
    replicas. It begins to track their health and load. It starts to route.
    You maintain no static endpoint list as you scale.
  3. **Set the provider on the agent**
In the agent config, name the self-hosted provider and your model. Supply
    the endpoint and the optional bearer token per workspace. Credentials are
    stored per tenant and never shared across workspaces.

    | Field | Value |
    |-------|-------|
    | `provider` | `omni` (duplex) or your self-hosted LLM provider (cascade) |
    | `model` | the model name you serve (e.g. `my-omni-8b`) |
    | Endpoint | control-plane / replica host (`host[:port]`, default `443`) |
    | Token | optional bearer token for your endpoint |

    Edit these fields in the console at `agent.firstbyte.fast` or through the
    control-plane API.
  4. **Prove it on a staging number**
Point a staging number or trunk at the agent. Place calls. Watch
    turn-taking, barge-in, and end-to-end latency before you move production
    traffic. This is the preview boundary. If a replica refuses the session,
    the runtime logs the refusal. The call then falls back to a silent
    passthrough and does not drop.

## Why self-host

  - **Data residency** — Call audio and transcripts never leave your infrastructure. The model
    runs where your compliance boundary is. This pairs naturally with a voice
    gateway that also runs on-prem.
  - **Model choice & control** — Run the exact open-weights model, version, and fine-tune you want. Change
    it on your schedule. You do not wait on a vendor's model lifecycle.
  - **Cost at volume** — At steady, high call volume, your own amortised GPUs can beat per-minute
    realtime API pricing on the inference leg.
  - **No new proxy tier** — The control plane reuses standard model-serving routing but carries
    traffic over QUIC. Thus you skip an extra proxy hop on every request.

## Next steps

  - **[BYO speech-to-speech](/modalities/voice/runtime/byo-speech-to-speech)** — The full duplex agent config, including the self-hosted `omni` provider.
  - **[BYO ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts)** — Self-host the language model stage of a cascade and mix in hosted speech.
  - **[Turn detection & barge-in](/modalities/voice/concepts/turn-detection)** — Learn what to tune when a self-hosted model has no server-side VAD.
  - **[Runtime configuration](/modalities/voice/runtime/configuration)** — Every runtime knob: providers, budgets, and guardrails.