# From Custom WebSocket Audio

> Move a bespoke WebSocket audio bridge onto FirstByte Voice with the PCM16 WebSocket proxy that already backs the Vapi vendor bridge.

You already run a **custom WebSocket audio bridge**. It is a service that
carries raw audio frames over a WebSocket. It sits between a browser/app and
your AI backend, or between a telephony/agent vendor and your orchestrator.
This page shows how to put that WebSocket audio source in front of
FirstByte Voice's QUIC transport, media plane, and telephony. You do not
rewrite your audio backend.

The short version: the concrete adapter that ships today is a **PCM16
WebSocket proxy**. It is the same proxy that backs the Vapi vendor bridge. Your
backend plugs in as the WebSocket peer that the proxy decodes to and re-encodes
from. FirstByte owns the customer-facing transport (QUIC/WebTransport, with
a WebSocket fallback). FirstByte also does the Opus ⇄ PCM16 codec work. Your
WebSocket bridge stays a PCM16 audio endpoint.

> **NOTE:**
> **What has a ready adapter.** Vapi's PCM16 WebSocket shape is wired in the
>   vendor bridge. LiveKit uses the same bridge as an Opus passthrough. **Twilio
>   Media Streams is a different path.** Mulaw over WebSocket maps to a
>   codec-passthrough SIP trunk, not this proxy. See
>   [/modalities/voice/migration/from-twilio](/modalities/voice/migration/from-twilio).

## Which shape of WebSocket bridge do you have?

"Custom WebSocket audio" covers two very different edges. They map onto
different FirstByte on-ramps. Pick the edge that matches yours before you
migrate.

  - **Backend-edge WebSocket** — A vendor or telephony provider streams audio to your **AI orchestrator**
    over a WebSocket. The format is typically **PCM16** (Vapi-style) or mulaw
    (Twilio-style). Your backend consumes and produces raw audio frames. → Map
    this onto the **PCM16 WebSocket proxy** below.
  - **Client-edge WebSocket** — A **browser or app** streams captured audio to your backend over a
    WebSocket, because it cannot open a raw media socket. → Adopt the engine's
    WebSocket media fallback and let the SDK own capture. See
    [Browser/app clients](#browser-app-clients).

## The adapter that exists: a PCM16 WebSocket proxy

The vendor bridge is a WebTransport handler. It sits between a customer media
leg and an **external audio backend reached over a secure WebSocket**. For the
Vapi shape, the flow is:

  1. **Customer audio arrives over QUIC/WebTransport**
A caller (browser, app, or a bridged phone leg) reaches the engine over
    QUIC/WebTransport. When UDP is blocked, the caller reaches the engine over
    the WebSocket fallback. Audio is framed Opus on that customer leg.
  2. **The bridge transcodes to PCM16 and cleans it**
On ingress, the bridge decodes Opus and resamples to **16 kHz PCM16**. It
    then runs **noise suppression**. Thus your backend receives clean,
    uncompressed audio. This is also why the cleanup lives at the edge and not
    in your service. The bridge then sends raw PCM16 to your WebSocket backend
    as binary messages.
  3. **Your backend replies in PCM16**
Your backend (the "sidecar") speaks its vendor's PCM16 WebSocket protocol.
    It publishes the caller into your agent. It streams the agent's PCM16 audio
    back over the same socket.
  4. **The bridge re-encodes to the caller**
The bridge re-chunks the return PCM16 into 20 ms frames, encodes them as
    Opus, and writes them back to the customer leg. Your backend never touches
    Opus, QUIC, or SRTP.

> **WARNING:**
> The WebSocket backend that terminates the PCM16 stream (the "sidecar") is
>   **external to the engine**. It uses the vendor's API to publish/subscribe.
>   The in-tree bridge provides the transport, the codec transform, the framing,
>   and the denoise. You (or the vendor's cloud) provide the PCM16 WebSocket
>   peer. You supply the vendor selection and credentials as deployment
>   configuration today. Per-tenant hydration from the connection token is a
>   follow-on.

## Map your custom source onto the proxy

Your job is to make your existing WebSocket audio backend look like the PCM16
peer that the bridge already knows. There are three cases, easiest first.

  1. **If your backend already speaks Vapi's PCM16 framing**
Point the vendor bridge at Vapi (`VENDOR = vapi`) and supply your
    `VAPI_API_KEY`. The bridge's PCM16 branch handles the codec transform end
    to end. No engine changes are needed. This is the turnkey path.
  2. **If your backend speaks a generic PCM16 WebSocket**
Conform your endpoint to the **PCM16 sidecar contract**: accept and emit
    **16 kHz mono PCM16** as binary WebSocket messages, one media leg per
    connection. The bridge treats it like the Vapi PCM16 branch.

    > **NOTE:**
> The shipped bridge branches on two codecs only: **PCM16** (Vapi shape)
>       and **Opus passthrough** (LiveKit shape). A framing that is neither is
>       **not a turnkey branch yet**. Conform to the 16 kHz PCM16 contract. Or
>       treat the bridge as a reference and adapt at your sidecar. Do not expect
>       the engine to speak your custom framing.
  3. **Route inbound phone calls straight at your WebSocket backend**
For PSTN/SIP trunks, set the trunk's **inbound AI WebSocket URL**
    (`inbound_ai_websocket_url`) to your endpoint. Inbound calls that match
    that trunk get their AI leg bridged to your WebSocket after the more
    specific routes are exhausted. The route order is:

    1. An explicit agent binding.
    2. A skill/queue target.
    3. The QUIC bridge URL.
    4. This WebSocket URL.

    Pair it with `inbound_ai_quic_url` if you also expose a QUIC endpoint.

## Browser/app clients {#browser-app-clients}

Your custom WebSocket bridge can really be a **client transport**: a browser or
app that ships captured audio to a backend. In that case, you usually do not
need a bespoke socket at all. The engine serves a **WebSocket media fallback**
(subprotocol `clutch-media`). It carries the identical browser ⇄ agent audio
leg for clients on networks where QUIC/UDP is blocked (Safari, strict corporate
firewalls). It uses the `ws/<sid>` namespace. Thus it never collides with the
WebTransport leg's `wt/<sid>`. The wire is framed Opus in both directions.

> **WARNING:**
> The QUIC → WebSocket **automatic fallback ladder ships in the native SDK
>   cores only**. The browser/TypeScript SDK is WebTransport-only today. Thus a
>   browser client either uses WebTransport, or targets the `clutch-media`
>   endpoint with the framed-Opus wire directly. Plan your browser rollout around
>   WebTransport support. See
>   [/modalities/voice/transport-web/browser-compatibility](/modalities/voice/transport-web/browser-compatibility).

Let the SDK own capture (device AEC/AGC/noise suppression, Opus encode, the
media-over-QUIC track). Do not re-implement capture in your WebSocket client.
See
[/modalities/voice/transport-web/browser-audio-capture](/modalities/voice/transport-web/browser-audio-capture).

## Wire and audio contract

**Frame format, sample rates, namespaces**

- **Customer leg (WebTransport or `clutch-media` WebSocket):** an optional
    handshake, then length-prefixed frames `[u32 BE length][Opus payload]` in
    both directions.
  - **Sidecar leg (PCM16 backends, e.g. Vapi):** **16 kHz mono PCM16** as binary
    WebSocket messages. LiveKit-shape backends instead get raw Opus passthrough.
  - **Engine agent bus:** **8 kHz PCM16** end to end. The bridge resamples
    between 8/16/48 kHz as needed. Phone legs are G.711 (µ-law/A-law) on the
    wire. The engine transcodes them internally.
  - **Namespaces:** WebTransport legs use `wt/<sid>`. The WebSocket fallback
    uses `ws/<sid>`. The audio-only track model is in
    [/modalities/voice/concepts/sessions-calls-tracks-streams](/modalities/voice/concepts/sessions-calls-tracks-streams).

## Related

  - **[Keep your existing runtime](/modalities/voice/runtime/keep-existing)** — When the vendor bridge — not a native pipeline — is the right home for your
    audio backend.
  - **[Custom runtime integration](/modalities/voice/integrations/custom-runtime)** — Bring your own realtime endpoint through the provider abstraction.
  - **[From Twilio Media Streams](/modalities/voice/migration/from-twilio)** — Mulaw-over-WebSocket → codec-passthrough SIP trunk, the other WebSocket path.
  - **[Migration overview & checklist](/modalities/voice/migration/checklist)** — The full cut-over sequence for trunks and agents that you point at FirstByte.