# Custom Agent Runtime

> Run your own conversation engine over the transport. Bridge to an external vendor, use the raw MoQT audio primitives, or use the LiveKit-compatible shim.

You possibly already have a runtime that works: a custom agent loop, a Pipecat
bot, a LiveKit app, or your own ASR/LLM/TTS chain. You do not have to rebuild it
to run on FirstByte. Keep your runtime and adopt only the **transport**. The
transport delivers a call's audio to your code and carries your reply back over
QUIC/MoQT on the single `:443` plane. You do not negotiate ICE/DTLS/SRTP. You do
not operate a media server.

There are three ways to connect a runtime. They are listed from least code to
most control:

  - **Vendor bridge** — A **config node** with no code. Point a call at an external orchestration
    vendor (LiveKit / Vapi / Twilio). FirstByte stays the transport in front
    of the vendor.
  - **Raw MoQT bridge** — Subscribe to the caller's **uplink**. Publish your reply as the
    **downlink**. You own capture, model, and playback. FirstByte moves the
    frames. **This is the stable path.**
  - **LiveKit-compat shim** — A surface with the shape of `livekit-client` over our transport. Change one
    import.
    <Tooltip tip="Preview">Preview.</Tooltip>

## Route 1 — bridge to an external vendor

Your runtime can be another orchestration product, for example a LiveKit Agent,
a Vapi assistant, or a Twilio app. In this case you do not write integration
code. You add a **`VENDOR_BRIDGE`** node to the agent's pipeline config. When
the runtime reads this node, it sends the call's media to that vendor over an
external **bridge sidecar**. It does not run our own ASR→LLM→TTS pipeline.
FirstByte stays the QUIC transport in front of the vendor's media backend.

> **WARNING:**
> **This route is a media bridge, not a packaged adapter.** The vendor bridge
>   routes audio to an **external sidecar**. You run this sidecar with your vendor
>   account. The sidecar ships separately from the engine. The sidecar moves audio
>   to and from the vendor. It does not import the vendor's SDK for you. There is
>   no Pipecat adapter.

> **NOTE:**
> **LiveKit Agents is the exception.** A packaged plugin is available:
>   `livekit-plugins-clutchcall` / `@clutchcall/livekit-transport`. The plugin installs into your
>   `livekit-agents` worker and replaces only its transport. You do not need a
>   sidecar or a LiveKit server. Set `vendor_provider` to `clutchcall`
>   (not `livekit`) to dispatch to a worker that runs the plugin. See
>   [LiveKit Agent Integration](/modalities/voice/integrations/livekit).

The config is a single node in the agent's pipeline document:

```json
{
  "entry_node": "BRIDGE",
  "pipeline": {
    "BRIDGE": {
      "node_type": "VENDOR_BRIDGE",
      "vendor_provider": "livekit",          // "livekit" | "vapi" | "twilio"
      "vendor_room": "support-room-42",       // LiveKit room / Vapi assistant id / Twilio app
      "vendor_creds_ref": "livekit-primary"   // key into this tenant's sealed vendor credentials
    }
  }
}
```

- **`vendor_provider`** (`string`, required) — The external orchestration vendor: `livekit`, `vapi`, or `twilio`. This value   selects the bridge protocol that the sidecar speaks. Set the value to   `clutchcall` to dispatch directly to a worker that runs the   LiveKit-Agents transport plugin. That path does not use a sidecar.
- **`vendor_room`** (`string`) — The vendor-side target: a LiveKit room, a Vapi assistant id, or a Twilio app.   For `clutchcall`, the target is the name that your plugin worker   registered.
- **`vendor_creds_ref`** (`string`) — A reference into the tenant's stored vendor credentials (sealed at rest). The   vendor's keys never appear in the pipeline document.

Edit this node in the console at `agent.firstbyte.fast` or through the
control-plane API. Then point a trunk or an outbound call at the agent. See the
vendor-specific pages for the exact room and credential shape:

  - **[LiveKit Agent](/modalities/voice/integrations/livekit)** — Run a LiveKit agent behind our transport.
  - **[Pipecat](/modalities/voice/integrations/pipecat)** — Run a Pipecat pipeline behind our transport.

## Route 2 — the raw MoQT audio bridge

This is the direct path, and most custom runtimes use it. You attach to a live
call. You receive the caller's audio frames. You send your reply frames back.
Each call is addressed by its **`call_sid`**. The call's audio lives on two
MoQT tracks under a room-scoped namespace:

| Track | Direction | You |
| --- | --- | --- |
| `voice/<sid>/uplink` | caller → you | **subscribe** to hear the caller |
| `voice/<sid>/downlink` | you → caller | **publish** your runtime's reply |

The SDK's `AudioBridge` wraps both tracks behind one handle. `attach()`
subscribes to the uplink and publishes the downlink for you. The logic between
`onUplink` and `publishDownlink` is yours: your VAD, your turn-taking, your
model.

  1. **Attach to the call**
Open the bridge with the call's `call_sid`. Select the codec that matches
    the leg. Use `opus` for browser and app callers. Use `g711_ulaw`,
    `g711_alaw`, or `pcm16` for phone callers.
  2. **Consume the uplink**
Your `onUplink` callback fires for each inbound audio frame. Send the frame
    to your runtime's input, for example your ASR or your speech-to-speech
    session.
  3. **Publish the downlink**
When your runtime produces reply audio, call `publishDownlink(frame)` for
    each encoded frame (for example, one 20 ms Opus packet). The caller hears
    the audio.
  4. **Close on teardown**
Release the handle when the call ends. This stops both tracks and ends the
    bridge.

  
    ```ts
    import { Voice } from "clutchcall_connect/voice";

    const voice = new Voice({
      baseUrl:   "https://engine.firstbyte.fast",
      apiKey:    process.env.CLUTCHCALL_API_KEY!,
      orgId:     "org_xxx",
      relayHost: "relay.firstbyte.fast",
    });

    // Attach to a live call by its call_sid: uplink is subscribed,
    // downlink is published — both as MoQT tracks under voice/<sid>.
    const bridge = await voice.audioBridge.attach(callSid, {
      codec: "opus",                        // opus | pcm16 | g711_ulaw | g711_alaw
      onUplink: (frame, tsUs) => {
        myRuntime.pushCallerAudio(frame, tsUs);   // feed YOUR model
      },
    });

    // When your runtime produces reply audio, play it back to the caller.
    myRuntime.onReplyAudio((frame) => bridge.publishDownlink(frame));

    // ...on hangup
    await bridge.close();
    ```
  
  
    ```python
    from clutchcall.voice import Voice

    voice = Voice(
        base_url="https://engine.firstbyte.fast",
        api_key=os.environ["CLUTCHCALL_API_KEY"],
        org_id="org_xxx",
        relay_host="relay.firstbyte.fast",
    )

    def on_uplink(frame: bytes, ts_us: int) -> None:
        my_runtime.push_caller_audio(frame, ts_us)   # feed YOUR model

    # Uplink subscribed, downlink published, under voice/<sid>.
    bridge = voice.audio_bridge.attach(call_sid, codec="opus", on_uplink=on_uplink)

    # Your runtime's reply audio → the caller.
    my_runtime.on_reply_audio(lambda frame: bridge.publish_downlink(frame))

    # ...on hangup
    bridge.close()
    ```
  

> **NOTE:**
> **Same shape in every SDK.** Go, Rust, Java, and .NET expose the same
>   `attach` / `publish_downlink` primitives through a shared native core. The
>   native bindings speak QUIC/MoQT directly. They do **not** include the
>   browser's WebSocket/WebRTC fallback leg. Plan for a QUIC-reachable network.
>   See the [transport SDK reference](/modalities/voice/api/transport) for
>   per-language method and event names.

> **NOTE:**
> **Namespace rule.** Track namespaces are room-scoped tuples. The prefix and
>   the suffix must not be empty. The media-over-QUIC engine rejects empty
>   tuples. The `voice/<sid>/{uplink,downlink}` convention obeys this rule. The
>   SDK builds the namespaces for you from the `call_sid`. If you build
>   namespaces manually, keep them room-scoped. For more about the object model,
>   see
>   [Sessions, Calls, Tracks & Streams](/modalities/voice/concepts/sessions-calls-tracks-streams).

### Where the `call_sid` comes from

You attach a bridge to a call that already exists. For an outbound call,
`voice.calls.originate` returns the `call_sid`. For inbound calls, point the
trunk's inbound rule at your integration. This routes the leg to your bridge
instead of the built-in runtime. Then attach as each call arrives. The
[telephony transport](/modalities/voice/transport-telephony/overview) section
covers trunk routing.

## Route 3 — the LiveKit-compatible shim

If your runtime is a browser LiveKit app, use the `livekit-compat` shim. The
shim exposes a surface with the shape of `livekit-client` (`Room`, `RoomEvent`,
tracks, participants) mapped onto our MoQT/QUIC transport. The core voice path
covers these actions: publish the mic, subscribe to remote audio, and exchange
data messages. For this path, migration is a one-line import change. There is
no SFU and no ICE.

> **NOTE:**
> **Preview.** The `livekit-compat` shim is experimental and audio-first. It
>   covers the core voice path. It is not yet a replacement for every
>   `livekit-client` API. Video and screenshare are deferred. The shim ships for
>   the **browser / TypeScript** client only. For a server-side LiveKit
>   **agent**, use the [transport plugin](/modalities/voice/integrations/livekit)
>   instead. The plugin replaces the agent session's audio IO. It does not
>   emulate a room. Pin a version and test your flows. The full walkthrough is in
>   [Keep Your Existing Agent Runtime](/modalities/voice/runtime/keep-existing).
>   It includes the `RTCRtpScriptTransform` requirement and the API mapping.

## Which route?

  - **[LiveKit Agents plugin](/modalities/voice/integrations/livekit)** — Use this route when your runtime **is** a `livekit-agents` worker. It is a
    packaged plugin. It needs no sidecar and no LiveKit server. It is shipped.
  - **Vendor bridge** — Use this route when your runtime is Vapi or Twilio orchestration, or a
    LiveKit room that must continue to run. It is config-only. You run the
    external sidecar.
  - **Raw MoQT bridge** — Use this route for a custom runtime, a non-browser runtime, or a
    server-side loop. It gives the most control. It is fully shipped in every
    SDK.
  - **LiveKit-compat shim** — Use this route for a browser LiveKit app that you want to move with minimal
    edits. It is a preview.

## Related

  - **[Keep Your Existing Runtime](/modalities/voice/runtime/keep-existing)** — The full browser guide: the shim, the transport primitives, and the codec
    tap that moves an existing WebRTC pipeline onto QUIC.
  - **[BYO Speech-to-Speech](/modalities/voice/runtime/byo-speech-to-speech)** — Keep **our** runtime and point it at your own realtime endpoint with
    per-tenant credentials.
  - **[Agent Runtime Overview](/modalities/voice/runtime/overview)** — What the managed runtime does when you want us to drive the conversation.
  - **[From Self-Hosted LiveKit](/modalities/voice/migration/from-livekit)** — The end-to-end migration path for a LiveKit deployment that moves onto our
    transport.