# Agent Runtime Overview

> The agent runtime drives a live conversation. It controls turn-taking, tool calls, and session lifecycle over a cascade or a duplex model.

The **agent runtime** holds the conversation. A call connects from a phone
caller on a SIP trunk, or from a browser microphone over QUIC. Then the runtime
must do all of these tasks:

- Decide when the caller stops talking.
- Send the caller's words through a model.
- Speak the reply.
- Stop immediately when the caller interrupts.
- Call your tools in the middle of a turn.
- Tear the session down cleanly when the call ends.

You describe the agent. The runtime drives the turns.

You do **not** have to use the runtime. FirstByte Voice has two separable
halves: a transport that moves audio, and a runtime that reasons over the
audio. If you use the FirstByte runtime, you get turn-taking and
tool-calling with no extra work. If you bring your own runtime, you use
FirstByte only as the transport. Both paths are first-class.
[Pick a path](#two-paths-ours-or-your-own) below.

## What the runtime owns

Give the runtime an agent definition and a live audio stream. The runtime then
owns the full conversational loop:

  - **[Turn detection](/modalities/voice/concepts/turn-detection)** — Decides when the caller finished a turn and when the caller barged in over
    the agent. Hold-and-confirm prevents a backchannel ("mhm", "yeah") from
    cutting the agent off.
  - **Model orchestration** — Runs the pipeline. The pipeline is a **cascade** (speech-to-text →
    language model → text-to-speech), or a **duplex** speech-to-speech model
    that does all three steps in one streaming session.
  - **[Tool calling](/modalities/voice/runtime/tool-calling)** — Lets the model call your HTTP endpoints and MCP tools in the middle of the
    conversation. Feeds the results back to the model. Keeps the caller
    engaged while a tool runs.
  - **[Session lifecycle](/modalities/voice/runtime/session-lifecycle)** — Owns the session from connect to teardown. This includes the idle
    watchdog, re-engage prompts, token and duration budgets, and a hard
    max-session ceiling.

All of this runs **inside the voice gateway**. The runtime runs in the same
process that terminates the call's media, pinned to the core that owns the
call. Audio never leaves the machine to reach the runtime. Turn detection and
barge-in signalling connect directly into the media path. This is why
interrupts feel instant.

## Two pipeline shapes

The runtime drives a conversation in one of two ways. You choose the shape for
each agent. You can mix providers within a cascade.

  
    A cascade chains three specialised models. The first stage transcribes
    the caller's audio. The text then goes to a language model. The last
    stage speaks the model's reply. You get the widest provider choice and
    full control of each stage.

    ```
    caller audio ─▶ ASR ─▶ LLM ─▶ TTS ─▶ agent audio
                   │             ▲
                   └─ on-device VAD decides "caller finished" ┘
    ```

    You can mix providers. For example, use Deepgram for transcription, your
    LLM of choice for reasoning, and ElevenLabs or Cartesia for speech. A
    local **on-device VAD** marks turn boundaries, so backchannels never wake
    the model.

    - **[Wire your own ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts)** — Set the provider and model for each stage.
  
  
    One streaming model takes audio in and sends audio out. There is no
    separate transcription or synthesis stage. You get lower latency and more
    natural prosody, but a narrower model choice.

    ```
    caller audio ─▶ ┌─────────────────────────┐ ─▶ agent audio
                    │  duplex realtime model  │
                    └─────────────────────────┘
    ```

    Point the agent at a cloud realtime endpoint, such as OpenAI Realtime or
    Gemini Live. Or serve a single duplex model on your own GPUs. With cloud
    realtime models, the provider can do its own end-of-turn detection
    (`server_vad`). The runtime still applies barge-in confirmation on your
    side.

    - **[Use a duplex model](/modalities/voice/runtime/byo-speech-to-speech)** — Configure a single speech-to-speech session.
  

> **NOTE:**
> Cloud duplex providers (OpenAI Realtime, Gemini Live) are supported today.
>   Support for your **own** duplex model on-prem is newer. See
>   [self-hosted inference](/modalities/voice/runtime/self-hosted-inference) to
>   learn what is production-ready and what is still in hardening.

## Two paths: ours, or your own

The runtime and the transport are decoupled. Decide how much of the stack you
want FirstByte to own:

  - **[Use the FirstByte runtime](/modalities/voice/runtime/builtin-runtime)** — The built-in runtime drives the conversation end-to-end. You supply an
    agent definition and provider credentials. The runtime handles turn
    detection, tool calling, and session lifecycle for you. **Start here for
    most agents.**
  - **[Keep your own runtime](/modalities/voice/runtime/keep-existing)** — You can keep an existing conversation engine, such as a LiveKit or Pipecat
    pipeline, or your own orchestration. Use FirstByte as **transport
    only**. It delivers audio in and carries your audio out. Your runtime owns
    the turns.

The rest of this section assumes the FirstByte runtime. If you bring your
own runtime, [Keep Your Existing Agent Runtime](/modalities/voice/runtime/keep-existing)
is the only page you need. You interact only with the
[web](/modalities/voice/transport-web/overview) and
[telephony](/modalities/voice/transport-telephony/overview) transport surfaces.

## Where the runtime gets its audio

The runtime is transport-agnostic. The same conversational loop drives a call
for each transport:

| Caller | Arrives as | Inside the runtime |
|--------|------------|--------------------|
| Phone (PSTN / SIP trunk) | G.711 (PCMU/PCMA) at 8 kHz | decoded to PCM16 |
| Browser / mobile app | Opus over QUIC | decoded and resampled to PCM16 |

Internally, the runtime uses a single **8 kHz PCM16** audio contract. An agent
behaves the same on a phone call and on a web call. The
[transport (web)](/modalities/voice/transport-web/overview) and
[transport (telephony)](/modalities/voice/transport-telephony/overview) pages
tell you how audio gets to the runtime.

## Configure it

Every agent is a config document. The document sets the pipeline shape, the
per-stage providers and models, the turn-detection tuning, the tools, and the
budgets. You edit the document in the console at `agent.firstbyte.fast` or
through the control-plane API. The gateway hot-reloads runtime tunables and
does not drop in-flight calls.

- **[Runtime configuration reference](/modalities/voice/runtime/configuration)** — This reference lists every knob: models, turn detection, codecs, budgets,
  and guardrails.

## Next steps

  - **[The FirstByte Agent Runtime](/modalities/voice/runtime/builtin-runtime)** — The built-in runtime supplies pipelines, providers, and turn-taking.
  - **[Turn detection & barge-in](/modalities/voice/concepts/turn-detection)** — Tune when the agent listens, talks, and yields the floor.
  - **[Tool calling](/modalities/voice/runtime/tool-calling)** — Give the agent HTTP and MCP tools in the middle of the conversation.
  - **[Session lifecycle](/modalities/voice/runtime/session-lifecycle)** — Learn the session lifecycle from connect to teardown, and the guardrails
    that bound it.