# FirstByte Agent Runtime

> The built-in voice agent runtime supplies pipeline nodes, a provider registry, turn-taking, and tool calling, wired end to end.

You do not assemble a voice agent from parts. FirstByte ships a complete
runtime that runs the whole conversation loop for you. The runtime listens,
decides, speaks, takes turns, and calls tools. It runs inside the same engine
that terminates the call. You do not stand up an orchestrator, and you do not
write glue code.

You describe the agent as a small config. The config sets the models, what the
agent says, and which tools the agent can use. The runtime does the rest.
Point the agent at a phone number or a browser session, and the agent talks.

## What "batteries included" means here

Everything below is already built and runs in-process. You opt into the pieces
you want. You override defaults where you need to.

  - **A ready pipeline** — Speech-to-text, language model, and text-to-speech nodes come pre-wired as
    a conversation graph. You can also use a single speech-to-speech node. You
    do not hand-build a DAG.
  - **A provider registry** — First-class ASR, LLM, TTS, and realtime providers sit behind one
    interface. Swap any stage when you change one string. Drop in your own
    keys.
  - **Turn-taking wired in** — Barge-in, end-of-utterance, and idle handling run end to end, from the
    caller's audio to the model and back. You write no client code.
  - **A built-in tool belt** — The agent can transfer, hold, route to a skill, or send DTMF with no
    setup. The agent can also call your own HTTP and MCP tools in the middle
    of the conversation.

## Two ways to run the conversation

The runtime supports two pipeline shapes. Pick one shape for each agent. Most
teams start with a cascaded pipeline. They then move latency-critical agents
to speech-to-speech.

| Shape | How it works | When to use |
|-------|--------------|-------------|
| **Cascaded** | Separate ASR → LLM → TTS nodes. You mix the best provider at each stage. | You want maximum control over model choice, cost, and voice. This shape is the easiest to reason about and swap. |
| **Speech-to-speech** | A single realtime node owns the audio path end to end. The provider does its own listening and speaking. | You want the lowest turn latency and the most natural prosody. |

  
    ```json agent.json
    {
      "entry_node": "ASR",
      "turn_detection": { "type": "silero", "silence_duration_ms": 500 },
      "pipeline": {
        "ASR":  { "node_type": "ASR",  "provider": "deepgram",  "next_node": "LLM" },
        "LLM":  { "node_type": "LLM",  "provider": "anthropic",
                  "model": "claude-sonnet-4-5",
                  "system_prompt": "You are a friendly support agent.",
                  "next_node": "TTS" },
        "TTS":  { "node_type": "TTS",  "provider": "elevenlabs",
                  "voice": "21m00Tcm4TlvDq8ikWAM", "next_node": "PUSH_AUDIO" },
        "PUSH_AUDIO": { "node_type": "PUSH_AUDIO", "next_node": "ASR" }
      }
    }
    ```
  
  
    ```json agent.json
    {
      "entry_node": "REALTIME",
      "pipeline": {
        "REALTIME": {
          "node_type": "REALTIME",
          "provider": "openai",
          "model": "gpt-realtime",
          "voice": "shimmer",
          "system_prompt": "You are a friendly support agent."
        }
      }
    }
    ```
    > **NOTE:**
> Some agents have the entry node `REALTIME` and no `turn_detection`
>       type. For these agents, the runtime auto-promotes turn detection to
>       `server_vad`. The realtime provider owns the audio path, so on-device
>       VAD would never run. If you specifically want gateway-side detection,
>       override the type back to `silero`.
  

You apply this config from the agent's **Config** tab in the Voice AI console
at `voiceai.firstbyte.fast`, or through the control-plane API. The system
stores the config per agent and
loads it when a call reaches that agent. See
[Configuration](/modalities/voice/runtime/configuration) for the full field
reference.

## Pipeline nodes

A cascaded agent is a graph of typed nodes. `entry_node` names where a turn
starts. Each node's `next_node` points to the next node. Beyond the three
conversation nodes, the runtime ships call-flow nodes. With these nodes, you
can build an IVR without leaving the config.

| Node | What it does |
|------|--------------|
| `ASR` | Streams caller audio to a speech-to-text provider. Emits interim and final transcripts. |
| `LLM` | Runs the transcript (plus history and tools) through a language model. Returns the reply. |
| `TTS` | Turns the reply text into audio. |
| `REALTIME` | Speech-to-speech: hands the audio straight to a realtime provider that listens and speaks. |
| `PUSH_AUDIO` | Streams synthesized audio back to the caller. On interrupt, the barge-in path clears its buffer. |
| `GREETING` | Plays an opening line at call connect, before the loop begins. |
| `GATHER` | Collects DTMF digits or a spoken response, with timeout and retry branches. |
| `MENU` | Branches by DTMF digit (press 1 for sales, 2 for support…). |
| `TRANSFER` | Hands the leg to a SIP or E.164 destination. |
| `HANGUP` | Ends the call. |

> **TIP:**
> You rarely author nodes by hand. The visual agent builder in the console
>   emits this graph for you. The JSON is the ground truth that the runtime
>   loads.

## The provider registry

Every ASR, LLM, TTS, and realtime backend sits behind one registry. To select
a provider, you set one string on the node. You write no code and do no
redeploy. The runtime brings credentials from your tenant vault (sealed at
rest) or from the environment. Thus you can bring your own keys for each
stage.

| Provider | ASR | LLM | TTS | Speech-to-speech |
|----------|:---:|:---:|:---:|:----------------:|
| `openai` |  | ✅ default | ✅ |  ✅ |
| `anthropic` |  | ✅ |  |  |
| `gemini` |  | ✅ |  | ✅ |
| `deepgram` | ✅ default |  | ✅ |  |
| `elevenlabs` | ✅ |  | ✅ default | ✅ |
| `cartesia` |  |  | ✅ |  |
| `ollama` |  | ✅ (self-hosted) |  |  |
| `xai` | ✅ |  | ✅ | ✅ |

> **NOTE:**
> The registry accepts more identifiers than the headline set above. It also
>   accepts regional and self-hosted ASR/TTS options, plus a self-hosted
>   inference gateway that runs your own models. See
>   [BYO ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts),
>   [BYO speech-to-speech](/modalities/voice/runtime/byo-speech-to-speech), and
>   [Self-hosted inference](/modalities/voice/runtime/self-hosted-inference).

**Defaults and fallbacks.** If you leave a cascaded stage's `provider` empty,
you get the defaults marked above (`deepgram` ASR, `openai` LLM, `elevenlabs`
TTS). An `LLM` node can also declare `llm_max_retries` and an
`llm_fallback_provider`. Then, when a turn fails, the runtime retries the
primary provider. After the retries, it falls back once to the second
provider. Only then does it give up. You get this resilience without retry
logic of your own.

If you did not configure a provider's credentials, the registry returns
nothing for that stage. It does not fail the whole call. An unconfigured ASR
drops the audio. A streaming TTS falls back to the one-shot synthesis path, so
the caller still hears the reply.

## Turn-taking, out of the box

Barge-in and end-of-utterance are wired from the caller's audio to the model
and back. You pick a mode and tune a few timings. The interrupt travels a
single control path for you.

- **`silero`** (default) — on-device voice-activity detection on the gateway.
  Right for cascaded pipelines.
- **`server_vad`** — the realtime provider decides turn boundaries. The
  runtime auto-selects this mode for `REALTIME` entry nodes.
- **`energy`** / **`none`** — a lightweight energy gate, or fully manual.

The runtime also runs an idle watchdog. The watchdog re-engages the caller
after silence, and then hangs up. Hold-and-confirm barge-in prevents short
backchannels like "mhm" from cutting the agent off. The full schema, the
defaults, and a barge-in support matrix live in
[Turn detection & barge-in](/modalities/voice/concepts/turn-detection).

## Tool calling

Your agent can act on the world in the middle of a conversation. The runtime
advertises a set of tools to the model each turn. The runtime then executes
whatever the model calls.

**Built-in telephony tools** are available to every agent with no setup. The
model can call `transfer_call`, `route_to_skill`, `hold_call` /
`unhold_call`, `send_dtmf`, `disconnect_call`, and `request_supervisor`. The
runtime issues the matching control action on the live leg. Per-trunk realm
rules govern where a call may be transferred.

**Your own tools** plug in three ways:

- **HTTP tools** — describe an endpoint and its JSON schema. The model calls
  the tool, and the runtime makes the request (bearer / basic / header auth
  supported).
- **MCP tools** — point at a Model Context Protocol server. The runtime runs
  `tools/list` and registers every tool that the server exposes.
- **Client tools** — declare a function that your app fulfills, for actions
  that must run in your environment.

To scope which tools a given node may call, set a per-node `tool_allowlist`.
An empty allowlist means all of the agent's tools. The full walkthrough is in
[Tool calling](/modalities/voice/runtime/tool-calling).

## Where sessions live

Each call gets a runtime session pinned to the core that handles the leg. The
session has its own turn-detector state, provider sessions, and deadlines.
Timeouts and the maximum session length are configurable. They hot-reload
without dropped in-flight calls. Running sessions keep the deadlines they
started with. See
[Session lifecycle](/modalities/voice/runtime/session-lifecycle).

## Next steps

  - **[Runtime overview](/modalities/voice/runtime/overview)** — How the runtime drives a live conversation, start to finish.
  - **[Configure an agent](/modalities/voice/runtime/configuration)** — Every field in the agent config, with examples.
  - **[Bring your own models](/modalities/voice/runtime/byo-asr-llm-tts)** — Swap in your own ASR, LLM, and TTS providers and keys.
  - **[Tool calling](/modalities/voice/runtime/tool-calling)** — Give the agent HTTP, MCP, and client tools.