# High Availability

> Run Voice across two or more nodes. A single host failure then never takes down calls, routing, or agent state.

A single FirstByte Voice node runs everything: signalling, media, routing,
and agent state. It runs fine that way. But one node is one failure domain. If
that host dies, everything on it dies with it. **High availability is
opt-in.** To turn it on, run **two or more engine nodes** and enable the
replicated control plane that keeps them in agreement. Below two nodes there is
nothing to fail over to. A single node is never truly HA, whatever you
configure.

When HA is off, there is **zero overhead**. The control plane runs in
single-node mode and never touches the network for consensus. You opt in only
when you deploy more than one node.

> **NOTE:**
> HA covers the **control plane**: membership, call/session ownership, and
>   routing. That failover is shipped and exercised. Media continuity across a
>   mid-call node loss has more conditions. Read
>   [What survives a failure](#what-survives-a-failure)
>   before you design around seamless mid-call survival.

## How it works

Every node runs the same engine. A **replicated control plane** makes a set of
nodes a **cluster**. This control plane is a single authoritative, replayable
log of cluster state. Every node holds a copy of the log. The log covers three
areas:

  - **Membership** — Which nodes are alive and in the cluster right now. Nodes **heartbeat**
    each other for liveness. The cluster declares a node down when the node
    stops heartbeating.
  - **Ownership** — Which node currently owns each call, session, and agent registration. Every
    live entity has exactly one owning node at a time.
  - **Routing** — Where new calls and control verbs should land: trunk bindings, agent
    availability, and queue/skill assignments. With this data, any node can
    steer a request to the node that owns the target.

The nodes themselves **replicate the log**. There is **no external coordinator
to run or maintain**. You do not deploy a separate consensus service, quorum
store, or lock manager next to the engine. The cluster reaches agreement
internally over the same secured transport that it already uses. The
operational surface stays at "run N engine nodes."

### Boot snapshot for fast recovery

A full replay of a long-lived cluster's history would make a restarting node
slow to rejoin. Instead, the control plane periodically writes a **boot
snapshot**. The snapshot is a compacted point-in-time image of membership,
ownership, and routing. A node that starts or restarts loads the latest
snapshot. It then replays only the tail of the log after the snapshot. The node
catches up to current cluster state in seconds and does not reprocess the full
history. This mechanism makes rolling restarts and node replacement quick.

### No single point of failure

The log and its snapshot live on **every** node. As a result, the loss of one
host cannot stop the cluster from making progress. If you lose a node, the
survivors still hold the authoritative state. They re-elect ownership for
everything that the dead node held. They keep accepting new work. There is no
primary database, no leader-only store, and no coordinator process whose death
is fatal.

## What happens on a node failure

  1. **The node stops heartbeating**
Its peers miss its heartbeats past the liveness deadline. They mark the
    node **down** in the replicated membership state.
  2. **Ownership is reassigned**
Surviving nodes take ownership, through the control plane, of everything
    that the dead node **owned**: call/session ownership and agent
    registrations. No operator action is required.
  3. **Routing moves to survivors**
The cluster steers new calls, control verbs, and agent traffic away from
    the dead node. This traffic goes to the nodes that now own the work. Trunk
    and relay bindings converge on the survivors.
  4. **The node rejoins cleanly**
When the host comes back, it loads the latest **boot snapshot**. It replays
    the log tail. It re-enters the cluster. Then it can take a share of
    ownership again.

## What survives a failure

Be precise about what "highly available" buys you. Control-plane failover and
media continuity are different guarantees.

  - **Shipped: control-plane failover** — Membership, ownership, and routing move to surviving nodes automatically.
    The cluster keeps **accepting and routing new calls** through the failure.
    The replicated log preserves agent state and registrations. Control verbs
    (including the [CSTA CTI](/modalities/voice/cti/overview) surface) continue
    to resolve against the survivors.
  - **Best-effort: media on the failed node** — A call loses its RTP path when its **media was terminating on the node
    that died**. The control plane recovers ownership and routing. But the
    cluster does not move audio in flight on the dead host to a survivor
    mid-stream. Clients that reconnect, and new legs, land on healthy nodes.

> **WARNING:**
> A hard node loss is **not seamless** for a call whose media was terminating
>   on the failed node. Plan your SLAs with this in mind.
>   The guarantee today is that the **cluster** stays available. New calls,
>   routing, and agent/CTI control keep working through a node failure. Calls
>   not pinned to the dead node are unaffected. Calls that actively terminate
>   media on the lost node drop, and those calls must reconnect.

## Deploy for HA

  1. **Run two or more engine nodes**
Run a minimum of **two** nodes. Run more for headroom, so that a single
    failure still leaves enough capacity. Each node is a full engine. There are
    no special-role hosts.
  2. **Front them with a relay or anycast**
Put the nodes behind `relay.firstbyte.fast` (or an anycast address).
    Clients then reach a single stable endpoint, and the relay steers them to a
    live node. Signalling and media land on the node that the relay selects.
    The control plane resolves ownership from there.
  3. **Enable the replicated control plane**
Turn on cluster mode. The nodes then form the replicated log and start to
    heartbeat. This is the opt-in piece. It does nothing on a single node. It
    engages only when peers are present.
  4. **Verify failover before you rely on it**
Drain or kill a node. Check that new calls still route. Check that agents
    stay registered. Check that CTI verbs resolve. Then bring the node back and
    check that it rejoins from its boot snapshot.

> **NOTE:**
> HA is an add-on to any self-managed deployment. It pairs with
>   [self-hosted / on-prem](/modalities/voice/quickstart/on-prem). On **managed
>   cloud**, we operate the multi-node cluster for you. You do not configure it.

## Related

  - **[Deployment models](/modalities/voice/concepts/deployment-models)** — Managed cloud, self-hosted, and hybrid, and which planes you run.
  - **[On-prem quickstart](/modalities/voice/quickstart/on-prem)** — Stand up the engine and control plane in your own environment.
  - **[CSTA CTI](/modalities/voice/cti/overview)** — Third-party call control that resolves against the same clustered nodes.
  - **[Voice architecture](/modalities/voice/concepts/architecture)** — The end-to-end signalling, media, and runtime path.