# Self-Hosted Deployment Issues

> Diagnose the problems unique to the voice engine on your own hardware: NIC ownership and reboot cutovers, DNS/tenant provisioning, sealed credentials, and unattached inference.

Most "the self-hosted engine is broken" reports trace back to four things that
work differently on your hardware than on managed cloud:

- The engine **owns its NIC**. There is no ordinary listening socket, and
  config changes are reboot cutovers.
- **DNS and per-tenant provisioning** route calls to the right place.
- **Credentials are sealed at rest**, and the engine must be able to unseal
  them.
- **Self-hosted inference** must be explicitly attached.

This page is the triage map for all four.

> **NOTE:**
> If you did not stand the deployment up yet, start with
>   [On-Prem / Self-Hosted](/modalities/voice/quickstart/on-prem). That page
>   describes what runs where, and the shape of the install. This page assumes
>   the system is already deployed and something is wrong.

## Triage first

Match the symptom. Then go to the section that explains the cause and the fix.

| Symptom | Likely cause | Section |
|---------|--------------|---------|
| `ss` shows **no `UDP :443` listener**, but the box is "up" | Expected — the kernel-bypass fast path owns the NIC | [NIC ownership](#nic-ownership-there-is-no-normal-socket) |
| You deployed a new build / config, and **nothing changed** | Config and builds are **reboot cutovers**, not hot reloads | [Reboot cutover](#reboot-cutover-changes-need-a-reboot) |
| Carrier `INVITE`s never arrive, or land on the **wrong tenant** | Missing `SRV`/CNAME records, or a wrong subdomain label | [DNS & provisioning](#dns-and-tenant-provisioning) |
| The QUIC/WebTransport handshake fails from the browser | A missing or expired TLS cert for the brand domain (SNI) | [DNS & provisioning](#dns-and-tenant-provisioning) |
| Calls reach the agent but **trunk auth or provider keys fail** | The engine cannot unseal sealed credentials (KEK mismatch) | [Sealed credentials](#sealed-credentials-wont-unseal) |
| The agent answers but **turns fail / fall back** to a cloud model | The on-prem model server is not attached, or egress is blocked | [Inference not attached](#inference-not-attached) |
| Reports and recordings stay **empty** | Control-plane services are down, or auth to the event plane is wrong | [Control plane quiet](#control-plane-or-observability-plane-is-quiet) |

---

## NIC ownership: there is no normal socket

The engine takes **full ownership of its network interface** with a
kernel-bypass zero-copy fast path. SIP, RTP, and QUIC all ride one interface
without the kernel's UDP stack. The most common false alarm: an operator runs
`ss -ulpn` (or `netstat`), sees **no `UDP :443` and no `UDP :5060` listener**,
and concludes the engine is down.

> **WARNING:**
> A missing `UDP :443` / `UDP :5060` socket in `ss` is **normal and expected**
>   when the engine runs. The fast path steers those packets to the process
>   below the kernel socket layer, so they never show as bound sockets. Do
>   **not** use `ss`/`netstat` as your liveness check.

Check liveness the correct way instead:

  1. **Confirm the process and its interface**
Check that the engine process runs, and that the fast-path filter is
    attached to the interface. Your deploy runbook names both. If the process
    is up and the filter is loaded, the data plane is live, even with no
    visible socket.
  2. **Hit the health endpoint**
The control-plane API exposes `/health` and `/healthz`. A green health
    check plus a live process is the correct "engine is up" signal. A socket
    table is not.
  3. **Place a real probe call**
Register a softphone against the tenant SIP domain and dial in, or send a
    test `INVITE` from a trunk. A call that lands and answers is the ground
    truth. See
    [Call not connecting](/modalities/voice/troubleshooting/call-not-connecting)
    if the call does not land.

> **NOTE:**
> The fast-path filter loads **before** the engine starts (as a pre-start
>   step), so a crash-loop cannot black-hole the interface. If you must hand the
>   NIC back to the kernel (to debug with ordinary tools), stop the engine and
>   unload the filter. You then see normal sockets again, but no calls flow
>   until the engine reclaims the interface.

## Reboot cutover: changes need a reboot

Because the process owns the NIC, a new build **or a config change is a reboot
cutover**, not a hot reload. If you shipped a change and behavior did not
move, the most likely cause is that the engine host was never rebooted onto
it.

| Symptom | Cause | Fix |
|---------|-------|-----|
| You shipped a new build, but the old behavior continues | The running process still holds the NIC on the previous image | Reboot the engine host to bring the data plane up on the new build |
| The engine ignores a trunk / dialplan / agent config edit | NIC-owning modules read owned state at start. Not all settings hot-reload | Reboot. Treat engine-side config as a windowed change |
| A brief call outage occurs during a deploy | Expected — the interface quiesces while the engine is down | Deploy inside a maintenance window. Drain traffic first |

> **WARNING:**
> Plan a **maintenance window** for every engine change. While the engine is
>   down, the interface is quiesced and in-flight calls drop. This is the
>   biggest operational difference from a typical service. Budget for it in your
>   runbook.

> **TIP:**
> Some runtime tunables do reload without a reboot. The dialplan hot-reloads
>   its rules, and agent timeouts respond to a signal. These values are hydrated
>   from the config store. But anything that touches the transport, the NIC, or
>   module load order needs the reboot cutover. When in doubt, reboot.

## DNS and tenant provisioning

The engine multiplexes every tenant onto one interface. It tells tenants apart
by the **leftmost DNS label** of the inbound SIP `INVITE`, and by SNI on the
QUIC plane. If DNS is wrong, calls never arrive, or they land on the wrong
workspace.

| Symptom | Cause | Fix |
|---------|-------|-----|
| Carrier `INVITE`s never reach the engine | The SIP `SRV` record is missing, or it points at the wrong host | Create the `SRV` for the tenant SIP domain → engine host. Check with `dig SRV` |
| Calls arrive but resolve to the **wrong tenant** (or none) | The leftmost label is wrong — a `<workspace-id>.sip.firstbyte.fast` mismatch | Fix the per-tenant CNAME so the label matches the provisioned workspace id |
| The browser QUIC/WebTransport handshake fails | There is no valid TLS cert for the brand domain, or it expired | Install a valid cert. The transport does SNI across brand domains |
| The console / relay is unreachable | `portal.firstbyte.fast`, `agent.firstbyte.fast`, `relay.firstbyte.fast` do not point at a host | Point each hostname at the correct box (engine vs control plane) |

Onboarding a workspace provisions the DNS records for you. But on a
self-hosted estate, **you own the zone**. Check that the records resolve:

  1. **Verify the per-tenant SIP subdomain**
Check that `<workspace-id>.sip.firstbyte.fast` resolves to your engine
    host, and that a SIP `SRV` record routes `INVITE`s to it. The engine reads
    the leftmost label (`<workspace-id>`) as the tenant. The label **must**
    match the workspace id exactly.
  2. **Verify the WebRTC / browser subdomain**
Check that `<workspace-id>.webrtc.firstbyte.fast` resolves to the engine
    for browser-legged calls.
  3. **Verify the console, portal, and relay hostnames**
`portal.firstbyte.fast` and `agent.firstbyte.fast` point at the
    control-plane box. `relay.firstbyte.fast` points at the relay (the same
    host for single-site, or the relay mesh for multi-site).
  4. **Verify TLS for the QUIC plane**
The public QUIC/WebTransport listener needs a valid cert for each brand
    domain it serves. It selects by SNI. An expired or missing cert looks like
    a transport failure, not a DNS failure. Check both.

> **NOTE:**
> A wildcard `*.firstbyte.fast` record proves nothing about a specific label.
>   Check that the **explicit** `sip`/`webrtc`/`SRV` records exist. Do not stop
>   at a wildcard answer. If trunk `INVITE`s are the problem, cross-check
>   [SIP trunk issues](/modalities/voice/troubleshooting/sip-trunk).

## Sealed credentials won't unseal

Tenant secrets — trunk digest passwords and agent provider API keys — are
**sealed at rest**. The engine and the control plane share one
**key-encryption key (KEK)**, so the engine can unseal a secret at call time.
There are no plaintext secrets on disk. This makes the deployment HIPAA /
SOC 2 defensible. But if the two sides do not share the **same** KEK, the
engine holds ciphertext it cannot open.

| Symptom | Cause | Fix |
|---------|-------|-----|
| Trunk registration / digest auth fails for every tenant | The engine cannot unseal the sealed trunk secret (KEK mismatch) | Align the engine's KEK with the control plane's KEK. Reboot the engine |
| Every agent's provider call returns 401 / auth error | The sealed provider API key cannot be unsealed | The same KEK mismatch — align the keys, then reboot |
| Old config works, but **newly saved** secrets break | The KEK was rotated on one side only | Re-seal secrets under the current KEK on both sides |

  1. **Confirm both hosts share one KEK**
The engine host and the control-plane host must have the **same**
    key-encryption key. A mismatch is the number-one cause of "auth fails for
    everything at once."
  2. **Re-save the affected secrets**
If the KEK was rotated, secrets sealed under the old key are unreadable.
    Enter the trunk secrets and provider keys again in the console, and save
    them. This re-seals them under the current KEK.
  3. **Reboot the engine**
The engine reads the KEK at start (NIC ownership → reboot cutover).
    Reboot the engine host after any KEK change, so it picks up the new key.

> **WARNING:**
> A KEK mismatch shows as a **fleet-wide** auth failure (every tenant, every
>   provider), not as a single bad credential. If exactly one trunk or one agent
>   fails, suspect that credential, not the KEK.

## Inference not attached

On a self-hosted engine, you can still use the managed-cloud speech providers
(cloud realtime, ASR/LLM/TTS), if the engine has egress to them. Your **own**
models on-prem are a separate, explicit step. You must register the model
server with the inference control plane, and the agent's provider must point
at your endpoint over QUIC. If either step is missing, the agent silently uses
whatever provider its config still names.

| Symptom | Cause | Fix |
|---------|-------|-----|
| The agent answers, but turns go to a **cloud** model you meant to replace | The agent provider is still set to the cloud provider | Point the agent at your self-hosted endpoint |
| The on-prem model is attached, but turns error or time out | The model server is not registered with the inference control plane, or it is unreachable over QUIC | Register the server. Check that the engine can reach it |
| Cloud providers fail on-prem | Your firewall blocks egress to the provider | Allow egress, or move fully to self-hosted inference |

  1. **Register the model server**
Register your GPU-backed model server with the inference control plane, so
    the engine can discover it and route to it over QUIC. Full steps are in
    [Self-Hosted Inference](/modalities/voice/runtime/self-hosted-inference).
  2. **Point the agent at your endpoint**
Set the agent's provider/endpoint to your on-prem model. Until you do, the
    engine uses whatever provider the agent config names — often a cloud one
    from an earlier test.
  3. **Confirm reachability, then place a call**
Check that the engine can reach the model host. Then dial in and watch a
    turn complete. Use
    [Latency breakdown](/modalities/voice/observability/latency-breakdown)
    to check that turns hit your GPUs.

> **NOTE:**
> **Honest status.** The QUIC-based inference control plane that discovers and
>   routes to your GPU pods is shipped. But the fully on-prem speech-to-speech
>   model path is **not yet live-verified end-to-end** in a customer deployment.
>   If you need on-prem GPU inference from day one, plan it as a joint bring-up,
>   not a drop-in. The managed-cloud model providers work against a self-hosted
>   engine today if egress is permitted.

## Control plane or observability plane is quiet

If calls connect and sound fine, but **reports, call traces, and recordings
never appear**, the fault is on the control-plane host, not the engine. The
engine sends every call-lifecycle event, QoS/MOS record, and recording to the
control-plane services. If those services are down or unauthenticated, the
engine produces the data, but the data never lands.

| Symptom | Cause | Fix |
|---------|-------|-----|
| Reports / call traces stay empty | The event pipeline service on the control-plane host is down | Start the control-plane `systemd` units. Check that they are healthy |
| Recordings never show in the console | The recording service or object store is down or misconfigured | Check the recording service and object-store units |
| Events are silently dropped | The event plane is **always TLS + SASL authenticated** — there is no plaintext mode | Fix the SASL credentials on both the producer and broker sides |
| A service does not start at all | There is **no Docker** — units run against a pinned Node runtime under `/opt` | Check that the pinned runtime exists and that the unit's paths are correct |

  1. **Check the control-plane units**
The admin API, config database, event pipeline, and object store all run
    as `systemd` units (no containers). Check that each unit is active and
    healthy. A stopped unit is the usual cause of empty dashboards.
  2. **Verify the event plane authentication**
The internal event transport is always TLS + SASL authenticated. There is
    no plaintext fallback. A credential mismatch drops events silently. Align
    the SASL credentials.
  3. **Confirm the pinned runtime**
Services bind to a pinned Node runtime under `/opt`. If a unit fails to
    start, check that the runtime path in the unit file exists on the box.

> **NOTE:**
> Some in-console dashboards are still being wired end-to-end on self-hosted
>   installs. If a specific panel is blank while call audio clearly works, check
>   that the underlying pipeline is up before you assume a bug. See
>   [Observability overview](/modalities/voice/observability/overview) for what
>   feeds each panel.

## There is no sandbox

> **WARNING:**
> A self-hosted deployment is a **single production environment**. There is
>   intentionally no separate sandbox toggle. Do your tests in a dedicated
>   workspace (or a second install). Never test against a "sandbox" that does
>   not exist. Treat prod as prod. This prevents a class of "why does my test
>   call bill / page the on-call" surprises.

## Related

  - **[On-Prem / Self-Hosted](/modalities/voice/quickstart/on-prem)** — The install itself: what runs where, and the reboot-cutover deploy flow.
  - **[Deployment Models](/modalities/voice/concepts/deployment-models)** — Managed, self-hosted, and hybrid — how to split the media and control planes.
  - **[Self-Hosted Inference](/modalities/voice/runtime/self-hosted-inference)** — Register your GPU model server and route turns to it over QUIC.
  - **[Call not connecting](/modalities/voice/troubleshooting/call-not-connecting)** — When the probe call itself does not land or answer.