# Inference Console

> Configure the voice AI: select models and providers, test in the playground, and monitor runtime health.

The Inference console is the ready-made web app that runs FirstByte
Inference. Inference is the self-hosted, OpenAI-compatible endpoint
behind your voice AI. From one place you can:

- Deploy and manage served models.
- Configure LLM/ASR/TTS vendor credentials.
- Test completions live.
- Monitor GPU and runtime health.

The console is for operators and developers who own an inference tenant.

## What's in the console

  - **Overview** — KPI strip, your tenant-scoped OpenAI-compatible endpoint, cURL quickstart,
    and the list of active models.
  - **Models** — The served-model registry. Deploy from HuggingFace or a weights path,
    inspect the deploy config, and remove models.
  - **Providers** — Credential manager and default routing for your LLM, speech-to-text (ASR),
    and text-to-speech (TTS) vendors.
  - **Playground** — Chat/completion console and voice-pipeline tester for live, streaming
    requests against a deployed model.
  - **Runtime** — GPU/CPU/memory gauges, self-test diagnostics, and the runtime config for the
    serving host.
  - **Analytics** — Token usage, latency percentiles, and per-model breakdowns over 24h / 7d / 30d.
  - **API keys** — Base URLs and copy-paste SDK quickstarts (Python, Node, cURL).
  - **Usage & billing** — Metered output-token usage, cost, and invoices for your tenant.

> **NOTE:**
> Most screens are scoped to the **tenant** that you are connected to (shown
>   in the Overview subtitle). If no tenant is connected, the base URL and the
>   quickstarts show a placeholder until you connect one.

## Read the overview and grab your endpoint

Start on **Overview**. Check that the fabric is online. Copy the endpoint that
your apps will call.

  1. **Open Overview**
The header shows your tenant and environment. Use the time-range toggle
    (**15m / 1h / 24h**) to scope the KPI strip. Use **Refresh** to re-pull
    the data.
  2. **Check the endpoint status**
The **OpenAI-compatible endpoint** card shows an **online / unavailable**
    pill and the supported routes: `/chat/completions`,
    `/audio/transcriptions`, `/audio/speech`, `/models`, `/health`.
  3. **Copy the tenant-scoped base URL and cURL**
Copy the **Base URL** (tenant-scoped) with the copy button. Then copy the
    **cURL quickstart** to make your first streaming request. Replace
    `<model-id>` with one of your deployed models.
  4. **Jump to a workflow**
Use **Open playground** to test live. Or click **Manage** on the **Active
    models** card to go to the model registry.

## Deploy a model

The **Models** screen is your served-model registry. The stat strip summarizes
model count, GPUs requested, replicas, and engines. The table lists every
deployed model with its engine, namespace, GPUs, replicas, revision, and status.

> **WARNING:**
> **Deploy model** and **Remove** are disabled unless registry writes are
>   enabled for your fabric. When registry writes are disabled, the console
>   shows a banner and greys out the buttons. An operator must enable registry
>   writes first.

  1. **Open Models and start a deploy**
Click **Deploy model** in the header. A **Deploy model** drawer opens on the
    right.
  2. **Choose a source**
Select **HuggingFace Hub** to pull a repo directly from the Hub. Or select
    **Custom path** to serve from a weights path already on the node or image.
  3. **Name the model**
Enter a **Name**. Use lowercase alphanumeric characters and dashes only
    (for example, `llama-3-8b`). The form shows an inline warning if the name
    is invalid.
  4. **Point at the weights**
For HuggingFace, enter the **repo id** (for example,
    `meta-llama/Llama-3.1-8B-Instruct`). For a custom path, enter the
    **weights path** (for example, `/models/llama-3-8b`).
  5. **Add HuggingFace auth (gated/private repos only)**
For a gated or private Hub repo, add an **HF token**. When you set a
    token, you must also enter a **Tenant / org**. The console seals the token
    under that tenant before it stores the token. Optionally, pin a
    **Revision** (a Hub commit or branch).
    > **TIP:**
> HF tokens are encrypted at rest for each tenant. The engine opens them
>       into the pod only at deploy time.
  6. **Set serving details**
Check the serving **Image**, **Engine**, and **Namespace**. Then check the
    **GPUs**, **Replicas**, and **Port**. Defaults are prefilled.
  7. **Deploy**
Click **Deploy**. This writes the desired state into the registry.
    > **NOTE:**
> A deploy records the desired state. It does not hot-reload a running
>       module. The fabric applies the new model on its next hydrate (module
>       start).

### Inspect or remove a model

  1. **Open a model**
Click any row in the **Deployed models** table to open its drawer. The
    drawer shows capabilities, sealed-credential status, the weights path, and
    the full deploy config (image, engine, namespace, replicas, GPUs, pinned
    revision, service port).
  2. **Filter or rescan**
Use **Filter models…** to narrow the table by name, path, engine, or
    namespace. Use **Rescan registry** in the header to re-pull the registry.
  3. **Remove**
Click **Remove** on a row, or click **Remove from registry** in the
    drawer. This removes the model from the registry. Registry writes must be
    enabled.

## Configure providers and default routing

The **Providers** screen manages vendor credentials. It also sets which vendor
is the default route for each category: **LLM providers**, **Speech-to-text
(ASR)**, and **Text-to-speech (TTS)**. The summary cards at the top show the
current default route and the number of healthy vendors in each category.

  1. **Review the categories**
Each category card lists its configured providers with name, **base URL**,
    **API key**, a health pip, and (when known) latency and model count.
  2. **Add a provider**
Click **Add provider** in the header to configure a new vendor credential.
  3. **Reveal a key**
On a provider row, use the eye toggle next to **API key** to show or hide
    the stored key. Providers that are served locally show **no credential ·
    local** instead of a key.
  4. **Test a provider**
Click **Test** on a row to run a health check against that vendor. Or
    click **Test all** in the header to check every configured provider. You
    cannot test offline providers.
  5. **Set the default route**
Click **Route** on a provider to make it the **default** for its category.
    The provider then shows a **default** tag and updates the summary card.

## Test a model live in the playground

The **Playground** has two modes: **Chat / completion** and **Voice
pipeline**. Select a mode with the toggle in the header.

  1. **Open the Chat / completion tab**
If no chat-capable model is deployed, the transcript shows **No models
    deployed yet** with a shortcut to **Models**. Otherwise the console is ready.
  2. **Pick a model and set parameters**
In the left panel, select a **Model**. Optionally, add a **System
    prompt**. Set **temperature** and **max_tokens** with the sliders. Toggle
    **stream (SSE)** on or off.
  3. **Send a message**
Type in the composer. Press **Enter** or **Send**. The transcript renders
    the exchange. During streaming, the header shows live time-to-first-token
    (**ttft**) and elapsed time.
  4. **Track the session**
The left panel's **This session** block shows tokens in, the endpoint
    (`/v1/chat/completions`), and the transport. Use **Clear** to reset the
    transcript.

> **TIP:**
> Switch to the **Voice pipeline** tab to test a mic-driven speech pipeline
>   and to see a stage-by-stage waterfall trace. If no voice pipeline is
>   configured, use **Configure a voice pipeline** to go to **Providers**.

## Monitor runtime health

The **Runtime** screen shows the live health of the serving host: a host banner
(host name, version, uptime, engine), resource gauges, self-test
results, and the serving config. The engine does not publish runtime stats
today. When it does not, the screen hides the banner and the gauges and
shows a note. The Routing card stays.

  1. **Read the resource gauges**
Three cards show **GPU** (compute utilization, VRAM, temperature, power),
    **CPU** (load, load average, tensor-parallel, max-model-len), and **System
    memory** (used, models mapped, page cache, swap).
  2. **Refresh the runtime**
Click **Refresh** to re-read the runtime blob that the engine publishes.
    The button does not run any test.
    The **Self-test results** card lists each boot diagnostic with
    pass/warn/fail status and timing. The banner pill shows whether all
    systems are nominal. It shows only when the engine reports self-tests.
  3. **Check the serving config**
The **Serving config** card lists the values that the engine reports (served
    model name, tensor-parallel size, GPU memory utilization, max context,
    dtype).

> **NOTE:**
> The console has no restart button. Restarts are a cluster-operator action,
>   not a tenant capability.

## Review analytics and billing

  1. **Open Analytics**
Use the range toggle (**24h / 7d / 30d**) to scope token usage, latency
    percentiles, per-model breakdowns, and utilization.
    > **NOTE:**
> Analytics populate after your fabric starts to emit completion
>       telemetry. Until then, the screen shows an empty state instead of zeros.
  2. **Open Usage & billing**
Review metered **output-token** usage, cost, and invoices. The
    **Metering** card explains how charges accrue. Usage is counted per
    request and scoped to your tenant for chargeback.

## Integrate from code (API keys)

The **API keys** screen gives you everything that you need to call the
fabric from an app.

  1. **Copy a base URL**
Copy the **Global** or **Tenant-scoped** base URL. The endpoint is
    OpenAI-compatible. Point the official OpenAI SDK at it: only the base URL
    changes.
  2. **Copy an SDK snippet**
Switch the language toggle between **Python**, **Node**, and **cURL**.
    Copy the streaming quickstart. Set your API key in the
    `CLUTCHCALL_CREDENTIALS` environment variable. Replace `<model-id>` with a
    deployed model.
  3. **Enable tool calling (optional)**
You enable tool/function calling server-side, for each model, in the
    deploy spec. After you enable it, the OpenAI SDK's `tools` field works
    unchanged.

> **WARNING:**
> You manage API keys under **API keys**. Treat them like passwords. Anyone
>   with a valid key can call your tenant-scoped endpoint and incur usage.

## Next steps

- [Inference — Details](/modalities/inference/details) — how the voice AI leg
  works end-to-end.
- [Inference — SDK Methods](/modalities/inference/sdk-methods) — attach an agent
  and drive it from code.
- [Turn Detection](/concepts/turn-detection) — turn-taking, barge-in, and
  cascade (ASR → LLM → TTS) vs speech-to-speech.