# Inference Runtime and Routing

> How the engine publishes runtime stats, what the resource gauges and self-tests show, and how routing strategies, weighted scorers, and the metric profile are published to a live module.

The **Runtime** screen is the operator view of the inference engine. It has two
parts. The first part shows the state of the host that runs the clutch-layers
model server: GPU, CPU, memory, boot self-tests, and serving config. The second
part is the **Routing** card. It shows the live routing strategy that
`mod_inference` applies when it picks a pod for a request.

The screen uses two read procedures, `runtime.health` and `runtime.routing`, and
one write, `runtime.setRouting`. Nothing on the screen reaches into the cluster
directly.

Afterburner serves only clutch-layers models, on model servers that
FirstByte runs. The Runtime screen has no other engine lane.

## What the runtime publishes and where it is stored

The runtime does not answer console queries. It **publishes a blob**, and the
console reads it.

| Data | Source | Read by |
| ---- | ------ | ------- |
| Host identity, version, uptime, engine | runtime blob in Redis (`clutchcall:inference:runtime`) | `runtime.health` |
| GPU / CPU / memory gauges | same blob | `runtime.health` |
| Self-test results | same blob | `runtime.health` |
| Serving config | same blob | `runtime.health` |
| Desired routing strategy | routing key published by the console | `runtime.routing` → `desired` |
| Effective routing strategy | `<routing key>.effective`, republished by a live module | `runtime.routing` → `effective` |

Three results follow from this design:

- A field shown as `—` was **absent from the published blob**. It is not a
  zero. The console does not substitute a default.
- Freshness is the property of the publisher. The console shows what is in
  Redis when it runs the query.
- **Today the engine does not write the runtime blob.** When the blob is absent,
  `runtime.health` reports that nothing is published. The console then hides the
  host banner, the gauges, the self-tests, and the serving config. A note tells
  you that the engine publishes no runtime stats. The Routing card stays on the
  screen, because it reads a different source.

## Host banner fields

The banner at the top of the screen shows only when the engine has published a
runtime blob. The console reads it straight from that blob.

| Field | Meaning |
| ----- | ------- |
| Host | The hostname that the runtime reports. |
| Version | The version string that the runtime published. |
| Uptime | Process uptime that the runtime reports. |
| Engine | The engine that the runtime names itself as. The console shows `not reported` when the blob has no value. |
| Systems nominal pill | A count of self-tests with status `pass` over all self-tests in the blob. The pill shows only when the blob holds at least one self-test. |

The banner has no Listener field. The console does not show a listener address
or port, because the engine does not publish one.

## Reading the resource gauges

Three cards show host resources. The donut in each card is a percentage. The
bars beside it are meters. The cards show only when the engine has published a
runtime blob.

**GPU** — device name, compute utilization (donut), then VRAM used against VRAM
total, temperature, and power draw against the reported power cap. The `CUDA`
tag shows only when the blob names a GPU device.

**CPU** — model and core count, load (donut), then 1-minute loadavg, the
tensor-parallel width, and the maximum context length in thousands of tokens.

**System memory** — host RAM total, used as a share of total (donut), then the
number of memory-mapped models, page cache, and swap.

Two rules apply when you read a number off this screen:

- **The memory donut needs a total.** If the blob has no memory total, the
  ratio is undefined. The donut shows `0%` and the sub-label reads
  `not reported`. It does not show `NaN%`.
- **Some bars are relative. They are not capacity limits.** Some rows have no
  natural maximum: tensor-parallel width, maximum context, models mapped, page
  cache, swap, and loadavg. For these rows, the full scale of the bar is the
  larger of the reported value and a fixed display floor. The bar shows
  movement. Read the number in the right-hand label, not the fill fraction.

VRAM, temperature, and power have a real denominator (`vramTotal`, a fixed
temperature scale, and `powerCap`). Their fill fractions are meaningful.

<a id="runtime-config-and-what-each-vllm-argument-means"></a>
## Serving config and what each field means

The **Serving config** card lists the values that the engine reports for the
model server. Every row comes from the runtime blob. The console fills in no
row itself. The console hides a row when the blob has no value for it. When the
blob has none of these values, the card says that the engine did not report
serving config.

| Field | Blob key | What it controls |
| ----- | -------- | ---------------- |
| Served model | `servedModel` | The model name that clients use in requests. Scraped metrics use the same name as key. |
| Tensor parallel | `tensorParallel` | The number of GPUs that hold shards of the model weights in one runtime instance. The CPU card shows the same value. |
| GPU memory use | `gpuMemUtil` | The fraction of GPU memory that the server may pre-allocate. This sets the size of the KV cache pool. |
| Max context | `maxModelLen` | The maximum context length in tokens. The CPU card shows the same value. |
| Dtype | `dtype` | The weight and activation datatype that the runtime loaded. |

The card has no fixed rows. It shows no bind address and no port, and it shows
no tool-choice flag.

## Self-tests and how to read them

Each entry in the self-test list is a boot diagnostic that the runtime
published. Each entry has four fields:

| Field | Meaning |
| ----- | ------- |
| `name` | The name of the diagnostic, as the runtime names it. |
| `detail` | The one-line result string from the runtime. This is where the substance is: which device, which path, which endpoint. |
| `ms` | The time that the check took, in milliseconds. |
| `status` | `pass`, `warn`, or a failure. `pass` shows green with a check. All other values show an alert icon: `warn` is amber and a failure is red. |

The console does **not** fix the set of tests. The screen shows the tests that
are in the blob, in the order published. If a check you expect is missing, the
runtime did not report it. This is a gap on the publisher side. It is not a
pass. When the blob has a runtime but no tests, the card says that no
self-tests were reported. The console does not show a "systems nominal" pill
without tests.

Use `ms` as a signal. A check that passes but takes much longer than the other
checks usually means that the resource it used was slow to respond. This
often shows later as latency, not as a failure.

### What Refresh does

The **Refresh** button calls `refetch()` on `runtime.health`. It re-reads the
runtime blob. It does **not** tell the runtime to run its boot diagnostics
again. It shows no fake progress.

- If the runtime has published a new blob since your last look, you see new
  results.
- If it has not, you see the same results again.

To run the boot diagnostics again, an operator must restart the runtime. The
console cannot do this.

## Routing strategies and the metrics they require

`runtime.routing` returns the strategy list for the org; the console renders one
button per strategy and does not hardcode names. Selecting a button only sets a
local draft — nothing is published until you press **Apply strategy**.

The important field alongside the list is `requires`: a per-strategy list of the
**scraped metrics that strategy needs in order to score at all**. The console
shows it inline under the strategy picker:

- If the selected strategy requires metrics, the hint names them and warns that
  without them the router **falls back to a random ready pod**. Remember this
  failure mode: a KV-aware or load-aware strategy whose metrics are not scraped
  does not give an error. It silently changes to random selection. Your latency
  then becomes the latency of no routing at all.
- If the strategy requires no scraped metrics, the hint says so — that strategy
  routes from prompt affinity and live in-flight load, both of which the module
  observes itself.

Before you adopt a metrics-dependent strategy, confirm the metrics named in the
hint are actually being scraped for the pods you are routing across.

## Weighted multi-scorer routing

The `multi (weighted)` option composes several scorers instead of picking one.
Its strategy string is `multi:` followed by comma-separated `name:weight` pairs.
Each scorer is normalized to `[0,1]` and the normalized scores are combined by
weight. The console seeds a starting spec of:

```
multi:least-request:3,least-kv-cache:1
```

`runtime.routing` returns `multiScorers` — the set of scorer names that can be
weighted. This set is narrower than the full strategy list: not every strategy
is weightable.

The spec you type goes to the live router **verbatim**, so the console parses it
before publishing and shows the first problem instead of publishing a broken
spec:

| Condition | What you get |
| --------- | ------------ |
| Spec is empty | Asked to enter at least one `name:weight` pair. |
| An empty entry (e.g. a double comma) | Asked to remove it and separate pairs with single commas. |
| An entry with no `:`, or a missing name or weight | Told the entry is not a `name:weight` pair, with an example. |
| A name that is not in `multiScorers` | Told that name is not weightable, and given the list that is. |
| A weight that is not a finite number above zero | Told the weight must be above zero. |

Parsing splits each pair at its **last** colon, so the name is everything before
it and the weight everything after. Validation errors surface when you press
**Apply strategy**; the input turns invalid and the message appears rather than
the mutation firing.

<a id="metric-profiles-per-engine"></a>
## Metric profile

The **Metric profile (static routes)** control selects **which engine's
`/metrics` names the scraper parses for pods in the static route table**. Pods
that the router discovers dynamically use their own engine label instead. This
setting governs only statically-routed pods.

The control has one option: `clutch-layers`. The Routing card shows the profile
in force next to the strategy. When no profile is reported, the card shows
`not reported`. If the engine reports a profile that is not `clutch-layers`,
the card names it, and the console can set only `clutch-layers`.

For `clutch-layers`, the scraper reads these metrics from the model server:

| Router metric | clutch-layers metric | Note |
| ------------- | -------------------- | ---- |
| `num_requests_running` | `clutch_active_slots` | Requests that run now. |
| `num_requests_waiting` | `clutch_queue_depth` | Requests that wait. |
| KV cache usage (GPU) | `clutch_kv_pages_used`, `clutch_kv_pages_free` | The router computes `used / (used + free)`. |
| KV cache usage (CPU) | same two metrics | A constant `0`. clutch-layers has no host-offload KV cache. The row exists so that `least-kv-cache` does not skip the pod. |
| Request queue time | `clutch_queue_wait_seconds` | A histogram. The server renders it only in some builds. |

The scraper does not read the prefix-cache counters, prefill and decode
histograms, throughput gauges, or a busy-time ratio. A strategy that needs one
of these metrics stays cold. This is on purpose: a wrong value is worse than no
value.

**Set the engine on every model.** A model with no engine set is scraped with
the built-in default profile of the module. That profile does not read the
clutch-layers metric names, so the metrics-based strategies fall back to a
random ready pod for that model. Set `engine` to `clutch-layers` in the
[model registry](/modalities/inference/model-registry).

The console publishes `routesEngine` **only when you pick a profile**. The
displayed value is the effective profile, with the desired profile as fallback.
If the console sent that displayed value again, it would rewrite a field that
you did not touch. When you do not touch the control, `setRouting` carries the
strategy only.

## Desired state, effective state, and the poll cycle

The routing card keeps two facts apart and never conflates them:

| Shown as | Field | Meaning |
| -------- | ----- | ------- |
| **In force (engine)** | `effective.strategy` | What a running `mod_inference` reports it has **applied**. |
| **Requested** | `desired.strategy` | What was last **published** from this console. `— (module default)` when nothing has been published. |
| **Metric profile** | `effective.routes_engine`, falling back to `desired.routes_engine` | The metric naming in use for static routes. |
| **Source** | `effective.source` | Where the reporting module says the strategy in force came from. |

The effective record is a **TTL'd beacon**: the module republishes it on every
poll. `engineReporting` is true only while that beacon is present. The status
pill reflects the combination:

| Pill | Condition |
| ---- | --------- |
| `engine in sync` | The engine is reporting and `inSync` is true (or nothing has been requested yet). |
| `pending pickup` | The engine is reporting, but what it has applied differs from what was requested. |
| `engine not reporting` | No beacon. |

When there is no beacon, the card explains that no live `mod_inference` is
publishing routing state at `<key>.effective`, and that a strategy published
here is **stored as desired state and adopted when a module starts polling**.
FirstByte deliberately does not show a green confirmed strategy in that
situation, because a strategy no engine has acknowledged may be steering nothing.

Switching strategies takes effect on the module's **next poll (~5 s)**. There is
no restart, and no request in flight is re-routed.

Two details of the control's behaviour follow from this model:

- The strategy picker initialises from **what is actually running** (`effective`),
  falling back to what was requested. It shows reality first, intent second.
- **Apply strategy** is enabled only when your draft differs from the applied
  strategy, or when you have explicitly picked a metric profile. Otherwise
  there is nothing to publish. **Reset** discards the draft and returns the
  picker to the applied state.

If the org has no strategies at all, the card replaces the controls with an
empty state: routing data could not be loaded, check that inference routing is
enabled for the org, then retry.

## Read-only deployments and operator-only actions

Two restrictions apply on this screen. They have different causes.

**Routing writes can be disabled per deployment, or for your account.** `runtime.routing` returns
`writable`. When it is false, every strategy button, the weights input, the
metric-profile selector, and **Apply strategy** are disabled, the card shows
`Read-only on this deployment`, and the apply button's tooltip names the gate:
`INFERENCE_DEPLOY_ENABLED`. Reads continue to work. You can still see what is
in force, what was requested, and which metrics each strategy needs.

A second gate applies per account. `runtime.routing` also returns `canWrite`.
It is false when your account is not an inference operator
(`INFERENCE_ADMIN_EMAILS`). In that case the controls are disabled and the card
shows `Read-only for your account`.

**The console has no worker restart.** A worker restart is a rollout that a
cluster operator does. It is not an action in the tenant console, so the screen
has no Restart button.

## Routed upstreams

Below the controls, the **Routed upstreams** table lists each model in the
static route table with its endpoint. For each endpoint it shows whether the
address answers, how many turns it answered, how many it did not, and the time
of its last answer.

The verdict comes from real turns. It is not a probe. An endpoint that nobody
has called reads `untried`. An endpoint that has never answered shows `never`
in the last-answer column. If the engine reports routing but not upstream
health, every endpoint reads `untried`. Routing still works in that case. The
table lists only the static route table. A model that k8s discovery serves does
not appear in it.