The Runtime screen is the operator view of the inference engine. It has two parts. The first part shows the state of the host that runs the clutch-layers model server: GPU, CPU, memory, boot self-tests, and serving config. The second part is the Routing card. It shows the live routing strategy that mod_inference applies when it picks a pod for a request. The screen uses two read procedures, runtime.health and runtime.routing, and one write, runtime.setRouting. Nothing on the screen reaches into the cluster directly. Afterburner serves only clutch-layers models, on model servers that FirstByte runs. The Runtime screen has no other engine lane.

​
What the runtime publishes and where it is stored

The runtime does not answer console queries. It publishes a blob, and the console reads it. Three results follow from this design:
  • A field shown as — was absent from the published blob. It is not a zero. The console does not substitute a default.
  • Freshness is the property of the publisher. The console shows what is in Redis when it runs the query.
  • Today the engine does not write the runtime blob. When the blob is absent, runtime.health reports that nothing is published. The console then hides the host banner, the gauges, the self-tests, and the serving config. A note tells you that the engine publishes no runtime stats. The Routing card stays on the screen, because it reads a different source.

​
Host banner fields

The banner at the top of the screen shows only when the engine has published a runtime blob. The console reads it straight from that blob. The banner has no Listener field. The console does not show a listener address or port, because the engine does not publish one.

​
Reading the resource gauges

Three cards show host resources. The donut in each card is a percentage. The bars beside it are meters. The cards show only when the engine has published a runtime blob. GPU — device name, compute utilization (donut), then VRAM used against VRAM total, temperature, and power draw against the reported power cap. The CUDA tag shows only when the blob names a GPU device. CPU — model and core count, load (donut), then 1-minute loadavg, the tensor-parallel width, and the maximum context length in thousands of tokens. System memory — host RAM total, used as a share of total (donut), then the number of memory-mapped models, page cache, and swap. Two rules apply when you read a number off this screen:
  • The memory donut needs a total. If the blob has no memory total, the ratio is undefined. The donut shows 0% and the sub-label reads not reported. It does not show NaN%.
  • Some bars are relative. They are not capacity limits. Some rows have no natural maximum: tensor-parallel width, maximum context, models mapped, page cache, swap, and loadavg. For these rows, the full scale of the bar is the larger of the reported value and a fixed display floor. The bar shows movement. Read the number in the right-hand label, not the fill fraction.
VRAM, temperature, and power have a real denominator (vramTotal, a fixed temperature scale, and powerCap). Their fill fractions are meaningful.

​
Serving config and what each field means

The Serving config card lists the values that the engine reports for the model server. Every row comes from the runtime blob. The console fills in no row itself. The console hides a row when the blob has no value for it. When the blob has none of these values, the card says that the engine did not report serving config. The card has no fixed rows. It shows no bind address and no port, and it shows no tool-choice flag.

​
Self-tests and how to read them

Each entry in the self-test list is a boot diagnostic that the runtime published. Each entry has four fields: The console does not fix the set of tests. The screen shows the tests that are in the blob, in the order published. If a check you expect is missing, the runtime did not report it. This is a gap on the publisher side. It is not a pass. When the blob has a runtime but no tests, the card says that no self-tests were reported. The console does not show a “systems nominal” pill without tests. Use ms as a signal. A check that passes but takes much longer than the other checks usually means that the resource it used was slow to respond. This often shows later as latency, not as a failure.

​
What Refresh does

The Refresh button calls refetch() on runtime.health. It re-reads the runtime blob. It does not tell the runtime to run its boot diagnostics again. It shows no fake progress.
  • If the runtime has published a new blob since your last look, you see new results.
  • If it has not, you see the same results again.
To run the boot diagnostics again, an operator must restart the runtime. The console cannot do this.

​
Routing strategies and the metrics they require

runtime.routing returns the strategy list for the org; the console renders one button per strategy and does not hardcode names. Selecting a button only sets a local draft — nothing is published until you press Apply strategy. The important field alongside the list is requires: a per-strategy list of the scraped metrics that strategy needs in order to score at all. The console shows it inline under the strategy picker:
  • If the selected strategy requires metrics, the hint names them and warns that without them the router falls back to a random ready pod. Remember this failure mode: a KV-aware or load-aware strategy whose metrics are not scraped does not give an error. It silently changes to random selection. Your latency then becomes the latency of no routing at all.
  • If the strategy requires no scraped metrics, the hint says so — that strategy routes from prompt affinity and live in-flight load, both of which the module observes itself.
Before you adopt a metrics-dependent strategy, confirm the metrics named in the hint are actually being scraped for the pods you are routing across.

​
Weighted multi-scorer routing

The multi (weighted) option composes several scorers instead of picking one. Its strategy string is multi: followed by comma-separated name:weight pairs. Each scorer is normalized to [0,1] and the normalized scores are combined by weight. The console seeds a starting spec of:
runtime.routing returns multiScorers — the set of scorer names that can be weighted. This set is narrower than the full strategy list: not every strategy is weightable. The spec you type goes to the live router verbatim, so the console parses it before publishing and shows the first problem instead of publishing a broken spec: Parsing splits each pair at its last colon, so the name is everything before it and the weight everything after. Validation errors surface when you press Apply strategy; the input turns invalid and the message appears rather than the mutation firing.

​
Metric profile

The Metric profile (static routes) control selects which engine’s /metrics names the scraper parses for pods in the static route table. Pods that the router discovers dynamically use their own engine label instead. This setting governs only statically-routed pods. The control has one option: clutch-layers. The Routing card shows the profile in force next to the strategy. When no profile is reported, the card shows not reported. If the engine reports a profile that is not clutch-layers, the card names it, and the console can set only clutch-layers. For clutch-layers, the scraper reads these metrics from the model server: The scraper does not read the prefix-cache counters, prefill and decode histograms, throughput gauges, or a busy-time ratio. A strategy that needs one of these metrics stays cold. This is on purpose: a wrong value is worse than no value. Set the engine on every model. A model with no engine set is scraped with the built-in default profile of the module. That profile does not read the clutch-layers metric names, so the metrics-based strategies fall back to a random ready pod for that model. Set engine to clutch-layers in the model registry. The console publishes routesEngine only when you pick a profile. The displayed value is the effective profile, with the desired profile as fallback. If the console sent that displayed value again, it would rewrite a field that you did not touch. When you do not touch the control, setRouting carries the strategy only.

​
Desired state, effective state, and the poll cycle

The routing card keeps two facts apart and never conflates them: The effective record is a TTL’d beacon: the module republishes it on every poll. engineReporting is true only while that beacon is present. The status pill reflects the combination: When there is no beacon, the card explains that no live mod_inference is publishing routing state at <key>.effective, and that a strategy published here is stored as desired state and adopted when a module starts polling. FirstByte deliberately does not show a green confirmed strategy in that situation, because a strategy no engine has acknowledged may be steering nothing. Switching strategies takes effect on the module’s next poll (~5 s). There is no restart, and no request in flight is re-routed. Two details of the control’s behaviour follow from this model:
  • The strategy picker initialises from what is actually running (effective), falling back to what was requested. It shows reality first, intent second.
  • Apply strategy is enabled only when your draft differs from the applied strategy, or when you have explicitly picked a metric profile. Otherwise there is nothing to publish. Reset discards the draft and returns the picker to the applied state.
If the org has no strategies at all, the card replaces the controls with an empty state: routing data could not be loaded, check that inference routing is enabled for the org, then retry.

​
Read-only deployments and operator-only actions

Two restrictions apply on this screen. They have different causes. Routing writes can be disabled per deployment, or for your account. runtime.routing returns writable. When it is false, every strategy button, the weights input, the metric-profile selector, and Apply strategy are disabled, the card shows Read-only on this deployment, and the apply button’s tooltip names the gate: INFERENCE_DEPLOY_ENABLED. Reads continue to work. You can still see what is in force, what was requested, and which metrics each strategy needs. A second gate applies per account. runtime.routing also returns canWrite. It is false when your account is not an inference operator (INFERENCE_ADMIN_EMAILS). In that case the controls are disabled and the card shows Read-only for your account. The console has no worker restart. A worker restart is a rollout that a cluster operator does. It is not an action in the tenant console, so the screen has no Restart button.

​
Routed upstreams

Below the controls, the Routed upstreams table lists each model in the static route table with its endpoint. For each endpoint it shows whether the address answers, how many turns it answered, how many it did not, and the time of its last answer. The verdict comes from real turns. It is not a probe. An endpoint that nobody has called reads untried. An endpoint that has never answered shows never in the last-answer column. If the engine reports routing but not upstream health, every endpoint reads untried. Routing still works in that case. The table lists only the static route table. A model that k8s discovery serves does not appear in it.