# Relay network and edge cache

> How stream PoPs register, which viewer regions route to each PoP, how liveness and the edge cache-hit ratio are computed, what PoP statuses mean for viewers, and what to do when telemetry is unavailable.

The **Relay network** screen is the operable view of the FirstByte MoQ
relay mesh. Where the architecture pages describe the mesh as a design,
this screen shows it as a list of points of presence (PoPs) with live
numbers attached: how many viewers each edge is serving, and how much of
that traffic it served from its own cache instead of pulling from origin.

Everything on the screen comes from one procedure:

```ts
trpc.streams.analytics.popHealth.useQuery({ orgId })
```

It joins four sources:

1. The **`stream_pop` catalog** in Postgres — the registry of PoPs: code,
   city, coordinates, role (`origin` or `edge`) and the PoP's public relay
   URL.
2. The **streams viewer mesh** in Redis — the hash the geo-router reads to
   send a viewer to a relay (region → relay URL). The screen matches each
   PoP's relay URL against it to show which regions route to that PoP.
3. The **`relay_heartbeat`** table in ClickHouse — one row per relay shard
   per second, which is what tells the screen a PoP is alive right now and
   how many QUIC sessions and MoQ tracks it holds.
4. The **`stream_pop_cache_1m_q` rollup** in ClickHouse, which supplies the
   last hour of cache-hit and viewer numbers.

The screen re-runs the query every 30 seconds, and the **Refresh** button
runs it on demand.

## How a PoP is registered

PoPs are not created from the console. There is no "add PoP" action on
this screen. A row appears in the `stream_pop` catalog when the platform
operator registers a relay deployment with
`server/scripts/register-stream-pops.ts` from the fleet inventory; the
console only reads that catalog. Each row carries a `role`: exactly one
PoP is the `origin` (publishers ingest there and every edge chains to it)
and the rest are `edge` PoPs (leaves that pull from the origin and serve
viewers from their cache).

That is why the empty state reads:

> No PoPs registered yet — PoPs appear when the relay deploy registers them.

An empty mesh is therefore a deployment fact, not a telemetry failure. If
you expect PoPs and see none, the question to ask is whether the relay
deploy in that environment came up and registered, not whether the
analytics pipeline is healthy.

The catalog also supplies each PoP's coordinates (`geo_lat`, `geo_lon`),
which is what places the pins on the **Edge relay map** — an
equirectangular world map, so a PoP registered with real coordinates lands
where it is. The larger pin is the origin; the dashed lines are the
fan-out chains from the origin to each edge. Pin position is catalog data;
the colour of a pin is the PoP's liveness (see
[PoP statuses](#pop-statuses-and-what-they-mean-for-viewers)).

## Fields on each PoP row

| Field | Source | Meaning |
| ----- | ------ | ------- |
| `code` | `stream_pop` (PG) | The PoP identifier, shown in mono in every panel. |
| `city`, `country_code` | `stream_pop` (PG) | Human-readable location, shown next to the code. |
| `geo_lat`, `geo_lon` | `stream_pop` (PG) | Pin position on the relay map. |
| `role` | `stream_pop` (PG) | `origin` or `edge`. |
| `relay_url` | `stream_pop` (PG) | The PoP's public MoQ base URL; the **Relay** column shows its host. |
| `status` | `stream_pop` (PG) | `active`, `maintenance`, or any other value (treated as bad). |
| `regions` | Redis viewer mesh | The viewer regions whose relay URL is this PoP's — the regions the geo-router sends here. Empty for an edge means no region routes to it; the origin shows `ingest`. |
| `heartbeat` | `relay_heartbeat` (CH) | The PoP's latest heartbeat in the last 5 minutes: age, open QUIC `sessions`, owned MoQ `tracks`. `null` when the PoP has not reported. |
| `hit_ratio` | `stream_pop_cache_1m_q` (CH) | Edge cache-hit percentage over the last hour. |
| `viewers` | `stream_pop_cache_1m_q` (CH) | Peak viewers attributed to that PoP in the last 5 minutes. |

The response also carries a top-level `telemetry_ok` flag. See
[When telemetry is unavailable](#when-telemetry-is-unavailable).

## What the cache-hit ratio counts

`hit_ratio` is a per-PoP number computed by the ClickHouse rollup and read
verbatim by the console — the screen does not recompute it. It is the
edge-cache hit share for that PoP, labelled **origin offload** on the KPI
card: the higher the number, the more of the fan-out that relay absorbed
locally instead of forwarding upstream to origin.

The window is the **last hour**, assembled from the one-minute buckets in
`stream_pop_cache_1m_q`. It is not an instantaneous reading. A PoP that
recovered two minutes ago still carries the previous fifty-eight minutes
of misses in its bar.

The **Edge cache hit** KPI at the top of the screen is the **plain
arithmetic mean of `hit_ratio` across every PoP that served any traffic
in the window**. It is not weighted by viewers, and it does not exclude
PoPs in maintenance; PoPs with no traffic at all are left out rather than
counted as `0%`. A small PoP with a bad ratio moves the headline number
exactly as much as your largest one does, so treat the KPI as a mesh-wide
smoke alarm and the per-PoP bars as the actual diagnosis.

The KPI card and the per-PoP bars use different colour cut-points, which
is worth knowing before you reconcile them:

| Surface | Green / accent | Warn | Bad |
| ------- | -------------- | ---- | --- |
| **Edge cache hit** KPI (mesh average) | ≥ 95% | ≥ 85% | below 85% |
| Per-PoP bar in the **Cache hit** column of the PoP table | ≥ 96% | ≥ 90% | below 90% |

So a PoP can sit in warn on its bar while the mesh average it feeds is
still green. That is expected, not a bug.

## Reading the fan-out ratio

The **Origin → edge fan-out** panel shows the chain every viewer's bytes
travel: the origin's relay host on the left, the number of edges and the
summed viewer count on the right, then one row per PoP. The origin
publishes once and each edge multiplies it — that is the number that
justifies the relay mesh, and the fan-out for one edge is simply its
viewer count in the **Points of presence** table.

Each row in the panel carries the PoP's liveness pip and the **viewer
regions** routed to it. Those regions come from the streams viewer mesh in
Redis (region → relay URL), matched against the PoP's `relay_url`; the
origin shows `ingest` because publishers, not the geo-router, send traffic
there. An edge with no region is registered and may be live, but the
geo-router is not sending it any viewers — a note under the list names
such edges. Repointing a region is an ops change to the mesh, not
something this screen does.

Two caveats when you quote per-PoP viewer counts:

- They are **last-5-minute peaks** from the rollup, so they lag the live
  session counts in the heartbeat by up to a minute.
- They are attributed by the `pop_code` the relay stamps on its samples. A
  relay started without `--set relay_pop=<code>` reports the default
  `edge`, and its samples do not join to any catalog row.

## PoP statuses and what they mean for viewers

Two things combine into the pin colour on the map, the pip in the fan-out
list and the **Liveness** cell: the catalog `status`, and whether the PoP
has sent a relay heartbeat recently. A relay emits one heartbeat per shard
per second; the screen treats anything older than 60 seconds as stale.

| Liveness | Indicator | What it means for delivery |
| -------- | --------- | -------------------------- |
| live | Green, counted under **Live** | `status` is `active` and a heartbeat arrived within 60 s. The PoP is in rotation and the geo-router can send viewers to it. |
| not reporting | Grey, counted under **Not reporting** | `status` is `active` but the PoP has never sent a heartbeat that reached the portal. This is a deployment fact — the relay has no telemetry path — not an outage, so it is not drawn in alarm colour. |
| maintenance | Amber, counted under **Maintenance** | `status` is `maintenance`. The PoP is deliberately out of normal service; it still appears in the catalog and still contributes its rollup numbers. |
| stale | Red, counted under **Stale / errored** | `status` is `active`, the PoP used to heartbeat and has not in the last 60 s. The relay or its telemetry sink is down. |
| anything else | Red, counted under **Stale / errored** | An unrecognised `status`. Treat it as a PoP to investigate — it is in the catalog but is neither healthy nor deliberately drained. |

The **Reporting** KPI is the number of live PoPs out of the total. When it
is short, read the Liveness cell of each PoP to tell a stale one (something
broke) from one that has never reported (nothing to fix on the relay). A
PoP in `decommissioned` status is not returned at all.

## Live relay counters versus the rollup

The screen deliberately stacks two sources of truth, in this order:

1. **Relay health** (the `RelayHealth` panel, scoped to the `stream`
   namespace) — a direct scrape of the relay's own counters, read now.
2. **The PoP KPIs and bars** — ClickHouse rollups, which lag behind by
   about a minute because they are assembled from one-minute buckets.

The live scrape sits above the rollup KPIs for a reason: **when the two
disagree, the live panel is the current truth.** During an incident, the
rollup is the shape of the last hour and the live panel is the shape of
right now. Use the live panel to decide whether something is still
happening, and the rollup to decide how long it has been happening and
which regions it touched.

## Delivery counters

Under the relay health numbers, a second row counts the ways the relay
loses or delays media. Before these counters, a viewer could miss objects
and no other number on the screen changed. The row is **relay-wide**: it
counts every namespace, not only your streams.

| KPI | What it counts | What to do when it is not zero |
| --- | --- | --- |
| **Resets/sec** | Outgoing subgroup streams that the relay reset, per second. The hint names the most frequent reason in the last interval: `delivery timeout`, `evicted` (the group left the cache while it was still being sent), `cancelled` (the viewer stopped it), `upstream` or `internal`. | A short burst when viewers leave is normal (`cancelled`). A steady rate of `delivery timeout` or `evicted` means viewers receive media later than the relay can keep it. |
| **Refused opens/sec** | Times the relay could not open a new stream for a viewer because the viewer's stream credit was used up. The media waits in a queue. | A short burst when a viewer joins is normal. A steady rate means viewers fall behind; look for slow viewer networks or a very high group rate. |
| **Too far behind** | Subscriptions that the relay ended because the viewer fell so far behind that the media it needed was gone. Lifetime count. | Each one is a viewer whose player stopped. Compare with Refused opens. |
| **Dropped** | Media that left no reset: queued groups that never got a stream, and incoming streams that arrived before the relay knew their subscription. Lifetime count. | Report it with the time it started. |

The same counters are on the engine's Prometheus endpoint as
`moqt_pub_subgroup_reset_total{reason}`, `moqt_pub_open_refused_total`,
`moqt_pub_publish_done_total{status}`, `moqt_pub_pending_dropped_total`,
`moqt_sub_stream_dropped_total{reason}` and the
`moqt_object_ack_latency_microseconds` histogram.

## Live tracks on the relay

The **Live tracks on the relay** panel lists every track of your live
inputs that the relay holds now, refreshed every 5 seconds. It reads
`trpc.observability.relayTracks.useQuery({ orgId })`, which returns only
tracks under `stream/<your org id>/`.

| Column | Meaning |
| --- | --- |
| **Input** | The live input the track belongs to. |
| **Track** | The track name, for example `video_src`, `audio` or `.catalog`. |
| **Kind** | `ingest`: the track your encoder publishes into the engine. `cross-shard`: a copy on another engine shard that serves viewers who landed there. `relay`: a track the relay pulled from another relay. |
| **Viewers** | Subscribers attached to the track now. |
| **Objects** | Objects the track has received since it started. |
| **Last object** | How long ago the last object arrived. It turns amber when the track has viewers and received nothing for more than 5 seconds: the state a viewer sees as a frozen player. |

A query that matches more tracks than its limit returns an error, not a
shortened list, so the panel never shows part of your tracks as if it
were all of them.

## When telemetry is unavailable

The `popHealth` response carries `telemetry_ok`. When the ClickHouse side
of the join is unavailable, the catalog still returns PoPs but the
cache-hit numbers cannot be trusted. The screen degrades rather than
showing zeros:

- The **Edge cache hit**, **Reporting**, **Live sessions** and **Viewers**
  KPIs render `—`, drop their severity colour entirely, and the cache-hit
  hint reads `telemetry unavailable`.
- Every per-PoP Liveness, Sessions, Tracks, Viewers and cache-hit value in
  the PoP table renders `—`. The heartbeat lives in ClickHouse too, so a
  PoP cannot be called live or stale while ClickHouse is unreachable.
- The **Telemetry health banner** at the top of the page explains the
  degraded state.

This is a display rule with a specific intent: a red `0.0%` would claim
that every edge is missing cache, which is a different and far more
alarming statement than "we cannot currently measure it". Alarm colour on
this screen is reserved for conditions that are genuinely wrong.

The same suppression applies when the mesh is simply **empty**. With zero
PoPs, the KPI shows `—` with the hint `no traffic yet` rather than a red
`0.0%`. No PoPs means no measurement, not a total cache miss.

Note that the per-PoP **bars** still draw at their last-read widths while
`telemetry_ok` is false; only the numeric labels blank out. Read the `—`
values, not the bar lengths, when the banner is up.

The **PoPs** KPI, the map pins and the viewer regions are **not**
suppressed by `telemetry_ok`: they come from the Postgres catalog and the
Redis mesh, which are separate from ClickHouse. Pins fall back to grey
(not reporting) while the heartbeat cannot be read.

## Diagnosing one unhealthy region

When a single PoP's bar is amber or red while the rest of the mesh is
green, work through the panels in this order:

1. **Check the status pip first.** If the PoP is in `maintenance`, its
   cache-hit number is expected to sag and needs no action — it is being
   drained on purpose, and it is still dragging the unweighted mesh
   average down. If the pip is red with no legend entry, the catalog state
   itself is the problem.
2. **Check the telemetry banner.** A degraded rollup makes every number on
   the page provisional. Resolve that before you chase a region.
3. **Compare against the live relay health panel.** The bar covers the last
   hour. If the live counters look normal, you are most likely looking at
   the tail of an incident that has already ended, and the bar will recover
   as the older one-minute buckets age out of the window.
4. **Read that PoP's viewer count in the fan-out panel.** A low hit ratio
   on a PoP with very few viewers has little effect on origin load, however
   red the bar looks; a low ratio on the PoP carrying most of the audience
   is where origin egress is actually going.
5. **Re-check the mesh average.** If exactly one PoP moved and the **Edge
   cache hit** KPI changed noticeably, that is the unweighted mean at work
   — a reminder that the KPI is a signal to open the per-PoP panel, not a
   measure of how much origin traffic you are actually paying for.

## Related

- [Telemetry](/platform/telemetry) — the metric, trace, and rollup streams behind these numbers
- [Architecture](/concepts/architecture) — where the relay mesh sits in the data plane
- [Authentication](/concepts/authentication) — relay tokens and the `namespace_auth` hook that gates edge subscribe