# Telemetry

> Metrics, traces, CDRs, and SIP packet capture.

The FirstByte gateway emits four streams of operational data. Each
stream targets a different consumer. You can opt out of any stream via
the gateway's deployment config.

| Stream                   | Protocol           | Consumer                         |
| ------------------------ | ------------------ | -------------------------------- |
| Prometheus metrics       | OpenMetrics scrape | VictoriaMetrics / Prometheus / Grafana / Mimir |
| Distributed traces       | OTLP / HTTP `:4318` | ClickHouse via the OpenTelemetry collector, Tempo, Honeycomb, Datadog |
| Call Detail Records      | HTTP JSONEachRow   | ClickHouse                       |
| SIP packet capture       | HEPv3 UDP          | Homer / heplify-server           |

## Metrics

A Prometheus-style endpoint exposes a fixed set of metric families.
The names are stable. These metrics are relevant for capacity planning:

| Metric                                | Labels                                        | Type      |
| ------------------------------------- | --------------------------------------------- | --------- |
| `clutchcall_calls_total`       | `tenant`, `trunk`, `direction`, `result`      | Counter   |
| `clutchcall_calls_active`      | `tenant`, `trunk`                             | Gauge     |
| `clutchcall_call_duration_seconds` | `tenant`, `trunk`, `result`               | Histogram |
| `clutchcall_rtp_packets_total` | `direction`, `codec`                          | Counter   |
| `clutchcall_rtp_loss_ratio`    | `tenant`                                      | Gauge     |
| `clutchcall_jitter_ms`         | `tenant`                                      | Histogram |
| `clutchcall_estimated_mos`     | `tenant`                                      | Histogram |
| `clutchcall_codec_engine_lag_us` | `shard`                                     | Histogram |
| `clutchcall_quic_handshake_seconds` | (none)                                   | Histogram |
| `clutchcall_admin_requests_total` | `method`, `result`                         | Counter   |

Scrape from `http://<gateway-host>:9091/metrics`.

## Tracing

Every RPC produces an OpenTelemetry trace with these standard fields:

- `service.name` = `clutchcall-gateway`
- `tenant.id`
- `call.sid` (when applicable)
- `method.id` (the RPC's `method_id` decimal)
- `transport` = `quic` | `webtransport` | `webrtc` | `sip`

Spans cover RPC parse, JWT validation, trunk lookup, dialplan node
execution, and SIP/RTP setup. Correlate with metrics via the `tenant`
label.

OTLP endpoint defaults: `http://localhost:4318` (HTTP) /
`http://localhost:4317` (gRPC). Override via the gateway's
`OTEL_EXPORTER_OTLP_ENDPOINT` env var.

On the platform, the OpenTelemetry gateway collector writes every span
into the ClickHouse table `otel.otel_traces`. The Monitoring screen reads
that table. It scopes spans to a tenant by the `tenant.id` attribute on
the span or on the resource.

## Call Detail Records

When a call ends, the gateway pushes a CDR row to ClickHouse via the
HTTP `JSONEachRow` interface. The schema is below. The `telephony_cdrs`
table is partitioned by month and indexed on `(tenant_id, timestamp_ms)`:

| Column              | ClickHouse type | Source                                        |
| ------------------- | --------------- | --------------------------------------------- |
| `call_sid`          | `String`        |                                               |
| `tenant_id`         | `LowCardinality(String)` |                                       |
| `trunk_id`          | `LowCardinality(String)` |                                       |
| `direction`         | `Enum8`         | `outbound` / `inbound`.                       |
| `from_number`       | `String`        |                                               |
| `to_number`         | `String`        |                                               |
| `start_timestamp_ms`  | `DateTime64(3)` |                                             |
| `answer_timestamp_ms` | `Nullable(DateTime64(3))` |                                   |
| `end_timestamp_ms`  | `DateTime64(3)` |                                               |
| `duration_seconds`  | `UInt32`        |                                               |
| `q850_cause`        | `UInt16`        | ITU-T Q.850 cause code.                       |
| `status`            | `LowCardinality(String)` |                                       |
| `codec`             | `LowCardinality(String)` |                                       |
| `packets_sent`      | `UInt32`        |                                               |
| `packets_received`  | `UInt32`        |                                               |
| `packets_lost`      | `UInt32`        |                                               |
| `bytes_sent`        | `UInt64`        |                                               |
| `jitter_ms`         | `Float32`       |                                               |
| `estimated_mos`     | `Float32`       | R-factor → MOS estimate.                      |
| `recording_url`     | `String`        | Empty if the call was not recorded.           |
| `agent_id`          | `LowCardinality(String)` | Set if the call routed through an AI agent DAG. |

These fields mirror the `CallEvent` that you receive on
`CHANNEL_HANGUP_COMPLETE`. The data is the same, but the gateway
persists it server-side.

## SIP capture (HEPv3)

For SIP-side debugging, the gateway can also send HEPv3 frames over UDP
to a Homer / heplify-server instance. Use this when a call fails before
any `CallEvent` reaches the SDK and you must inspect the actual SIP
exchange.

Configure via:

```yaml
# gateway config snippet
heplify:
  enabled: true
  endpoint: 192.0.2.10:9060
  capture_id: 2001
```

Disable this in production unless you actively debug an issue. HEPv3
doubles the SIP-side packet rate.