# Browser Voice Agent

> Click-to-talk in the browser: capture the mic as encoded Opus and play the agent's reply, all over QUIC. There is no WebRTC transport and no SFU.

Build a click-to-talk button that connects a browser user to a server-side AI
voice agent. The SDK captures the user's microphone, encodes it as Opus, and
publishes it as the call's **uplink**. The agent's synthesized speech arrives
as the **downlink** and plays back through WebCodecs. Media travels as MoQT
over QUIC / WebTransport end-to-end. The code uses the browser's WebRTC stack
only as a **codec + capture tap**, never as the transport. There is no SFU in
the path.

Three helpers make this a few lines of code instead of a WebCodecs project:

- **`captureMicrophone`** — runs the browser's AEC / AGC / noise-suppression
  and Opus encoder in a loopback `RTCPeerConnection`. It then diverts the
  *encoded* frames onto your uplink track. Raw PCM never crosses the wire.
- **`OpusPlayer`** — decodes received Opus with WebCodecs and renders it
  through an `AudioWorklet` ring buffer.
- **`attachCaller`** — the browser-caller side of the audio bridge. It
  subscribes to the downlink (`voice/<sid>/downlink`, the audio that the cloud
  sends the caller). It publishes the uplink (`voice/<sid>/uplink`, the
  caller's mic).

> **WARNING:**
> The browser media plane is **WebTransport-only today**. There is no WebSocket
> fallback rung in the browser SDK yet. The QUIC→WebSocket ladder ships only in
> the native cores. `captureMicrophone` also *hard-refuses* to run if
> `RTCRtpScriptTransform` is missing. It does not fall back to an insecure
> main-thread path. In practice you need a current Chromium (also required for
> the WebCodecs `AudioDecoder` playback), Edge, Firefox 133+, or Safari TP. See
> [Browser compatibility](/modalities/voice/transport-web/browser-compatibility).

## How the pieces connect

  1. **Mint the call server-side**
Your backend originates the call with an `agent` attached. It hands the
    browser back the `sid` plus a browser-scoped token. Your secret API key
    never ships to the client.
  2. **Start playback**
Create and `start()` an `OpusPlayer`. This must happen inside a user
    gesture so that the `AudioContext` can resume.
  3. **Attach as the caller**
`attachCaller(sid, { codec: "opus", onDownlink })` subscribes to the
    agent's audio and pushes each frame into the player.
  4. **Capture the mic**
`captureMicrophone(...)` forwards encoded Opus frames onto the uplink. The
    agent then hears the user.

## Front end: the click-to-talk component

This is the whole browser side. It assumes a page with a dial button. It holds
a **browser-scoped** key: a publishable, `voice`-scoped, short-TTL token, *not*
your server secret (see [the backend](#minimal-backend) below).

The browser mic capture is inherently a Web API, so this half is TypeScript
only. The [backend](#minimal-backend) shows both TypeScript and Python.

```ts
import { Voice } from "clutchcall_connect/voice";
import { captureMicrophone, OpusPlayer } from "clutchcall_connect/moqt";

// A publishable, voice-scoped token your backend hands the page — never the
// server API key. See the backend section.
async function mintSession(): Promise<{ sid: string; token: string }> {
  const res = await fetch("/api/voice/session", { method: "POST" });
  if (!res.ok) throw new Error(`session mint failed: ${res.status}`);
  return res.json();
}

async function startAgentCall() {
  const { sid, token } = await mintSession();

  const v = new Voice({
    baseUrl:   "https://engine.firstbyte.fast",
    apiKey:    token,            // browser-scoped, short-lived
    orgId:     "org_abc",
    relayHost: "relay.firstbyte.fast",
  });

  // 1. Playback graph. Opus decodes at 48 kHz mono. Create + start() inside the
  //    click handler so the AudioContext is allowed to produce sound.
  const ctx    = new AudioContext();
  const player = new OpusPlayer(ctx, { sampleRate: 48000, channels: 1 });
  await player.start();

  // 2. Attach as the caller: subscribe the downlink (agent → user), publish the
  //    uplink (user mic → agent). onDownlink is (frame, tsUs); player.push is
  //    (tsUs, frame) — note the argument order flips.
  const bridge = await v.audioBridge.attachCaller(sid, {
    codec:      "opus",
    onDownlink: (frame, tsUs) => player.push(tsUs, frame),
  });

  // 3. Capture the mic. captureMicrophone writes encoded frames to an
  //    AudioPublication; adapt the bridge's publishUplink to that shape.
  const mic = await captureMicrophone(
    { write: (tsUs, frame) => bridge.publishUplink(frame, tsUs) } as any,
    {
      audioConstraints: {
        echoCancellation: true,   // the browser's AEC — the engine has none
        autoGainControl:  true,
        noiseSuppression: true,
      },
    },
  );

  // 4. Teardown. Stop capture and playback, then close the bridge (which drops
  //    both tracks and the transport).
  async function hangUp() {
    mic.stop();
    player.close();
    await ctx.close();
    await bridge.close();
  }

  return { sid, hangUp };
}

// Wire it to a button. The first click starts the call; the AudioContext and
// getUserMedia both need this user gesture.
const btn = document.querySelector<HTMLButtonElement>("#talk")!;
let session: { sid: string; hangUp: () => Promise<void> } | null = null;

btn.addEventListener("click", async () => {
  if (session) {
    await session.hangUp();
    session = null;
    btn.textContent = "Talk to the agent";
    return;
  }
  btn.disabled = true;
  session = await startAgentCall();
  btn.disabled = false;
  btn.textContent = "Hang up";
});
```

> **NOTE:**
> `captureMicrophone` runs the browser's echo-cancellation, gain control, and
> noise suppression. It then diverts the **encoded** Opus frames onto the uplink.
> Raw PCM never leaves the tab. The engine has no server-side AEC/AGC. Those
> constraints must do their job in the browser, or the agent hears its own
> voice. Details are in
> [WebRTC diversion](/modalities/voice/transport-web/webrtc-diversion) and
> [RTCRtpScriptTransform](/modalities/voice/transport-web/rtcrtpscripttransform).

## Minimal backend

The browser must not hold your secret API key. A small server route mints the
call and attaches the AI agent that answers. The route returns the `sid` plus a
publishable token scoped to this one call. The route creates the call over a
`voice`-routing trunk that you provision for web sessions. `to` / `from` label
the leg. For a pure-web agent, they do not need to be dialable numbers.

  
```ts
// POST /api/voice/session  → { sid, token }
import { Voice } from "clutchcall_connect/voice";

const v = new Voice({
  baseUrl: "https://engine.firstbyte.fast",
  apiKey:  process.env.CLUTCHCALL_API_KEY!,   // server secret — stays here
  orgId:   "org_abc",
});

export async function handleSession() {
  const call = await v.calls.originate({
    to:      "web:agent",          // label for the web leg
    from:    "web:caller",
    trunkId: "trunk_web",          // your web-routing trunk
    agent:   "healthcare-assistant",
  });

  // Issue a short-lived, voice-scoped token bound to this sid with your own
  // token service, then return it alongside the sid.
  const token = await issueScopedToken({ sid: call.sid, scope: "voice", ttlSec: 300 });
  return { sid: call.sid, token };
}
```
  
  
```python
# POST /api/voice/session  → { "sid": ..., "token": ... }
import os
from clutchcall.voice import Voice

v = Voice(
    base_url="https://engine.firstbyte.fast",
    api_key=os.environ["CLUTCHCALL_API_KEY"],   # server secret — stays here
    org_id="org_abc",
)

def handle_session() -> dict:
    call = v.calls.originate(
        to="web:agent",              # label for the web leg
        from_="web:caller",
        trunk_id="trunk_web",        # your web-routing trunk
        agent="healthcare-assistant",
    )
    # Issue a short-lived, voice-scoped token bound to this sid.
    token = issue_scoped_token(sid=call.sid, scope="voice", ttl_sec=300)
    return {"sid": call.sid, "token": token}
```
  

> **NOTE:**
> `agent` binds a server-side agent (ASR→LLM→TTS or a speech-to-speech provider).
> The agent answers the call and drives both legs. You do not open an
> `AudioBridge` on the server. To run your own dialog logic instead, drop `agent`
> and bridge the audio yourself. See
> [Bring your own ASR / LLM / TTS](/modalities/voice/runtime/byo-asr-llm-tts).

## Local development

WebTransport with a browser needs a secure context and a valid certificate. For
local work:

- Serve the page over HTTPS. `getUserMedia`, WebTransport, and WebCodecs all
  require a secure context.
- Use a short-lived **ECDSA** dev certificate (≤14 days). Pass its SHA-256 to
  the transport via `serverCertificateHashes`. The QUIC handshake rejects RSA
  certs.
- Dial `127.0.0.1`, **not** `::1`. The IPv6 loopback breaks the WebTransport
  handshake.

See [WebTransport](/modalities/voice/transport-web/webtransport) and
[Browser audio troubleshooting](/modalities/voice/troubleshooting/browser-audio)
for the full setup and common failures (no downlink audio, mic refused,
one-way audio).

> **NOTE:**
> Native **mobile** (Swift / Kotlin) apps are on the roadmap, not shipped. There
> is no native mobile SDK today. A browser tab on the device is the supported
> path. Track the plan in
> [Mobile apps](/modalities/voice/transport-web/mobile-app).

## Related

  - **[Browser audio capture](/modalities/voice/transport-web/browser-audio-capture)** — The capture pipeline in depth: loopback PC, the worker transform, framing.
  - **[JavaScript SDK](/modalities/voice/sdk/javascript)** — Full signatures for `Voice`, `attachCaller`, `captureMicrophone`, `OpusPlayer`.
  - **[Turn detection](/modalities/voice/concepts/turn-detection)** — Tune VAD, barge-in, and idle timeouts for the agent leg.
  - **[Support handoff](/modalities/voice/recipes/support-handoff)** — Escalate the browser call from the AI agent to a human. The sid stays the
    same.