How Voice Agent Starter works

The Next.js client is intentionally small. It negotiates a single peer connection with the SFU, attaches the microphone as the outbound track, listens for the inbound TTS track, and renders a UI that exposes the agent’s current state. There is no audio worklet, no JS audio buffer manipulation; the browser handles audio playback through the WebRTC pipeline directly. The mic track has Opus configured at 48 kHz mono with discontinuous transmission disabled (we want continuous audio so VAD on the server is straightforward).

The UI surfaces the four states from the turn manager: listening, deciding, speaking, interrupted. This is not just for users , during integration testing, watching the state badge tick through the cycle catches more bugs than any console log.

The SFU is one mediasoup worker running inside its own process. It accepts the WebRTC peer connection from the browser and creates two plain RTP transports to the model worker, one for inbound audio (mic to model) and one for outbound (TTS to speakers). The plain transports are local to the host; in production we colocate the SFU and the model worker to avoid an extra network hop.

mediasoup configuration is in sfu/config.ts. The two parameters most worth tuning are the worker count (one per CPU on the SFU host) and the RTP listen IP, which must be reachable from the model worker. The wiki has the Fly.io recipe.

The brain. Four states, four transitions:

• LISTENING: VAD is active, STT is consuming audio, partial transcripts are accumulating. Transition to DECIDING when a confident-final transcript arrives.
• DECIDING: LLM adapter is generating. As soon as tokens start arriving, transition to SPEAKING.
• SPEAKING: TTS is producing audio chunks; the SFU is forwarding them. Transition to LISTENING when TTS completes naturally.
• INTERRUPTED: VAD detected new mic onset during SPEAKING. Cancel TTS, drain SFU buffer, reset LLM context with an “[user interrupted]” marker, transition to LISTENING.

Each transition emits a log line and an OpenTelemetry span. The state machine is the same across every adapter pairing.

Each adapter is a TypeScript module exporting one async iterator factory. STT takes audio chunks in, yields transcript events. LLM takes a chat history in, yields token strings. TTS takes a text iterable in, yields audio chunks (Opus packets, 20ms frames).

The defaults: Deepgram for STT (excellent streaming partials, low first-token latency), OpenAI Realtime for LLM (the streaming hot path is well-tuned), Cartesia for TTS (sub-200ms first-audio, natural prosody). All three are network calls. Replacing any one is a single file edit.

An adapter authoring guide in the wiki covers the gotchas: partial-transcript handling, confidence thresholds, end-of-turn detection, and how to handle a TTS provider that emits PCM not Opus (transcode in the adapter, do not break the contract).

The hardest part. When VAD detects user speech during SPEAKING:

1. The TTS adapter’s in-flight stream is cancelled via its AbortController. Most providers honour this immediately and stop generating new audio.
2. The SFU producer for the TTS track is told to flush. mediasoup exposes producer.pause(); we then immediately resume to ensure no stale frames are sent.
3. The LLM adapter context is appended with a marker: the agent’s reply was interrupted, the next turn should not assume the prior reply was heard.
4. State transitions to LISTENING; STT begins consuming the new turn.

Each step is tested in isolation and integrated with timing assertions. The acceptance criterion: the user must hear no more than 80 milliseconds of TTS after their interrupting word begins.

Local development is pnpm dev:sfu + pnpm dev:worker + pnpm dev:web with a shared .env. Docker compose is available for one-command boot. Production deployment is documented for Fly.io with two machines: one for the SFU (UDP ports exposed), one for the model worker. The Next.js client deploys anywhere Next.js deploys.