Bilingual Voice Agent on LiveKit

The client, a datacenter and telecom infrastructure company, needed a bilingual voice kiosk for its facility inauguration: a reception-desk agent that could answer questions in English or Urdu using one internal reference document. The engineering reality was CPU‑only inference. No GPU was available for STT, LLM, or TTS, and no single TTS engine handled both English and Urdu well, which meant two separate synthesis pipelines had to run inside one real-time conversation loop. Every latency problem in the system traced back to CPU cycles being scarce, not to model quality. That ruled out the usual fix for a slow-feeling voice agent, which is to throw more parallel compute at synthesis.

The first implementation waited for the complete LLM response before handing it to TTS. Langfuse traces showed tts_first_frame latencies of 7.47 and 12.19 seconds across two conversation turns, meaning the kiosk sat silent for over ten seconds in the worst case before saying anything back.

Turn detection carried a separate risk. With min_endpointing_delay tuned to 0.1 seconds, a user's natural mid-sentence pause could register as the end of their turn, and the kiosk would start responding before they had finished speaking. A false positive on turn detection would interrupt a user mid-sentence in front of live event guests, with no graceful way to recover. That made real speech-boundary detection, not a shorter timeout, the hard requirement.

The pipeline was originally instrumented as a single end-to-end latency number, which told us users were waiting but not why. I restructured observability around stage-level spans, so VAD, STT, retrieval, generation, and TTS each produced their own timing data in Langfuse rather than being folded into one aggregate figure. That's what made tts_first_frame visible as a distinct metric in the first place and what turned "the kiosk feels slow" into "synthesis isn't starting until the full response is generated." The instrumentation was then separated from the core pipeline logic so it didn't stay scattered across the agent implementation, keeping the pipeline readable while preserving that visibility. An end-to-end number tells you a user is waiting. Stage-level spans tell you where the wait is coming from.

tts_first_frame latency measured 7.47 and 12.19 seconds in the initial full-buffer implementation, captured through Langfuse tracing across two conversation turns. Switching to sentence-level streaming synthesis, a dedicated executor for TTS, and a reduced frame size (200ms to 20ms) removed the full-response wait that caused those delays. No confirmed post-fix tts_first_frame measurement exists; an architectural estimate put the budget at 720 to 1120ms, but that figure was never validated against a live measurement and isn't presented here as an achieved result.

The kiosk was deployed and operated as the reception agent at the client's facility inauguration, handling English and Urdu conversations grounded in the client's reference document. No data exists on incidents during the event or usage beyond it.

The system was built entirely around the absence of a GPU: sentence-level streaming was the ceiling CPU-bound synthesis allowed, and the dedicated executor existed to keep that synthesis from starving the rest of the event loop. With dedicated GPU inference, sample-level TTS streaming becomes possible instead of sentence-level, and the roughly 600 to 1000ms round trip that remote Whisper STT added, the single largest latency contributor in the pipeline, mostly disappears. Most of the tuning work in this project, frame size, executor isolation, buffering strategy, existed to work around scarce CPU cycles rather than to solve a modeling problem. Given the choice again, I would push for dedicated inference endpoints per stage before accepting a shared CPU host as the deployment target, rather than optimizing around that constraint after the fact.