How to Build Your Own Voice AI Platform on LiveKit (at a Fraction of Retell's Cost)
- A self-hosted cascading stack on LiveKit runs cheaper per minute than a managed voice platform. You stop paying an orchestration margin on every second of audio.
- TTS and transport dominate the cost, not the LLM. Optimising the model first is the most common and most expensive mistake.
- Below roughly 20,000 minutes a month, a managed platform is still the correct commercial decision. Build when volume, control, or provider-swap speed justify owning the pipeline.
What does it actually take to run your own voice AI platform?
Four moving parts and one hard constraint. The parts: transport (getting audio in and out with low jitter), speech-to-text, a language model, and text-to-speech. The constraint: the whole round trip has to finish fast enough that a human does not notice the gap. Everything else in this article is downstream of those five facts.
LiveKit handles the hard part: WebRTC transport, room state, and telephony ingress. It deliberately does not decide which STT, LLM, or TTS you use. That separation is the entire opportunity. A managed platform bundles all four and charges a blended per-minute rate. Run the orchestration yourself and you pay each provider directly at list price.
I built this stack in production for an AI interview platform serving 20+ enterprise clients, after first shipping on a managed vendor. Both decisions were right at the time, and the second is why this article has real numbers in it.
- 1Caller speaks0 ms
Audio enters through SIP (phone) or WebRTC (browser) and lands in a LiveKit room. Your agent worker is already joined to that room as a participant.
- 2Streaming transcriptionpartial results
Audio frames stream to the STT provider continuously. You are not waiting for the caller to stop talking. You accumulate partial transcripts and watch for an endpoint.
- 3Turn detection firesthe hard part
A silence threshold alone produces agents that interrupt people mid-sentence. Semantic endpointing decides whether the caller has finished a thought, not merely paused for breath.
- 4LLM generates, streamingfirst token matters
You send the transcript plus conversation state and stream tokens back. Time-to-first-token is the metric that matters here, not total completion time.
- 5TTS streams backsentence by sentence
Synthesis begins on the first clause, not the finished paragraph. Audio is published back into the LiveKit room while the model is still writing the rest of the sentence.
- 6Barge-in handlingcancel everything
If the caller starts speaking, you must stop playback, cancel the in-flight TTS request, and discard the pending generation. Getting this wrong is the single most common reason a demo feels robotic.
Where does the latency budget actually go?
Natural conversational turn-taking sits at roughly 200 to 300 milliseconds between human speakers. Voice agents cannot hit that yet, and they do not have to. But cross about 800 milliseconds and callers start talking over the agent, and past a second the interaction feels distinctly like a machine.
Treat 800ms as a budget and give every component a line item. Once the budget is written down, the optimisation targets stop being a matter of opinion.
What does a voice agent really cost per minute?
Managed platforms publish blended per-minute rates. Retell's own cost calculator splits its pricing into voice infrastructure, LLM, and TTS components, which is unusually transparent and makes it the fairest baseline to compare against. LiveKit publishes its own calculator that defaults to a per-minute figure for a comparable cascading setup.
Both of those are worth checking directly rather than trusting any blog post, including this one. Prices in this category move quarterly, and a stale number is worse than no number.
The structural point survives the price changes: a self-hosted cascading stack removes the orchestration margin and lets you choose each component on its own merits. The savings are real but not free. You trade vendor margin for engineering time.
When should you not build this?
Most teams asking this question should stay on a managed platform, and it costs me consulting revenue to say so. The build only pays back when at least one of three things is true: your volume is high enough that per-minute margin dominates, you need control that the platform will not give you, or you need to swap providers faster than a vendor will adopt them.
The volume threshold sits somewhere around 20,000 minutes a month for most teams. Below that, the engineering time to build, monitor and maintain the pipeline costs more than the margin you save. Above it, the arithmetic flips quickly.
- Live in days, not weeks
- One vendor, one bill, one support channel
- Blended per-minute rate includes orchestration
- Provider choice constrained to their roster
- Run both stacks behind one routing layer
- Shift traffic percentage by percentage
- Compare quality on identical live calls
- The safest path if you already have customers
- Pay each provider at list price directly
- Swap STT, LLM or TTS the week a better one ships
- Own turn detection and interruption logic
- You now own reliability, monitoring, and on-call
What breaks in production that never breaks in a demo?
Demos are single calls, on good networks, with cooperative speakers. Production is none of those things. The failure modes below are the ones that consume real engineering weeks, in roughly the order teams encounter them.
| Failure mode | What the caller experiences | Where to fix it |
|---|---|---|
| Naive silence-based endpointing | Agent interrupts mid-sentence | Semantic turn detection, not a fixed silence timer |
| Barge-in not cancelling TTS | Agent talks over the caller, both unintelligible | Cancel playback and in-flight synthesis on speech start |
| Provider concurrency ceiling | Calls fail at peak, not under test | Know each vendor's concurrent-session cap before launch |
| Tail latency on LLM | Long silences, caller says 'hello?' | Alert on p95 per stage; route to a faster model on timeout |
| No conversation-level tracing | Cannot reproduce a complaint | Log the full per-turn waterfall with a call ID from day one |
| Cold-start on agent workers | First call after idle is slow | Keep a warm worker pool; pre-connect provider sockets |
What would I build first?
Do not start with the architecture. Start with one call path, end to end, on the cheapest components that work, and instrument every stage before you tune any of them. You cannot optimise a pipeline you cannot measure, and the stage you assume is slow is usually not the one that is.
Then make the pipeline swappable. The most valuable property of a self-hosted stack is not today's cost. It is that when a better or cheaper provider ships next quarter, you can adopt it in an afternoon instead of waiting for a platform to support it.
If you want this built and instrumented properly, that is exactly what voice AI development is. And if you are only trying to size the decision, the voice AI cost calculator will get you a defensible number in about a minute.
Frequently asked questions
→What do you need to build your own voice AI platform?
Four components and one constraint: transport (LiveKit handles WebRTC and telephony ingress), streaming speech-to-text, a language model, and streaming text-to-speech. The constraint is that the full round trip must finish fast enough that a human does not notice the gap.
→How fast does a voice agent need to respond?
Natural human turn-taking sits around 200-300ms. Voice agents cannot hit that yet, but crossing roughly 800ms is where callers begin talking over the agent, and past a second the interaction feels distinctly mechanical.
→Which component costs the most?
Text-to-speech, followed by transport. In LiveKit's own calculator default, TTS is roughly $0.030 of a $0.0672 minute while the language model is about $0.0014, the smallest line item by a wide margin.
→What breaks in production that never breaks in a demo?
Naive silence-based endpointing that interrupts people mid-sentence, barge-in that fails to cancel in-flight synthesis, provider concurrency ceilings that only appear at peak, and tail latency on the language model.
→Should I build this at all?
Below roughly 20,000 minutes a month, no. A managed platform is the correct commercial decision because engineering and on-call cost more than the margin saved. Above it, owning the pipeline typically repays the build within a quarter.