Which part of the scaling are you thinking about? For inference, most people that are doing 10k calls per day are using first-party, hosted services for STT, LLM, and TTS. You definitely can host your own models, but for two reasons, most people are not. First, the open weights models are still less capable than the commercial models. (Though I think that will change.) And second, it's actually more expensive to host your own models than to use the first-party, hosted, APIs until you are significantly bigger than 10k calls a day. For hosting the voice agent itself, the basic answer is that you need to get various bits and pieces of Kubernetes auto-scaling set up for your specific use case and cloud provider. (You can also look and see if Pipecat Cloud fits your needs. It's "docker push for voice agents".)

It's great to build things to learn! If there are latency issues that I can help you think through/diagnose, very happy to do that. Typical things that contribute to latency are: - using WebSockets instead of WebRTC (for edge-to-cloud, but this is probably not your case since you mention telephony), - long network round trips between all your providers (you want your voice agent very close to where your telephony provider terminates the PSTN, and to all your inference providers) - TTFT variance from your providers (you want to build observability tooling into your agent so you can measure and track this), - configuration parameters for Deepgram, etc - turn detection configuration - not streaming between models

Nice to see everyone here! Looking forward to hearing about what everyone is working on.