🧙 AI Agency Wizardry 🧙 - Connect OpenAI Realtime API With Twilio

So, most of you would have heard the big news in the OpenAI sector over the last few days, they have released the real-time API. This means that OpenAI will now manage all of the voice calls on their server, essentially replacing services like VAPI.

In this demonstration, I'll show you how to connect Twilio to a Replit to manage calls with the OpenAI real-time API. All the audio from the call is sent to the OpenAI system via the real-time API. It uses the OpenAI LLM to process the context of the call and then uses OpenAI voices to send back an audio response that is played over the phone.

If you would like to build your own test system and follow along with this video, I have included the Replit URL that you can fork. As long as you have an OpenAI API key and a Twilio account, you will be able to build your own system for demonstration purposes.

The Replit Clone URL: https://replit.com/@AndrewViddusRea/Connect-OpenAI-Realtime-API-With-Twilio?v=1

From my testing, the latency is about equivalent to what VAPI is currently producing. The big difference, however, is the pricing while testing, it has been costing between $0.16 and $0.21 per minute, depending on how much talking the OpenAI voices do. Here is the pricing from the OpenAI website, and as you can see, it's not cheap.

The Real-Time API uses both text tokens and audio tokens. Text input tokens are priced at $5 per million, and $20 per million output tokens. Audio input is priced at $100 per million tokens, and output is $200 per million tokens. This equates to approximately $0.06 per minute of audio input and $0.24 per minute of audio output. Audio in the Chat Completions API will be the same price.

Currently, there seem to be only three voices available, but there may be more voices added in the future, so please keep an eye on OpenAI's documentation.

Quick Note: Their call interrupt handling is really poor at the moment and that's one big issue that they will likely address because there's not much you can do to change it. There is a setting variable you can pass, but it doesn't really make a difference, I played with it for quite a while.

81 comments

🧙 AI Agency Wizardry 🧙 - Connect OpenAI Realtime API With Twilio