Someone asked is our "new girl" okay? not knowing it is just AI run by me and my client.

Today I'm going to walk through the actual stack because most write-ups skip this part entirely.

Most people in this space talk about AI agents in theory. I actually deployed one for a real business and the results were weird enough that I think it's worth writing up properly.'

We built a voice agent that handles inbound calls. Not a phone tree, not press 1 or something like that. An actual conversational agent.

The stack:

The telephony layer receives the call and streams audio in real time. That audio goes to a speech-to-text engine the key here is latency. If transcription takes more than 300ms people feel it, and the conversation starts sounding robotic. Getting this right took a few iterations.

The transcribed text hits the LLM. I'm using a system prompt that gives the agent a specific persona, a defined scope (it doesn't try to answer questions), and hard limits anything specific gets redirected to "please speak to our team directly." or "Ask to trasnfer the call if needed".

The LLM response goes to a TTS engine and gets streamed back as speech. The whole roundtrip has to stay under a second for it to feel like a real conversation. Under 700ms is the sweet spot where user don't notice any difference.(Can Speak multiple languages ).

The booking piece connects to a workflow automation layer that talks to the Client calendar. When a patient confirms a time, it creates the appointment, logs the call, and sends a confirmation.

What actually surprised me:

The after-hours call volume was significant. I expected maybe 15–20% of bookings to happen outside business hours. It was closer to 35%.

The other thing: People were more patient with the AI than I expected. As long as the voice didn't sound synthetic and the agent didn't loop or get confused on simple inputs, people just used it normally. Drop off happened when the agent tried to handle something outside its scope and fumbled. Keeping the agent's scope tight matters more than making it do more things.

The failure modes nobody talks about:

If the STT gets a name wrong and books under the wrong user that's a real problem. You need a confirmation step that reads back the full booking/any important details before finalising.

LLM context management matters more than people think. If a user calls back 10 minutes later to change their booking or try to reschedule, the system needs to handle that gracefully without treating it as a brand new conversation.

Silence handling. If a user doesn't respond for 3 seconds, what happens? If the agent does nothing, the caller thinks the line dropped. If it immediately speaks, it interrupts people who are just thinking. Small thing, massive UX difference.

The part that actually keeps the system from cancelling

Without this running, the after-hours calls go unanswered again. That's not a software inconvenience that's real revenue walking out the door every night. Once you've shown someone what they're losing, the ROI conversation is already over.

I'm not going to pretend this was easy to build the first time. But now that the architecture is solid, deploying it for a new business takes a fraction of the time.

Happy to answer questions in the comments especially around the latency piece, that's where most people get stuck first.

1 comment

Someone asked is our "new girl" okay? not knowing it is just AI run by me and my client.