My VAPI assistant is behaving unpredictably — lag, mishearing names, tone glitches, repeating itself, breaking flow. Anyone seen this?

Hey everyone,I’m building a voice booking assistant for Client's Company using VAPI, and I’m running into a bunch of strange behaviour that doesn’t line up with my prompt or tool logic.

I’m hoping someone who has worked with complex VAPI assistants can help me diagnose what’s going on.

I have attached the VAPI prompt to this post, I have deleted my client's company from the prompt so if there are any gaps in tool names or text that is why

🔥 Issues I'm Experiencing:

1. The assistant sometimes lags badly

There are delays where the assistant just freezes for a second or two before responding.

2. It often says “I could not find that restaurant” even when the restaurant exists

The ID lookup tool works perfectly when I test it directly, but inside VAPI the assistant frequently fails to pick up names and jumps straight to “could not find that restaurant”.

It especially struggles with:

“Alla Bolognese” → hears it as something completely different
Restaurant names in general

3. The call audio breaks up / interrupts / talks over the user

It keeps speaking before I have a chance to confirm details.It will:

Interrupt me mid-sentence
Repeat itself
Apologise multiple times even though nothing actually happened
Talk over the user during confirmations

It feels like endpointing or turn-taking is misfiring.

4. Repeats itself and seems “stuck” sometimes

At random points, it loops:

Apologising repeatedly
Repeating the same line
Re-explaining a step it already completed

This doesn’t match the prompt at all.

5. Tone is inconsistent and sometimes shifts dramatically

This one is wild:

It speaks in a normal tone
Pauses
Then suddenly shouts with excitement, like a different TTS voice style
Then goes back to normal

It’s like ElevenLabs suddenly spikes in emotion even though stability is low and the prompt asks for calm, consistent tone.

6. Sometimes speaks in another language (!!)

While repeating an email address, the assistant suddenly switched to an incomprehensible language for a sentence before coming back.

Never seen this before.

7. Goes off-script or contradicts itself

Examples:

I confirmed all details→ It then said it couldn’t find the place→ Then it cancelled the booking when I said “don’t cancel”
It sometimes jumps forward in the flow before collecting required variables
Sometimes it skips tool calls entirely and pretends it already ran them

8. Struggles with accurate speech recognition

Deepgram usually works fine for me, but here:

It mishears restaurant names
Mishears emails
Mishears phone numbers
Mishears dates/times

Even when spoken clearly.

🧩 What I’ve Tried

Cleaned and simplified system prompt
Reworked tool descriptions
Checked all tool responses
Adjusted cadence, endpointing, silence timeout
Tested tools outside VAPI (they work perfectly)
Tried with and without background noise
Tried different voice models

Still seeing the same behaviour.

❓ What I'm Hoping to Understand

Are others seeing stability issues with VAPI lately?
Could this be a turn-taking / endpointing issue?
Is GPT-4o inside VAPI known to “go off script”?
Could the assistant be mixing messages due to overlapping tool calls?
Are Deepgram transcripts sometimes causing chaotic behaviour?
Is there something in my system prompt that could be conflicting with VAPI’s runtime?

💬 Any insight would be massively appreciated

This behaviour is extremely inconsistent and doesn’t match what the tool logic or prompt dictate — so I feel like I’m missing something deeper at the VAPI runtime level.

Happy to share any additional details if needed.

Thanks in advance!

1 comment