WhatsApp AI Agent Ignoring Vector Database – What Could Be the Cause?

I have created a WhatsApp agent that is supposed to answer questions about an event. Technically, everything works as I envisioned it, but I have a major problem. The agent is supplied with event information via a Supabase vector storage. I uploaded around 30 pages of a Word document as a knowledge base there (following the example from Nate’s RAG YouTube video).

When I query the agent via WhatsApp, it is supposed to retrieve its knowledge from the vector database every time. However, it occasionally responds without searching for specific information in the database. It then simply replies that it doesn’t have the information or it makes up fictional information, even though the information is actually available in the database—it just doesn’t call the tool. When I point out that it should check its database, it does so and usually finds the information.

Of course, I have noted in the system message that it should always refer to the database. But I don’t understand why it sometimes does this and sometimes doesn’t. The system message is quite long and extensive because the agent has to consider many factors to answer correctly. Could that be the reason?

Or could it be due to the context window or the model I’m using? I have ChatGPT-4o Mini in use. In the screenshot, I’ve only included the agent node.

Does anyone have an idea? If you need more information, just let me know. Thanks!

7 comments