Shipped my open-source agent testing tool today — would love feedback from this community

Been heads-down on this for a while with my friend Yashrajsinh Rathod, and we finally got it to a clean, tested state today — sharing here first before the wider launch.

The problem: testing AI agents sucks. Every test run hits a real LLM API (slow, costs money, and the non-determinism means your CI is flaky by design). Most of the "agent observability" tools out there (Langfuse, Braintrust) solve this by having you send your traces to their cloud and run evals there.

What we built instead: Fixtura — record your agent's actual tool calls once (with permission checking baked in), then replay that recording offline, as many times as you want, with zero live API calls. The recordings become literal test fixtures, like snapshot tests, that can gate your CI without touching a live LLM.

It's local-first on purpose — your traces never leave your machine unless you want them to.

Try it in about 30 seconds:

pip install fixtura

python quickstart.py

GitHub: https://github.com/yash161004/fixtura-core

Would genuinely love this community's eyes on it, especially if any of you are building agent products and dealing with flaky non-deterministic tests. What's your current testing setup look like — are you mocking LLM calls, or just eating the cost/latency of hitting the real thing every run?

— Yashrajsinh Rathod & Meet Patel

0 comments

Shipped my open-source agent testing tool today — would love feedback from this community