01
Confident wrong answers
The agent cites a policy that doesn't exist, calls the wrong tool, or quietly skips a step — and nothing alerts.
AI agent reliability testing
Know your AI agent works — before your customers find out it doesn't.
Vettwise tests your agent against hundreds of real scenarios, shows you exactly where and why it fails, and sets up the checks that keep it reliable after every release.
Running an AI agent from Gorgias, Intercom, Zendesk or similar? AI agent QA for CX teams
audit_run · support-agent
sample · illustrative
- PASS refund_policy / standard request
- FAIL refund_policy / order older than 30 days↳ cites an exception that doesn't exist
- PASS order_status / lookup by email
- FAIL product_search / ambiguous name↳ calls cancel_order instead of search
- FLAKY address_change / multi-turn↳ loses context at turn 4 in 2 of 5 runs
- PASS escalation / hand-off to human
196 pass31 fail13 flaky240 scenarios
The problem
Green dashboards. Wrong answers.
Uptime checks tell you your agent responded. They don't tell you it was right. Most agent failures look perfectly healthy in your logs.
02
Silent regressions
A prompt tweak or model upgrade fixes one case and breaks five others nobody re-tested.
03
"How do you know it works?"
Enterprise buyers ask. "We tried it a lot" isn't an answer that closes deals.
The offer
Agent Reliability Audit
A two-week, fixed-price audit that tells you exactly how reliable your agent is, why it fails, and what to fix first.
Each additional agent +$1,500, up to four agents. Five or more: custom quote.
* For more than one agent, the audit takes up to three weeks.
- 2 weeks*
- Fixed price
- $4,000 · first agent
Scenario test set
100–300 test cases built from your real customer conversations, docs and known edge cases.
Full test run
Your agent scored on correctness across the whole trajectory — answers, tool calls and hand-offs, not just uptime.
Failure report
Every failure mode, its root cause, and a prioritised list of what to fix first.
A harness you keep
The test suite and scripts are yours, ready to re-run on every release.
After the audit
Eval & regression setup
from $9,500Turn the audit into an automated safety net. Your test suite runs on every prompt, model or knowledge-base change, with pass/fail thresholds that stop regressions before they reach customers.
Reliability retainer
from $2,500 / monthReliability that keeps up with your agent. Every month we re-run the full suite, turn new production failures into test cases, test major changes before release, and send you a short reliability report.
How it works
From first call to fix list in two weeks.
-
Intro call
20 minutes. What your agent does, what "correct" means for you, and where it worries you.
-
Scope & access
We agree scope, sign an NDA and DPA, and you share a sample of real customer conversations, your docs and test access.
-
Test & analyse
We build the scenario set, run it repeatedly, and dig into every failure until we know the cause.
-
Report & handover
A walkthrough call, the written report, and the test harness handed to your team.
Timeline shown for one agent. Audits of two to four agents take up to three weeks.
Why not just use a tool?
Tools give you dashboards. We give you answers.
Eval platforms are great — once someone decides what "correct" means, writes realistic scenarios, and reads the failures. That's the part we do, inside whatever stack you already use.
- LangSmith
- Braintrust
- Langfuse
- DeepEval
- your own code
Who's behind Vettwise
Eddie Dimitrov, founder
AI engineer and product owner with three and a half years building agentic systems — most of it spent on the unglamorous part: validating that agentic flows actually do what they're supposed to, before and after they ship.
You work with me directly, from first call to final report.
Questions
What do you need from us?
A sample of real customer conversations or traces from production (usually a few hundred), the docs your agent relies on, and a staging environment or test account. Plus someone who can answer what "correct" looks like.
Do you really need real customer conversations?
Yes, that's where the real failures are. Synthetic tests alone miss how your users actually phrase things. You can remove personal data before sharing, or we handle it under a data processing agreement.
Our agent isn't live yet. Can you still help?
Yes. A pre-launch audit uses your docs, expected customer questions and test conversations instead of production data. It's less representative than a live audit, but it's the cheapest moment to find failures: before any customer does. Same price.
How does pricing work for multiple agents?
The audit is $4,000 for one agent. Each additional agent is $1,500, up to four in total, and we also test the handoffs between them. For five or more agents, we put together a custom quote. One agent means one customer-facing assistant with its own instructions and tools.
Do you need access to production?
No. We prefer staging or a test account where tools can't take real actions like issuing refunds or placing orders. If production is all you have, we test read-only and mock anything that changes data.
How do you handle our data?
We sign an NDA and a data processing agreement before any access. Your data is used only for your audit, accessed only by us, and deleted within 30 days of the final walkthrough. We confirm the deletion in writing.
Do we need a specific eval platform?
No. We work inside the tools you already use, or leave you a lightweight harness in your own codebase.
What kinds of agents do you test?
Customer support, sales, workflow and document agents: anything that talks to users or takes actions through tools.
Can you fix what you find?
The audit tells you what to fix and in what order. If you'd like, we can implement the fixes, quoted per fix list, or set up regression testing so they stay fixed.
Does the audit guarantee our agent won't fail?
No test can prove an agent will never make a mistake. The audit finds and explains failures within the agreed scope, so you can fix the ones that matter before your customers find them.
Find out where your agent breaks — before your users do.
A 20-minute call. No prep needed, no obligation.