Most AI agents get tested the same way. Someone types a few questions, the answers look good, and the agent ships. Two weeks later a prompt gets tweaked, something breaks, and nobody can say when it started.
The fix isn't a better model. It's a test you can run again. Here's the one we build on every engagement.
1. Collect real cases before you write a prompt
Pull 25 to 50 examples of the actual work. Real invoices, real questions, real requests. Scrub them if you need to, but don't invent them. Invented cases are too clean.
Include the ugly ones on purpose:
- The scanned document that's rotated.
- The request that's missing a required field.
- The question that has no answer in your documents.
- The case where the right move is to stop and ask a person.
If the agent only ever sees easy cases, the test will only ever tell you it's fine.
2. Write down the right answer for each case
For every case, record what a good result looks like. Keep it specific enough that two people would grade it the same way.
- For extraction, the exact field values.
- For a research answer, the facts that must appear and the source they come from.
- For an action, which tool should be called, with what inputs, or that no tool should be called at all.
This is the slow part. It's also the part that makes everything after it fast. Have the person who owns the workflow sign off on the answers, not the engineer.
3. Score three things, not one
A single accuracy number hides the failures that hurt most. Track these separately:
- Was the result correct? Field match, fact match or the right decision.
- Did it stop when it should have? An agent that guesses on a bad input is worse than one that's wrong on a good one.
- Did it stay in bounds? No tool calls it wasn't allowed to make, no data it wasn't allowed to read.
Where you can, score with code. Exact match, numeric tolerance, a required citation. Use a model as a grader only for things code can't judge, and spot-check that grader against a person.
4. Set the pass line before you see the results
Decide what "good enough to release" means while you're still neutral. For example: every must-stop case stops, no out-of-bounds actions, and at least 45 of 50 cases correct.
Write it down. If you pick the bar after you see the score, the bar will end up wherever the score is.
5. Run the whole set on every change
A new prompt, a new model version, a new tool, a changed retrieval setting. Each one gets the full run before it reaches users. Compare against the last run, case by case.
Averages lie here. A change can lift the overall score while breaking three cases that used to pass. Look at what flipped.
No pass, no release. That one rule is what turns a demo into a system you can keep.
6. Feed production back into the set
Every failure a user reports becomes a new case, with its right answer. The set grows where the agent is weakest. After a few months it describes your workflow better than any requirements document.
What this costs
For one workflow, the first case set usually takes a few days of work with the person who owns the process. After that, a run takes minutes.
Skip it and you pay later, in the hours spent arguing about whether the agent got worse or always did that.
Where to start
Pick one workflow. Pull 25 real cases. Write the right answers. You'll learn more about the process in that afternoon than in a month of demos.
That case set is the first thing we build in a Workflow Diagnostic, and it becomes the acceptance test for anything we build after. If you'd like help setting one up for agents on Palantir AIP or your own stack, get in touch.