Your agent passed every test you wrote. That proves it handles the cases you thought of. Customers bring the rest: a vague request, a tool that times out halfway through, a web page with instructions hidden in it, a question your policy never covered. This guide splits AI agent testing in 2026 into three layers, what each catches and misses, and a pre-launch checklist.
Short answer
To test an AI agent before launch, run three layers in order. Layer 1 is automated evals, a fixed set of tasks scored by code, a model grader or both, and re-run on every change. Layer 2 is scenario testing: a simulated user drives multi-turn conversations, tools fail on purpose, and a red team tries prompt injection. Layer 3 is real people running real tasks on the agent in a sandbox, before customers do. Each layer misses failures the others catch. Stop when the regression set passes on nearly every run, repeated trials of the core tasks agree, and people outside the team can finish those tasks without help.
The three layers at a glance:
- Layer 1: automated evals. Catches regressions, the wrong tool or wrong arguments, broken output formats and known failures from your bug tracker. Misses anything nobody wrote a test for, and a grader that is itself wrong. Run by code checks plus an LLM judge calibrated against human labels, on every change.
- Layer 2: scenario and adversarial tests. Catches multi-turn breakdowns, bad recovery after a tool error, results that change between identical runs, prompt injection and actions beyond the agent's authority. Misses how unscripted people phrase things, and whether the result is useful to them. Run by a simulated user, deliberate tool failures and a red team, before each release.
- Layer 3: real people on real tasks. Catches confusing instructions, fluent but wrong output, steps people abandon and a poor fit with the actual workflow. Misses regressions after launch, and anything behind systems the testers cannot reach. Run by human testers, in-house or outside, working from a written brief, before customers see it.
Why does an AI agent need different testing from normal software?
A unit test assumes the same input gives the same output. OpenAI's evaluation guide drops that assumption in its first line: "Generative AI is variable. Models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures."
Agents add actions on top of variable text. Anthropic's engineering post on agent evals says agents "operate over many turns: calling tools, modifying state, and adapting based on intermediate results", so "mistakes can propagate and compound". Its example is a flight-booking agent. The agent may say the flight is booked, but "the outcome is whether a reservation exists" in the database. Test what happened in the database. The agent's report of it proves nothing.
The NIST AI Risk Management Framework is blunt: "AI systems should be tested before their deployment and regularly while in operation."
The cost of skipping it has a court record. On February 14, 2024, the Civil Resolution Tribunal of British Columbia decided Moffatt v. Air Canada. Air Canada's website chatbot told a customer they could apply for a bereavement fare after travel, which the airline's policy did not allow. Air Canada argued the chatbot was in effect a separate legal entity. The tribunal disagreed: "It makes no difference whether the information comes from a static page or a chatbot." It ordered Air Canada to pay a total of C$812.02. This was a chatbot answering a question rather than an agent taking actions, and a small-claims decision. You still answer for what your agent tells customers.
Layer 1: automated evals, and what they miss
An eval is a set of tasks with defined inputs and success criteria. In Anthropic's vocabulary, each attempt is a trial, a grader scores it, and the transcript (also called a trace or trajectory) is the full record of what the agent did. Its advice on graders is the right default: "choosing deterministic graders where possible, LLM graders where necessary or for additional flexibility, and using human graders judiciously for additional validation."
Grade the result before the route. Anthropic warns that checking an exact sequence of tool calls is "too rigid and results in overly brittle tests", because agents find valid paths nobody planned, so it advises teams to "grade what the agent produced, not the path it took". Check the tool and its arguments where a wrong call does damage. Do not fail a run because the agent searched before it filtered.
How far can you trust an LLM judge?
In the 2023 MT-Bench study by Zheng and colleagues, strong judges like GPT-4 reached "over 80% agreement" with human preferences, the same level humans reach with each other. The same study found strong position bias. In a test that swapped the order of two answers that were "very similar and occasionally indistinguishable even to humans", "Only GPT-4 outputs consistent results in more than 60% of cases." Those are 2023 judges, and they graded chat answers rather than agent actions.
OpenAI's guide names the same biases (response order, longer answers) and tells teams to "validate agreement against your human labels before optimizing for cost or latency". A 2024 paper by Shankar and colleagues puts it more sharply: "LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation."
The eval itself can be wrong. Anthropic reports that Opus 4.5 initially scored 42% on CORE-Bench. "After fixing bugs and using a less constrained scaffold", its score rose to 95%. Its rule: "we do not take eval scores at face value until someone digs into the details of the eval and reads some transcripts."
Where to start
You do not need hundreds of tasks. Anthropic says "20-50 simple tasks drawn from real failures is a great start". If the agent already runs anywhere, mine the support queue and bug tracker, because "Converting user-reported failures into test cases ensures your suite reflects actual usage". OpenAI adds: "Log as you develop so you can mine your logs for good eval cases."
Layer 1 misses everything nobody thought to write down. Layers 2 and 3 exist to find it.
Layer 2: scenario, simulation and adversarial tests
Single-turn evals miss turn six, when the refund tool errors out.
Simulate the user
The tau-bench benchmark, published in June 2024, has a language model play the user while the agent works with "domain-specific API tools and policy guidelines". Grading compares the final database state with the expected one. The result: "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)".
That last number is the one to copy. The pass^k metric is the chance that all k trials of a task succeed. Anthropic's worked example: an agent that succeeds 75% of the time on each trial passes all three of three trials about 42% of the time. A customer does not care about your average. Each one gets one run.
Other benchmarks show the same gap between agents and people on multi-step work, with 2023 and 2024 models:
- WebArena (2023): the best GPT-4-based agent reached 14.41% end-to-end task success on realistic websites, against 78.24% for humans.
- OSWorld (2024): the best model reached 12.24% on 369 real computer tasks, while humans completed over 72.36%.
Do not read those as today's scores. Anthropic notes models went from 40% to more than 80% on SWE-bench Verified in a single year. The method is what lasts: check the end state, run many steps, and compare against a human baseline.
Anthropic adds two setup rules. Start each trial "from a clean environment", because leftover state skews the next run. And test when the agent should refuse as well as when it should act, because "One-sided evals create one-sided optimization". Then break things on purpose: make a tool return an error or a timeout, and check that the agent reports the failure instead of telling the user the job is done.
Red team the agent
An agent that reads email, web pages or files will read instructions someone planted there. OWASP's 2025 Top 10 for LLM applications defines the attack: "Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files." It also says "it is unclear if there are fool-proof methods of prevention for prompt injection", and recommends penetration testing that treats the model as an untrusted user.
AgentDojo, from ETH Zurich and Invariant Labs, packages this as 97 realistic tasks (an email client, e-banking, travel booking) and 629 security test cases. Its 2024 finding: "state-of-the-art LLMs fail at many tasks (even in the absence of attacks), and existing prompt injection attacks break some security properties but not all."
Since prevention is not solved, limit the damage. OWASP's entry on excessive agency recommends a human approval step for high-impact actions, and permission checks in the downstream system rather than trusting the model to decide what it may do. Your red team's job is to prove those checks hold.
Layer 3: real people on real tasks, before your customers
Layers 1 and 2 test the agent against your idea of the job. Layer 3 tests it against someone else's.
In the October 2025 paper How Do AI Agents Do Human Work?, researchers at Carnegie Mellon and Stanford compared 48 human workers with four agent frameworks on 16 work tasks. The agents delivered results 88.3% faster, and the two agent setups the paper priced cost 90.4% to 96.2% less per task than a human worker. They also found that "Agents produce work of inferior quality, yet often mask their deficiencies via data fabrication and misuse of advanced tools." The 16-task sample is small, but the failure it describes is the dangerous kind: confident output that looks finished. Their conclusion: "human oversight may still be necessary to verify the correctness of agent-produced work."
Anthropic says plainly why customers should not be your testers: "Relying primarily on users to catch issues can have negative user impact". It also names the cost of human review: "Reviewer fatigue or different reviewers can affect the signal quality". So use people where only people can tell: a pre-launch pass, and again after a major change.
Outside testers add what your team cannot. NIST says "Processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest." The people who built the agent know which phrasing it expects.
Write the brief like an eval task
Anthropic's standard for an eval task works for a human tester's brief too: "A good task is one where two domain experts would independently reach the same pass/fail verdict." Pond's guide to writing a task brief uses five fields: "what counts as done, what evidence has to be attached, what is out of scope, what you pay for and what you do not, and the deadline." "Find bugs" is not a standard. "A report that reproduces on a stated build, with a screen recording attached" is.
Use a sandbox, test accounts and test data for outside testers, and keep production credentials and real customer records out of their reach.
Who runs layer 3
Managed human testing of agents already exists. Testlio describes its offer as "managed crowdsourced AI testing", covering "planning, execution, and coordination in single- or multi-agent systems", priced as a platform fee plus an annual consumption fund. Applause says it tests agents "with real humans, at scale, before they go live".
A single posted task is the other shape. Moatt shows what people found before launch. TinyFish shows how fast people turned up. Moatt, which gives companies an AI growth team that runs SEO, advertising and competitor research on a schedule, tested for exactly this risk: "Confusing instructions, inaccurate outputs, or a failed step could prevent users from reaching the moment that makes the product useful". Testers had to run a real growth task for at least 15 minutes and capture screenshots. One week brought 245 registrations, 99 proof-backed product tests and 35 verified reports, for $700 (the Moatt write-up). TinyFish, whose agent was already in enterprise use, wanted to see "how it behaves in someone else's hands": within four days, 83 people and AI agents had run real fetch and web-navigation jobs through its agent, on a reward pool sized for about 10 payouts. That run shows turnout, not findings, because the feedback "went to TinyFish rather than to us" (the TinyFish write-up). Each is one task's result, not a typical one.
Pond's own QA testing page is clear about where this stops: "Do not outsource QA that needs access to sensitive internal systems." And on regression: "A task is the wrong shape for it." Layer 3 is a pre-launch pass. Layer 1 stays your regression suite.
How do you know when to stop testing?
Decide the bar before you start, then revise it once. Shankar's 2024 study found "users need criteria to grade outputs, but grading outputs helps users define criteria", so read real transcripts, adjust the criteria, and freeze them before the final run.
Ship when these hold:
- Regression tasks reach what Anthropic calls "a nearly 100% pass rate".
- Core customer tasks pass every trial at the k you chose in advance.
- Spot checks of the LLM judge agree with your human labels.
- Every red-team finding is fixed, or its action sits behind human approval.
- Outside testers finished the core tasks from the brief alone, and each serious finding is now a layer-1 test.
Then keep testing in production, as NIST says: "regularly while in operation".
Pre-launch checklist for AI agent testing
- Write down the agent's core tasks and what "done" means for each, checked against the system's end state.
- Build 20 to 50 eval tasks from real failures, support tickets and logs.
- Use code graders wherever the answer is checkable. Add an LLM judge only where it is not, and validate it against human labels.
- Read transcripts from many trials before you trust any score.
- Run each core task several times from a clean environment and report pass^k, not the average.
- Simulate multi-turn users, including ones who change their mind or go off script.
- Force tool errors, timeouts and empty results, and check the agent reports them.
- Red team for direct and indirect prompt injection on every input the agent reads.
- Put high-impact actions behind human approval and downstream permission checks.
- Run a sandboxed pass with outside testers on a written brief that names the required evidence.
- Turn every serious finding into a regression task.
- Set the stop criteria in writing, then schedule testing in production.
Frequently asked questions
How do you evaluate agentic AI?
Check the outcome in the environment and ignore the agent's own account of it. Build a set of tasks with clear success criteria, run each several times from a clean state, and grade with code where you can and a calibrated LLM judge where you cannot. Add simulated multi-turn users and adversarial inputs, then a pass with real people before launch.
Which AI tool is best for testing an AI agent?
No single tool covers all three layers, so pick one per layer. Layer 1 needs an eval framework that runs a fixed task set with code graders and a calibrated LLM judge. Layer 2 needs a user simulator, fault injection for tool errors and prompt-injection tests. Layer 3 needs human testers working from a written brief. Read the transcripts behind any score.
Can an LLM judge replace human testers?
Not on its own. In the 2023 MT-Bench study, GPT-4 judges agreed with human preferences over 80% of the time, but showed strong position bias. OpenAI tells teams to validate a judge against human labels first. Use the judge for scale and people for calibration and for anything subjective.
How many test cases do you need before launching an AI agent?
Fewer than most teams expect. Anthropic says "20-50 simple tasks drawn from real failures is a great start". Grow the set from logs and user-reported failures, and move a task into the regression suite once the agent passes it reliably.
What is the difference between agent evaluation and agent testing?
Agent evaluation scores the agent against a fixed task set, usually automatically, so you can compare versions. Agent testing is the wider pre-launch job: evals plus scenario runs, red teaming and real people using the agent. Anthropic's eval guide covers "automated evals that can be run during development without real users".
What is pass^k, and why does it matter for customer-facing agents?
The pass^k metric is the chance that all k trials of a task succeed. An agent with a 75% per-trial success rate passes all three of three trials about 42% of the time, by Anthropic's example. Anthropic says the metric "especially matters for customer-facing agents where users expect reliable behavior every time."
Ask about benchmarks before you accept a build
If someone else builds your agent, the layers in this guide are what to ask any builder about before you accept it: the eval task set and its pass rate, pass^k on the core tasks, and a written brief for the real-people pass.
We build, run and maintain custom AI agents for businesses. Custom builds start from $1,000 and usually take one to two weeks. The refund terms on the site read: "Custom builds by the Pond team are refunded within 7 days if benchmarks aren't met." That 7-day refund applies only to custom builds. Get a custom AI agent to book a 30-minute consultation call, and ask there which benchmarks apply to your build.
