Your agent passed.The path it took didn’t.
Agents don’t fail like prompts — they fail in the path. A loop that burns tokens. A tool called with hallucinated arguments. A destructive action with no confirmation. A run that costs 10x budget. Grade only the final answer and every one of these passes — until production.
The Agent Reliability Harness scores the whole trajectory — tool choice, arguments, steps, cost, and policy — and returns one honest verdict: SHIP / HOLD / FIX. The CLI fails the build on FIX or a regression, so an unreliable agent can’t ship quietly.
If you can’t evaluate it, you can’t ship it.
Agent evals mostly grade the final answer — so they sail right past how the agent actually behaved. The answer looks right; the agent looped five times, skipped a required step, or deleted a record without confirming to get there. None of it shows up until a customer hits it.
Regular software has a fix: tests that fail the build when a change breaks something that worked. This kit brings that discipline to agent runs — a deterministic battery of trajectory checks and a hard gate, so “it answered correctly in the demo” isn’t your release process.
The answer is right; the agent repeated the same call five times and burned tokens to get there. An answer-only eval scores it a clean pass.
A correct outcome reached by deleting a record without confirmation is still an incident. The path, not the answer, is where the risk lives.
'Correct but ten times too expensive' never appears in a final-answer eval — until it appears on the bill.
Pick a broken agent. Watch the path fail.
Five trajectories — one clean, four broken. Each broken run still passes the success check and fails the path. The per-check verdicts and reliability scores here are the exact output of the shipped example suite. (Simulation, no real agent; it resets on reload.)
Simulation · no real agent
Pick a trajectory — read the scorecard
Repeats the same search five times, then answers.
Verdict: FIX
reliability 0.90 · 1 check failingNotice task_success still passes — the agent reached the “right” answer. A final-answer-only eval would call this a win and miss the failure entirely.
This is a simulation — and it resets on reload.
The real harness scores your own agents’ traces across all six evaluators, exits non-zero on a FIX verdict or a regression vs a committed baseline, and drops into CI with the included GitHub Actions workflow. Python, framework-agnostic, zero dependencies.
Get the Agent Reliability Harness — $149
How the gate works, in one image
How the Agent Reliability Harness grades an agent trajectory across six evaluators and ships only when every one passes — the same math the demo runs, as a diagram you can share or embed anywhere.
View & embed the full diagramSix evaluators across the whole trajectory.
Whether the goal was actually met — a custom checker or required substrings in the final answer. The one check answer-only evals already do.
Wrong, missing, or out-of-scope tools — checked against allowed / required / forbidden lists, so a disallowed or skipped tool fails the run.
Malformed or hallucinated tool arguments, validated against per-tool JSON schemas. Uses jsonschema if present, a built-in validator otherwise.
Blown step budgets and loops — repeated identical calls are flagged as a possible loop, the failure that quietly doubles token spend.
Token and dollar budgets — the 'correct but 10x too expensive' failure that never shows up in an answer-only eval.
Safety and behavior rules as code — e.g. confirm before a destructive action, never call tool X. Ships with a confirm-before-delete policy helper.
Each check skips itself when you don’t set an expectation for it — so you grade only what you care about, and a skipped check never inflates or penalizes the score.
A complete kit, fully tested, ready for CI.
Six evaluators, the Trace/Case/Expect model, runner, report, and a CLI. Runs on the standard library; jsonschema is used automatically if installed.
One clean run and four broken ones — loop, unconfirmed delete, bad arguments, over budget. Each evaluator catches its problem; the suite lands on FIX.
Feed it traces from any framework: a JSON loader, an OpenAI tool-call message adapter, and a tiny schema you can map LangGraph / CrewAI logs onto.
A GitHub Actions file plus non-zero exit on FIX, below --fail-under, or any regression vs a committed baseline — drops straight into a pipeline.
A readable console scorecard, a JSON report you commit as the baseline, and a Markdown report for PRs — one source, three formats.
23 unit tests covering every evaluator, a README, and an honest note on what it is and isn't. Small enough to read end to end.
Four steps, then it runs itself.
Per test case, declare allowed/required/forbidden tools, argument schemas, step and cost budgets, and policies. Each check skips itself when you don't set an expectation.
Use the Trace schema, the JSON loader, or the OpenAI adapter. Any framework — you don't run your agent through the harness, you hand it the trace it produced.
Per-check pass/fail, a reliability score, and a SHIP / HOLD / FIX verdict. Export to console, JSON, or Markdown.
The CLI exits non-zero on FIX, below --fail-under, or on any regression vs a baseline. Drop in the included GitHub Actions workflow and every PR is checked.
What this is — and isn’t.
This is an offline / CI evaluation harness, not a runtime guardrail: it evaluates traces, it doesn’t execute your tools or agents or sit in the live request path. Pair it with runtime guardrails for defense in depth. It’s deterministic and rule-based by default — reproducible and effectively free, no LLM-judge bill — and you can add a judge to the success check when you want one.
The default thresholds and policies are a sensible starting battery, not a universal standard — tune the budgets and rules to your own risk tolerance. And it grades the trajectory you give it: the quality of the verdict is only as good as the expectations you write. The kit is small enough to read end to end — which matters for a tool that decides what ships.
Anyone shipping agents to production.
Agent builders who need to catch loops, unsafe paths, and bad tool calls before users do; platform and ML engineers who want a reproducible eval wired into CI and trended over time; teams shipping agents on a schedule who want a real gate — a verdict and a regression check on every PR; and consultants who want a repeatable reliability harness to evaluate clients’ agents. It’s the fourth gate in the dev-tools line: retrieval, security, regressions, and now agent trajectories.
Prompt Regression Lab
$89The functional-regression gate for prompts: snapshot a baseline, diff every change, fail CI on any regression. Same shape as the Harness — deterministic, CI exit codes, baseline diffing — applied to prompt outputs instead of agent trajectories.
Prompt Injection Red Team Kit
$99The security gate. Run OWASP-mapped injection and prompt-leak probes against your app and gate CI on a severity-weighted verdict. The Harness checks whether the agent behaves; the Red Team Kit checks whether it can be made to misbehave.
RAG Retrieval Grader
$89The retrieval gate. Score Recall@k, MRR, and nDCG against a gold set and fail CI when retrieval drops. The Grader proves the right context reaches the agent; the Harness proves the agent then uses it on a safe, efficient path.
The questions engineers actually ask before wiring it in.
A Python toolkit that evaluates AI agents at the trajectory level — not just the final answer. It checks tool choice, argument validity, step efficiency, cost, and policy across a run, then returns a clear SHIP / HOLD / FIX verdict you can gate CI on.
Final-answer evals miss how the agent got there. An agent can return a correct answer while looping, calling a disallowed tool, deleting a record without confirmation, hallucinating arguments, or costing 10x the budget. This harness catches those silent failures — in the bundled example, every broken agent passes the success check and still fails the trajectory.
Yes. You don't run your agent through the harness — you feed it traces. There's a simple Trace schema (task, steps, final output), a loader for JSON traces, and an adapter for OpenAI-style tool-call messages. A thin adapter from any framework's logs is all you need.
No. The checks are deterministic and rule-based by default, so results are reproducible and effectively free. You can plug an LLM judge into the success check if you want one, but it's optional and never gates the deterministic core.
Yes. The CLI exits non-zero on a FIX verdict, below a --fail-under threshold, or on any regression versus a committed baseline. A ready-to-use GitHub Actions workflow is included — the same regression idea as the Prompt Regression Lab, applied to agent trajectories.
The Python package (six evaluators, runner, report, CLI), a worked five-scenario example, a JSON traces sample, a pytest suite (23 tests), a CI workflow, and docs. Python 3.9+ with zero required dependencies; jsonschema is used automatically if it's installed, otherwise a built-in validator handles your tool schemas.
No — it's an offline/CI evaluation harness; it doesn't execute your tools or agents or sit in the live request path. Use it alongside runtime guardrails for defense in depth. The default thresholds are a sensible starting battery, not a universal standard — tune them to your own risk tolerance.
Stop shipping agents on hope.
Score the whole trajectory and gate every deploy on a verdict you can trust. Python, framework-agnostic, zero dependencies. One-time $149, yours to keep.
Sold by RedHub AI LLC · Secured by Stripe · redhub.ai