How the gate works

How the Agent Reliability Harness gate works

This tool grades one AI-agent trajectory, run as a one-case suite, across six evaluators before it ships. Each signal is scored 0–1; the verdict is the weakest one — then a gate that can only make it worse.

The 6 signals

01task_success

02tool_choice

03argument_validity

04step_efficiency

05cost_budget

06policy

SHIP

every case passes and the reliability score is 0.90 or more

HOLD

short of SHIP, with 80% or more of the cases passing - a suite of several cases

FIX

under 80% of the cases pass - on one trajectory, any failing evaluator

Verdict gate · sets the verdict

A case passes only when every graded evaluator passes. The suite reads SHIP only when every case passes and its reliability score is 0.90 or more; short of that it reads HOLD while 80% or more of its cases pass, and FIX below 80%. One trajectory is a one-case suite, so a single failing check is a 0% pass rate and reads FIX, whatever its reliability score.

FIX

Why: The looping-agent trajectory reaches the right final answer - task_success still passes - but it takes 6 steps against a budget of 5, repeating the same call 5 times. step_efficiency scores 0.40 and is the only failing check, so the case fails. Run as a one-case suite, that is a 0% pass rate and a FIX verdict at a reliability score of 0.90, even though five of six evaluators pass. weakest signal 0/1 · 5 of 6 checks pass, reliability 0.90.

Fix firststep_efficiency - 6 steps > budget 5; repeated identical call x5 (possible loop)

A right final answer didn’t save it. One failing check fails the case, and a one-case suite at a 0% pass rate reads FIX whatever its reliability score. Where a check fails, the fix first row names it. Grades the guardrails and trajectory you describe. It does not run or change any agent, and it never scores a person.

RedHub AIthe AI that tells you when to doubt it redhub.ai/systems/agent-reliability-harness