Tool Gate · How it works
How the Prompt Regression Lab gate works
How the Prompt Regression Lab diffs a 5-case suite's v2 output against its v1 baseline: a regression gate fails the run when any case that passed before now fails — which is how 3 of 5 cases still pass and the run still reads REGRESSED.
How the gate works
This tool grades a prompt edit (v2) against a committed baseline (v1) test suite. It reads 5 test cases and rolls them into one verdict — the worst, never the average — and then a gate that can only make it worse.
The 5 test cases
| Test case | Verdict |
|---|---|
| love_it (positive to JSON)fix first | REGRESSEDbaseline pass, now fail · output dropped JSON for plain text |
| great_support (positive to JSON) | REGRESSEDbaseline pass, now fail · output dropped JSON for plain text |
| terrible (negative to JSON) | PASSbaseline pass, still pass |
| arrived (neutral to JSON) | PASSbaseline pass, still pass |
| reply_len (reply under 200 chars) | PASSbaseline pass, still pass |
The verdict bands
A case that passed in the committed baseline and fails now is a regression - even one regression forces REGRESSED and fails CI (exit 2), regardless of how many other cases still pass.
In the worked sample it reads REGRESSED — Three of five cases still pass, but love_it and great_support both passed in the committed baseline and now fail - v2 dropped JSON formatting on the positive-sentiment cases. One regression is enough: the gate reads REGRESSED and fails CI (exit 2), never averaging away the two broken cases. The one thing to fix first: love_it - restore JSON-formatted output for positive-sentiment replies; it silently switched to a bare string in v2..
Common questions
Can every test case look survivable on the How the Prompt Regression gate works and the company still read REGRESSED?
Yes. The roll-up takes the worst of 5 test cases — never the average — then a worsen-only gate, the Regression gate gate, escalates when trouble clusters. In the worked sample it reads REGRESSED because three of five cases still pass, but love_it and great_support both passed in the committed baseline and now fail - v2 dropped JSON formatting on the positive-sentiment cases. One regression is enough: the gate reads REGRESSED and fails CI (exit 2), never averaging away the two broken cases.
What is a worsen-only gate?
A dispositive rule that can only lower a verdict, never raise it. One fatal flaw overrides an otherwise-set of survivable lines, because a single disqualifying gap shouldn't hide behind an average of the healthy ones. The tool also names the one thing to fix first — here, "love_it - restore JSON-formatted output for positive-sentiment replies; it silently switched to a bare string in v2.".
This simulation runs in-browser only; the real kit runs your suite against your own model. Not a substitute for reviewing actual output quality.
Embed this diagram
Free to share and embed with attribution (CC BY 4.0) — keep the link to redhub.ai.
Interactive — renders the live diagram
<iframe src="https://redhub.ai/visuals/tool/prompt-regression-lab.html" title="How the Prompt Regression Lab gate works — RedHub AI" width="760" height="1751" loading="lazy" style="border:0;width:100%;max-width:760px"></iframe>
<p style="font:14px/1.5 system-ui,sans-serif"><a href="https://redhub.ai/visuals/tool/prompt-regression-lab">How the Prompt Regression Lab gate works</a> — by <a href="https://redhub.ai">RedHub AI</a>, the AI that tells you when to doubt it.</p>Image + link — a static picture for any blog
<a href="https://redhub.ai/visuals/tool/prompt-regression-lab"><img src="https://redhub.ai/visuals/tool/prompt-regression-lab-share.png" alt="Diagram of the Prompt Regression Lab: a five-case suite diffed v2-vs-v1, a regression gate, and the run reading REGRESSED on two newly-failing cases." width="760" loading="lazy" style="max-width:100%;height:auto;border-radius:16px"></a>
<p style="font:14px/1.5 system-ui,sans-serif"><a href="https://redhub.ai">RedHub AI</a> — the AI that tells you when to doubt it.</p>
This is how Prompt Regression Lab works. More diagrams in the Visual Field Guide.