How the roll-up gate works

How the Prompt Regression gate works

This tool grades a prompt edit (v2) against a committed baseline (v1) test suite. It reads 5 test cases and rolls them into one verdict — the worst, never the average — then a gate that can only make it worse.

The 5 test cases

01love_it (positive to JSON)fix firstbaseline pass, now fail · output dropped JSON for plain textREGRESSED

02great_support (positive to JSON)baseline pass, now fail · output dropped JSON for plain textREGRESSED

03terrible (negative to JSON)baseline pass, still passPASS

04arrived (neutral to JSON)baseline pass, still passPASS

05reply_len (reply under 200 chars)baseline pass, still passPASS

SHIP

every case still passes vs the committed baseline

HOLD

no regressions, but not every case currently passes

REGRESSED

any case that passed in the baseline now fails

Regression gate · worsen-only

A case that passed in the committed baseline and fails now is a regression - even one regression forces REGRESSED and fails CI (exit 2), regardless of how many other cases still pass.

REGRESSED

Why: Three of five cases still pass, but love_it and great_support both passed in the committed baseline and now fail - v2 dropped JSON formatting on the positive-sentiment cases. One regression is enough: the gate reads REGRESSED and fails CI (exit 2), never averaging away the two broken cases. 2 of 5 regressed · 3 pass · CI exit 2.

Fix firstlove_it - restore JSON-formatted output for positive-sentiment replies; it silently switched to a bare string in v2.

No single test case is the crisis — the clustering is. The roll-up takes the worst, the gate escalates when trouble compounds, and it names the one thing to fix first. This simulation runs in-browser only; the real kit runs your suite against your own model. Not a substitute for reviewing actual output quality.

RedHub AIthe AI that tells you when to doubt it redhub.ai/systems/prompt-regression-lab