Change a prompt without breakingwhat already worked.
You tweak the wording, eyeball two outputs, and ship — with no idea what you quietly broke for the other twenty cases. Swap a model and the surface area gets worse. The JSON your code depends on silently turns into prose; a fixed edge case comes back.
The Prompt Regression Lab brings regression testing to prompts: run a suite against your model, snapshot a baseline you commit to git, and diff every change. A case that passed before and fails now fails the build — so a quiet regression never ships.
Prompt iteration is mostly vibes.
You tweak the wording, eyeball two or three outputs, and ship — with no idea what you quietly broke for the other twenty cases. Swap a model or bump a version and the surface area gets worse. None of it shows up until a customer hits it.
Regular software has a fix: regression tests that fail the build when a change breaks something that worked. This kit brings that discipline to prompts — a committed baseline and a hard gate, so “looks fine to me” isn’t your release process.
The structured output your downstream code parses silently becomes plain text after a wording change. Nothing errors — until the parser does, in production.
A reply that was tight and on-format gets longer, chattier, or reorders its fields. No single test fails, but the contract your UI assumed quietly moved.
An edge case you fixed last month comes back after an unrelated tweak. Without a baseline, regressions like this are invisible until someone reports them.
Edit the prompt, watch the gate flip.
The baseline is prompt v1. Flip the current prompt to v2 — an innocent-looking edit — and watch the diff flag the cases that regressed and turn the verdict red. Same logic the real kit runs against your own model. (Simulation, no real model; it resets on reload.)
Simulation · no real model
Edit the prompt — catch the regression
Verdict: regressed
3/5 pass · 2 regressed · 2 driftedA case that passed in the baseline now fails. CI fails (exit 2) until you fix it — or update the baseline on purpose.
| Case | Baseline | Current | Status |
|---|---|---|---|
| love_itpositive → JSON | pass | fail | REGRESSED |
| terriblenegative → JSON | pass | pass | — |
| arrivedneutral → JSON | pass | pass | — |
| great_supportpositive → JSON | pass | fail | REGRESSED |
| reply_lenreply ≤ 200 chars | pass | pass | — |
v2 dropped JSON formatting on the positive cases — easy to miss by eyeballing, caught instantly against the baseline.
This is a simulation — and it resets on reload.
The real kit runs your suite against your own model in Python or TypeScript, diffs against a baseline you commit to git, and fails CI (exit 2) on any regression. Deterministic checks, A/B compare, HTML report, zero dependencies, fully offline.
Get the Prompt Regression Lab — $89Deterministic checks, plus the diff that names what moved.
JSON validity, field equality, regex, contains, equals, one_of — the structural contracts your downstream code depends on, asserted deterministically.
Max/min length and latency budgets, so a verbose or slow regression fails the gate before a user ever feels it.
A case that passed in the committed baseline and fails now — the headline signal. It fails the build (exit 2) so it can't ship quietly.
Cases that newly pass (fixes) and outputs that changed without flipping pass/fail (drift), so nothing in the suite changes silently.
Two prompts or two models over the same suite, with the winner on weighted pass rate and exactly which cases differ between them.
A bring-your-own judge for subjective checks (tone, helpfulness) that augments — but never gates — the deterministic core.
Two engines, fully tested, at parity.
Checks, cases, runner, baseline diff, scoring, A/B compare, and a CLI. No third-party packages; runs in air-gapped CI.
The same checks, scoring, baseline diff, and verdict, matched to the Python behavior — for JS/TS stacks DeepEval can't serve.
contains / regex / equals / one_of / json_valid / json_field / length / latency — plus an optional BYO LLM judge that never gates the core.
Snapshot to a committed JSON file; every run reports regressions, fixes, output drift, and new or removed cases.
A GitHub Actions file and ship=0 / hold=1 / regressed=2 exit codes that drop straight into any pipeline.
Console and HTML reports, a guide to writing checks that catch real breakage, and an honest comparison to promptfoo / DeepEval / Braintrust.
Four steps, then it runs itself.
List the inputs you care about and the deterministic checks each output must pass. Author it in code or JSON.
Run once against your model and commit baseline.json to git — your golden record of what currently passes.
On every change, the kit reruns the suite and diffs against the baseline: regressions, fixes, and drift.
A regression fails the build (exit 2). Update the baseline deliberately when a change is a genuine improvement.
What this is — and isn’t.
This is a pre-ship regression gate, not an observability platform: no tracing, dashboards, run history, or production monitoring — pair it with Braintrust or LangSmith for those. It ships a focused, deterministic check library, not fifty scored metrics; for G-Eval-style grading, pair it with DeepEval or RAGAS. It doesn’t call a model for you — you wire your own target, which is what makes it model-agnostic.
And a SHIP verdict is only as good as the cases you write: it means “nothing regressed and you cleared your bar,” not “this prompt is objectively good.” The kit is small enough to read end to end — which matters for a tool that decides what ships.
Anyone shipping LLM features on a schedule.
Engineers and founders who put prompts or models into production and can’t afford silent breakage on the next tweak, teams running mixed Python/TypeScript stacks who want one consistent gate, and consultants who want to leave clients with a real CI guardrail instead of a spreadsheet of vibes. It’s the third gate in the dev-tools trio: retrieval, security, and now functional regressions.
RAG Retrieval Grader
$89The retrieval gate. Score Recall@k, MRR, and nDCG against a gold set and fail CI when retrieval drops. The Grader proves the context reaches the model; the Lab proves the prompt on top of it doesn't regress.
Prompt Injection Red Team Kit
$99The security gate. Run OWASP-mapped injection and prompt-leak probes against your app. Same shape as the Lab — deterministic, dual-engine, CI exit codes — for the security dimension instead of functional regressions.
Prompt Evaluation & Versioning System
$49The broader practice: a drop-in eval framework, multi-model benchmarking, quality scoring, and a Notion war-room. Use it to run your eval practice; drop in the Lab as the tight, deterministic pre-ship gate. Many teams want both.
The questions engineers actually ask before wiring it in.
It runs a suite of test cases against your LLM app, snapshots the pass/fail results to a baseline you commit to git, and on every later run diffs against that baseline. A case that passed before and fails now is a regression — and the kit fails your CI build (exit code 2) so it can't ship silently. It's regression testing for prompts and models, not open-ended 'is this good' scoring.
Those are great and this is complementary — a common pattern is to run a lightweight CI gate plus a heavier platform. This kit is the gate, built around one primitive the others under-serve: baseline diffing for regressions. It's deterministic, dependency-free, runs offline, and ships in both Python and TypeScript (DeepEval is Python-only). For 50+ scored metrics use DeepEval/RAGAS; for dashboards and production tracing use Braintrust/LangSmith. See COMPARISON.md in the kit.
They're complementary RedHub dev tools at different scopes. The Prompt Evaluation & Versioning System ($49) is the broad practice — a drop-in eval framework, multi-model benchmarking across Claude/GPT/Gemini, quality scoring, versioning conventions, a dashboard, and a Notion war-room — the patterns and templates to build evaluation inside your own repo. The Prompt Regression Lab ($89) is one focused primitive done deeply: deterministic baseline-diff regression gating with a hard CI exit code, in Python and TypeScript with matched behavior, small enough to read end to end. Use the Eval System to benchmark models and run your eval practice; drop in the Regression Lab as the tight pre-ship gate that fails the build the moment a change regresses what worked. Many teams want both.
No. The core checks are deterministic — substring, regex, JSON validity, field equality, length, latency, and so on — so the gate runs free, offline, and reproducibly. An optional bring-your-own LLM judge handles subjective checks (tone, helpfulness) but is never required and never blocks the deterministic core.
That there were no regressions versus your baseline and the weighted pass rate cleared your bar (default 90%). It does not mean your prompt is 'good' in the abstract — the verdict is only as strong as the cases you write. The kit includes a guide to writing checks that catch real breakage instead of noise.
Python (zero dependencies) and TypeScript (dependency-free ESM), with matched behavior across both — the same checks, scoring, baseline diff, and verdict. It's model-agnostic: you wire your own target function, so it works with any provider or a local model. Both ship with a CLI, tests, and a GitHub Actions workflow.
Deliberately, with a single flag (--update-baseline), and you commit the new baseline.json in the same pull request as the prompt change. That keeps improvements from silently masking regressions and lets reviewers see exactly what behavior changed.
Yes. A/B compare runs two versions over the same suite and reports which wins on weighted pass rate, plus exactly which cases regressed or improved in version B versus version A. It's the fast way to answer 'is my new prompt actually better?'
Ship prompts like you ship code.
A baseline, a diff, and a hard CI gate — in Python and TypeScript, zero dependencies. One-time $89, yours to keep.
Sold by RedHub AI LLC · Secured by Stripe · redhub.ai