Treat your prompts like code.Without buying another platform.
The Prompt Evaluation & Versioning System is a drop-in framework for dev teams shipping AI features — a code repo, a dashboard template, and a Notion war-room for tracking prompt performance, running regression tests when models change, versioning prompts like code, and benchmarking cost-vs-quality across Claude, GPT, and Gemini. Runs in your repo with your API keys. No telemetry, no cloud, no vendor lock-in.
Every team shipping AI features hits this wall by month three.
Prompt strings drift in your codebase with no version history beyond git. Eval criteria live in three different Notion docs and a stale Slack thread. The PM has opinions about what “good” looks like. Nobody has numbers.
Then a model update ships. Your accuracy quietly drops 12% on a specific customer segment. Nobody notices until support tickets start landing two weeks later.
You can build the eval pipeline yourself. Most teams do — eventually. The kit is what saves the team a focused week and three months of arguing about conventions.
- 01Prompts edited in PRs with no eval impact assessment
- 02Model upgrade ships. Customers report regressions before you do.
- 03PM keeps asking "can we switch to a cheaper model?" — no one has data
- 04Three different prompt versions running in three different services
- 05No rollback path when a prompt change breaks production
- 06Eval criteria reduce to "looks good to me" reviews in Slack
Six deliverables. Engineered to ship together.
The Eval Framework Repo
TypeScript and Python eval harness with adapters for OpenAI, Anthropic, and Google AI SDKs. Golden dataset runner, pluggable scorers, CI integration. Drops into any existing project.
The Dashboard Template
Next.js dashboard for visualizing eval runs over time — accuracy trends, cost-per-1K, latency percentiles, regression alerts. Deploy to Vercel or run locally. Your data, your infra.
The Notion War-Room
Prompt-change RFC template, model migration decision doc, eval criteria scorecard, regression incident postmortem. Where non-engineers participate in prompt governance.
The Versioning Convention
Semantic versioning spec for prompts (major / minor / patch), changelog format, deprecation policy, rollback playbook. The conventions your team would otherwise argue about for three sprints.
The Benchmark Scaffolds
Pre-built benchmark templates for classification, extraction, summarization, generation, and tool-use tasks. Point them at your prompt and golden dataset. Get cost-vs-quality numbers across providers.
The Cost Models
Pricing tables for Claude (Opus / Sonnet / Haiku), GPT (4o / 4o-mini / o1), Gemini (Pro / Flash). Per-token costs surfaced in every eval run. Updated as providers change pricing.
export default definePromptConfig({
prompts: {
customer_intent_classifier: {
version: '2.4.2',
provider: 'anthropic',
model: 'claude-sonnet-4-6',
template: './prompts/customer_intent.md',
eval: {
dataset: './golden/intent_v3.json',
scorer: 'llm-judge',
thresholds: { accuracy: 0.92, hallucinationRate: 0.02 },
},
benchmark: ['anthropic/claude-sonnet-4-6', 'openai/gpt-4o', 'google/gemini-pro'],
},
},
})Each one engineered for ship-day reliability. Each one repeatable.
Regression Testing on Every Prompt Change
Run golden dataset evals on every prompt diff. Fail CI when accuracy, F1, or hallucination rate crosses your threshold. Catch the silent regressions before they ship to production.
Model Migration Decisions
Compare your current model against a candidate — GPT-4o vs Sonnet 4.6, Haiku 4.5 vs Flash, Opus vs o1 — on cost, quality, and latency across your real workloads. Replace vibes-based migration calls with numbers.
Prompt Versioning Like Code
Semantic versioning convention (major / minor / patch), changelog structure, deprecation tracking, rollback patterns. Every prompt change has a version, a diff, an owner, and an audit trail.
Cost-vs-Quality Benchmarking
Pre-built scaffolds for classification, extraction, summarization, generation, and tool-use workloads. See exactly what you're trading when you move from premium to budget models — across providers.
Real failure mode. Model provider ships a quiet update. Accuracy craters. The kit blocks the ship.
The kit ships the runner, the diff format, the threshold logic, the war-room link generation, and the CI exit codes. Drop into your pipeline. Block bad ships. Open the war-room ticket for the team to triage.
Claude. GPT. Gemini. Plus your stack.
First-class adapters for the three major providers. SDK compatibility with TypeScript and Python projects. CI integration with the runners your team already uses.
Anthropic
Native message format, tool-use eval, cost models updated per release.
OpenAI
Chat completions, function calling, structured outputs, reasoning model latency tracking.
Generative AI SDK, system instructions, function declarations, safety setting comparison.
Your eval pipeline shouldn’t ship with vendor lock-in.
Every other tool in this category is a platform — you send them your prompts, your eval data, sometimes your customer inputs. You commit to their stack. You pay per seat or per eval. You hope they don’t pivot pricing or sunset features your workflow depends on.
This kit is the opposite trade-off on purpose. The patterns, the conventions, the runner, the dashboard, the war-room — all of it lives in your repo. The kit is a one-time $49 you fork, extend, and own.
The trade: you set up the pipeline yourself instead of clicking a button on someone’s dashboard. The reward: sovereignty.
Your repo, your infra
Eval framework runs where your code already runs. No new service, no third-party cloud, no extra deployment.
Your API keys, your bills
Provider calls go directly through your OpenAI, Anthropic, and Google accounts. No reseller markup. No surprise per-eval charges.
Your conventions, your team
Fork the kit. Extend it. Build your own scorers. The kit is patterns to own, not workflows to comply with.
Be honest with yourself before you click buy.
- You ship AI features in production and prompts are load-bearing.
- You have 10+ prompt strings drifting in your codebase.
- You’ve already had at least one silent regression incident.
- You’re a senior engineer, eng manager, or technical founder.
- You want patterns to own, not a platform to depend on.
- You have one prompt in your codebase. You don’t need this yet.
- You want a fully managed platform with a UI for everything.
- You don’t have anyone on the team who reads TypeScript or Python.
- You’re not shipping AI features to real users yet.
- You want vendor support — this is a kit, not a service.
Eval the prompts, then guard the whole pipeline.
Evaluation is one layer; the rest of the reliability stack covers the others. The Prompt Regression Lab catches when a prompt change quietly breaks, the Agent Reliability Harness tests the agent that runs the prompts, and the RAG Retrieval Grader grades the retrieval feeding them.
The questions engineers actually ask before pulling out the company card.
No, and that's deliberate. Those are platforms — you commit to their stack and send them your eval data. This kit ships the patterns, conventions, and templates to build prompt evaluation inside your own repo. If you already use a platform, the kit still gives you better eval criteria, regression patterns, and Notion ops templates. If you don't, the kit lets you skip the platform decision until you actually need to scale.
The eval framework ships in TypeScript and Python with adapters for the OpenAI, Anthropic, and Google AI SDKs. Drops into any project using those — Next.js, Express, FastAPI, Django, Lambda, plain Node scripts. The dashboard template is a Next.js app you can deploy to Vercel or run locally. The Notion war-room imports directly into any Notion workspace.
The repo includes benchmark scaffolds for classification, extraction, summarization, generation, and tool-use tasks. Point it at your prompt and golden dataset, then run it across Claude (Opus, Sonnet, Haiku), GPT (4o, 4o-mini, o1), and Gemini (Pro, Flash). Outputs cost-per-1K-evals, p50 and p95 latency, and pluggable quality scoring — LLM judge, embedding similarity, exact match, or your own scorer.
Not different in kind — different in time. The repo is what your senior engineer would architect on a focused week. The conventions are what your team would argue about in code review for three sprints. The Notion war-room is what your eng manager wishes they'd set up before the first prompt regression incident. You can build this yourself. The kit means you don't have to.
Usually a senior engineer, AI/ML engineer, eng manager, or technical founder. The dashboard and code repo need someone who reads TypeScript or Python. The Notion war-room is built for cross-functional reviewers — product managers, content people, ops — to participate in prompt-change governance without writing code.
That's exactly what the kit is built for — catching the moment a model update silently changes prompt behavior. RedHub AI updates the kit's model registry and benchmark scaffolds when major versions ship. Existing buyers get lifetime updates included in the $49.
No. The kit runs in your repo, on your infra, with your API keys. No telemetry, no cloud component, no data ever leaves your stack. That's the whole 'kit, not platform' point — you keep sovereignty over your eval pipeline.
30-day no-questions refund. If the eval framework, dashboard, and Notion war-room don't replace the week your team would otherwise burn building this from scratch, you shouldn't have paid. Email RedHub AI support and the $49 comes back.
Stop shipping prompts on vibes.
Ship them on numbers.
$49 once. Drop the kit into your repo. Get regression tests, model migration data, prompt versioning, and Claude/GPT/Gemini benchmarks by end of week.
Sold by RedHub AI LLC · Secured by Stripe · redhub.ai