Your RAG isn’t hallucinating.It’s retrieving the wrong chunks.
When a RAG answer is wrong, the blame lands on the model — but usually the model never got the right context. The retriever ranked the answer at position 9, or missed it entirely, and the model guessed.
The RAG Retrieval Grader scores the retrieval step on its own — Recall@k, Precision@k, MRR, nDCG — then tells you to ship, hold, or fix, lists every query where the answer never reached the model, and gates it in CI.
Broken retrieval looks exactly like a working app.
When a RAG answer is wrong, the blame lands on the model. But most of the time the model did its job — it just never got the right context. The retriever ranked the answer at position 9, or missed it entirely, and the model filled the gap with a confident guess.
You can’t fix what you can’t see. Without isolating retrieval, every fix is a guess — re-prompt, re-chunk, swap the embedding model, raise the temperature, and hope. The Grader replaces the guessing with a number.
The right chunk exists, but retrieval buried it below the cutoff. The model never sees it, so it answers from the noise it did get — confidently, and wrong.
Sometimes the answer simply never surfaces. The query looks handled, the response looks fluent, and the failure is invisible until a user hits it in production.
An embedding swap or a chunking tweak quietly drops recall. Nothing errors, nothing alerts — the bar just lowers, and no one notices until trust erodes.
Grade a sample eval set right here.
Six real support questions, scored live with the exact formulas that ship in the kit — verified against both the Python and TypeScript engines. Two of them are quietly failing. Flip a chunk’s relevance and watch the scores — and the verdict — move.
Live grader · sample eval set
6 queries · customer-support knowledge base
Verdict: HOLD — FIX FIRST
Retrieval is leaking. Some answers never reach the model. Fix the misses below before shipping.
Threshold: Recall@5 ≥ 90% ship · 75–90% hold · <75% block. Tunable in the kit.
Recall@5
83%
Of the right answers, how many were retrieved
Hit Rate@5
83%
Queries with at least one right chunk
MRR
0.51
How high the first right chunk ranks
nDCG@5
0.57
Ranking quality, weighted by position
Precision@5
20%
Share of retrieved chunks that are relevant
Per-query results · tap a chunk to flip its relevance and watch the scores move
How do I reset my password?
first hit @ rank 1What is your refund policy?
first hit @ rank 1Do you offer annual billing?
first hit @ rank 3How do I export my data?
miss⚠ 1 relevant chunk exists in the KB but was never retrieved — the model never sees the answer.
Can I add team members?
first hit @ rank 2Is there an API rate limit?
first hit @ rank 5This demo scores 6 sample queries and resets on reload.
The full kit grades your entire eval set against your live retriever — Pinecone, Weaviate, pgvector, Chroma, Qdrant, LangChain, or LlamaIndex — and fails the build in CI the moment recall drops. Ship retrieval you can prove.
Get the RAG Retrieval Grader — $89Five metrics, in plain English.
Of the chunks that should have been retrieved, how many showed up in the top k. The headline metric — low recall means the answer never reached the model.
The share of queries where at least one correct chunk appeared in the top k. A blunt but honest “did we get anything right” check.
Mean Reciprocal Rank — how high up the first correct chunk lands, on average. Rewards putting the answer at the top, not buried at position 8.
Ranking quality weighted by position. Captures whether your best chunks sit above your weaker ones, not just whether they appear.
The share of retrieved chunks that are actually relevant. Surfaces noisy retrieval that floods the context window with junk.
Everything to grade and gate retrieval.
Recall@k, Precision@k, MRR, nDCG@k, Hit Rate — Python + TypeScript, fully tested (identical scores across both).
Pinecone, Weaviate, Qdrant, Chroma, pgvector, LangChain, LlamaIndex + a template for any custom retriever.
A labeling template, workflow, and a bootstrapper that drafts a starter eval set from your docs.
Run a full eval from the terminal and print a scored report — with the verdict — in seconds.
A drop-in GitHub Actions job that fails the build when retrieval drops below your threshold.
An HTML report generator with the ship/hold/fix verdict and a ranked list of misses to share with your team.
Four steps to provable retrieval.
Pair real questions with the chunks that should answer them. Use the template, or bootstrap a starter set from your existing documents.
Point an adapter at your vector DB or framework retriever. One small function for custom stacks.
Score the whole set from the CLI. Get Recall@k, MRR, nDCG, and a ship/hold/fix verdict with every miss listed.
Set your minimum recall and add the GitHub Action. Now no change can quietly degrade retrieval without failing the build.
Next to the eval frameworks — complementary, not a replacement.
Most RAG-eval tools score quality with an LLM acting as judge — strong for generation quality like faithfulness, but for retrieval it adds cost, rate limits, and run-to-run variance, and a judge can’t reliably tell a wrong-but-plausible chunk from the right one. The Grader answers a narrower, harder-to-fake question with exact arithmetic against a gold set: did the chunk that actually answers this question show up, and how high did it rank?
| Tool | Approach | Ground truth | Retrieval gate / verdict | Languages | Delivery |
|---|---|---|---|---|---|
| RAG Retrieval Grader | Deterministic IR metrics | Required (bootstrapper incl.) | Built-in ship/hold/fix + CI gate | Python + TypeScript | Buy-once kit · offline · no telemetry |
| RAGAS | LLM-as-judge (reference-free) | Optional | Metrics; gating is DIY | Python | Open-source library |
| DeepEval | LLM-as-judge · pytest-native | Optional | Threshold gates in CI | Python | Open-source (+ hosted platform) |
| TruLens | LLM-as-judge (RAG triad) + tracing | Optional | Metrics + tracing; gating is DIY | Python | Open-source |
| Arize Phoenix | LLM-as-judge + observability | Optional | Dashboards + evals | Python | Open-source · self-hostable (+ platform) |
| LangSmith | Tracing + evaluators (LLM/custom) | Optional (datasets) | Eval runs + dashboards | Python + JS/TS | SaaS (free + paid tiers) |
It’s not a replacement — it’s the layer they leave to you.
Use an LLM-judge framework for faithfulness and groundedness. Use the Grader to prove retrieval deterministically and gate it in CI — no judge tokens, no variance, reproducible on every run. Most teams need both layers.
Comparison reflects each tool’s general positioning as of June 2026; the space moves quickly, so verify current capabilities on each project’s docs. Names belong to their respective owners.
Built for teams shipping RAG to real users.
Founders and dev teams running a support bot, internal knowledge assistant, docs search, or any retrieval-backed feature — anyone who needs to answer “is our retrieval good enough to ship?” with evidence instead of vibes. Exactly what you get for $89, and what you don’t: a deterministic retrieval grader — not an LLM-judge framework for faithfulness (use both).
AutoChunk API
EnterpriseThe other half of RAG quality: permission-aware chunking with ACL metadata on every chunk. Chunk your corpus there, then prove the chunks actually surface with the Grader. Chunk there, grade here.
Prompt Evaluation & Versioning System
$49Once retrieval is provable, version and regression-test the prompts on top of it. The Grader gates the context; the Eval System gates the generation — together they keep the whole pipeline honest in CI.
The questions engineers actually ask before wiring it in.
It scores the retrieval step of your RAG pipeline using standard information-retrieval metrics: Recall@k, Precision@k, Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (nDCG@k), and Hit Rate. It then returns a ship / hold / fix verdict against tunable thresholds and lists every query where retrieval missed.
End-to-end evals grade the final generated answer, which blends retrieval quality and model quality together. When an answer is wrong you cannot tell whether retrieval failed or the model did. The Grader isolates retrieval, so you know whether the right context ever reached the model in the first place.
The kit ships with adapters for Pinecone, Weaviate, Qdrant, Chroma, and pgvector, plus retriever adapters for LangChain and LlamaIndex. Any retriever that returns ranked chunks for a query can be wired in with a small adapter function included as a template.
You need a gold set: real questions paired with the chunks that should be retrieved to answer them. The kit includes a gold-set template, a labeling workflow, and a helper that bootstraps a starter set from your existing documents so you are not labeling from scratch.
Yes. The kit includes a CLI and a ready-to-use GitHub Actions workflow. Set a minimum Recall@k (and any other metric) and the build fails the moment a prompt change, embedding swap, or chunking tweak drops retrieval below your bar.
Both. The grader, metrics engine, adapters, and CLI are provided in Python and TypeScript so it drops into whichever stack your RAG service already uses. Both engines are test-covered and produce identical scores.
Those frameworks mostly score quality with an LLM as judge — strong for generation quality (faithfulness, groundedness) but, for retrieval, costly, rate-limited, and variable run to run. The Grader scores retrieval deterministically against a gold set, so results are exact, reproducible, free to run, and gateable in CI. It's complementary, not a replacement: pair it with an LLM-judge framework for faithfulness; use the Grader to prove retrieval without burning judge tokens.
An instant download: the metrics engine (Python + TypeScript), retriever adapters, a gold-set template and bootstrapper, the CLI, the GitHub Actions CI workflow, an HTML verdict report generator, and a setup guide. One-time purchase, yours to keep, free updates within the version.
Stop guessing. Grade your retrieval.
Score Recall@k, MRR, and nDCG, surface every miss, and gate it in CI — Python or TypeScript, any vector DB. One-time $89, yours to keep.
Sold by RedHub AI LLC · Secured by Stripe · redhub.ai