Jul 13, 2026

Diagnosing Your RAG Project Through Evals

So you have a running RAG project. Your retriever fetches some context chunks from the corpus, then your generator takes those chunks plus the user’s query and generates an answer. How can you tell whether the generated answer actually addresses the user’s query, and if it doesn’t, how do you diagnose what or where things went wrong?

We can diagnose it using evals. Evals (short for evaluations) help diagnose where in the pipeline an answer went wrong. Evals can answer whether something went wrong in the retrieval step or in the generation step. It can help you diagnose layers such as retrieval quality, answer grounding, answer quality and production performance. RAGAS (the one I used in my project) frames it as context recall, context precision, faithfulness, and response relevancy.

Here’s how I like to think about it: a RAG system is a student taking an open-book exam. The student gets a question, flips through the book to find the relevant pages (that’s your retriever), then writes an answer based on those pages (that’s your generator).

Evals are the teacher checking the student’s work. A good teacher doesn’t just mark the final answer right or wrong. She checks the process:

“Did the student find all the important pages?”
“Are the pages the student picked actually relevant, or did they grab useless ones too?”
“Did the student make stuff up?”
“Did the answer actually answer the question?”

Each of those questions maps to a metric.

Metrics

These are the indicators to observe whether there’s something wrong in your pipeline.

Context Recall

“Did the student find all the important pages?”

Context recall measures how much of the relevant documents (or pieces of information) were successfully retrieved. Recall is about not missing anything important. Ragas definition.

Context Precision

“Are the pages the student picked actually relevant?”

Context precision is a metric that evaluates the retriever’s ability to rank relevant chunks higher than irrelevant ones for a given query in the retrieved context. Ragas definition.

Faithfulness

“Did the student make stuff up?”

Faithfulness measures how factually consistent the generated answer is with the retrieved context. Ragas definition.

Response Relevancy

“Did the answer actually answer the question?”

Response relevancy measures how relevant a response is to the user’s query. Ragas definition.

Here’s the part that makes these four useful for diagnosis: Context Recall and Context Precision measure your retrieval pipeline, while Faithfulness and Response Relevancy measure your generation pipeline. By looking at the pattern across these metrics, you can narrow down where in your whole pipeline something probably went wrong.

For example, low recall plus low response relevancy usually points to retrieval: the model can’t answer well because it wasn’t given the right material. High recall plus low response relevancy points more to generation: the right material was there, but the model still didn’t answer the question directly. You can answer questions like “Is the model hallucinating?” (low faithfulness), “Did the retriever retrieve the correct chunks?” (low recall or precision), “Why did the model answer that using the retrieved contexts?”, and many more.

There are still more metrics that you can use with your RAG project but these are the four that I use. You can also add Latency which measures how long it takes for your RAG to generate an answer, though that’s more of a production monitoring metric than something RAGAS scores.

Judge

So who actually does the grading? Someone (or something) has to read the question, the retrieved pages, and the answer, then give the scores. That’s the judge.

Going back to the exam analogy, the judge is the teacher. The judge checks things like “Did the student use the right pages?”, “Did the answer match the pages?”, “Did the student make stuff up?” and “Did the answer actually answer the question?”.

Production-grade RAG projects ideally require human judges that are experts in the domain of your corpus (in my case, that would be lawyers) but you can also use LLMs-as-judge as I did to save some costs. You basically hire another AI to be the teacher. For sensitive domains like law, it’s still worth checking your LLM judge against a small set graded by actual humans, so you know if you can trust its scores.

How?

Before evaluating our RAG project, we’ll need an answer sheet (AKA gold test dataset) which contains questions and answers curated from your corpus. The correct answers are called ground truths.

Note what’s not in the gold test dataset: the retrieved contexts and the generated answer. Those are the student’s work, not the answer key. They come from actually running your RAG against each question. After the run, you save them into your evaluation results together with the original question and ground truth. If you wrote the student’s work into the answer key yourself, there’d be nothing left to evaluate.

In my project, I’m using Philippine legal documents as a corpus, so each entry starts with a question and its ground truth like this:

{
	"question": "What is online libel?",
	"ground_truth": "Under Cybercrime Prevention Act of 2012, Chapter II, Section 4(c)(4), libel committed through a computer system or any other similar means which may be devised in the future is unlawful or prohibited acts of libel"
}

If that question looks familiar, it’s the same one that once tripped my pipeline because of a badly shaped chunk (I wrote about that in The Text Existed. The Chunk Was Wrong.). That failure is exactly why it earned a spot in my answer sheet: once a question burns you, you keep it around so you’ll notice if it ever breaks again.

We give the question to our RAG and capture what it produced:

{
	"retrieved_contexts": [
		"The Revised Penal Code (Act No. 3815) Article 353. Definition of libel.-A libel is a public and malicious imputation of a crime, or of a vice or defect, real or imaginary, or any act, omission, condition, status, or circumstance...",

		"Cybercrime Prevention Act of 2012 (Republic Act No. 10175) CHAPTER II — Section 4(c)(4) (4) Libel. The unlawful or prohibited acts of libel as defined in Article 355 of the Revised Penal Code, as amended, committed through a...",

		...
	],
	"answer": "Online libel is libel committed through a computer system, punishable under Section 4(c)(4) of the Cybercrime Prevention Act of 2012..."
}

Then all four pieces (question, ground truth, retrieved contexts, and the generated answer) are passed to RAGAS to evaluate the metrics. Ideally, you’ll need a lot of data in your dataset to check what part of your RAG needs improvement, rather than judging from one lucky or unlucky question.

Let’s see how the judge grades this exact example.

Once passed into RAGAS, we’ll get results like this:

faithfulness	answer relevancy	context precision	context recall
1.00	0.94	0.83	1.00

Based on the scores RAGAS gave, we can deduce that we have a healthy pipeline with slightly noisy retrieval.

Let’s walk through it. Faithfulness is 1.00, so every claim in the answer is backed by the retrieved contexts. The student didn’t make stuff up. Response relevancy is 0.94, so the answer addresses the question head-on. Context recall is 1.00, so nothing important from the ground truth was missed. Three good scores.

Then there’s context precision at 0.83. Context precision is a retrieval metric, so the generator is probably not the main issue here. The retriever found useful material, but some less useful chunks may have been mixed in or ranked higher than they should have been. In the exam analogy, the student found the right pages and answered honestly, but she also photocopied a few pages she never needed.

So what do you do about it?

This is the whole point of evals: each bad score gives you a better place to start looking.

Low context recall. The retrieved context doesn’t support enough of the important claims in the ground truth. Look at your chunking strategy, try a different embedding model, or increase top-k so the retriever casts a wider net.
Low context precision. The right chunks are there but junk is ranked too high. Add a reranker or tighten your top-k.
Low faithfulness. The generator is making stuff up. Work on your prompt (something like “only answer from the given context”), or use a better model.
Low response relevancy. The answer wanders off the question. Again, prompt work, or check whether the question even has an answer in your corpus in the first place.

For the 0.83 context precision above, the fix lives on the retrieval side: tighten your top-k or tune the reranker so any off-topic chunks get pushed down before they ever reach the generator.

Abstention

Since my corpus is legal documents, abstention is important. Sometimes the best answer your RAG can give isn’t an answer, but a refusal to answer: “I don’t know based on the provided documents.”

Here’s a real one from my project. I asked my RAG an annulment question, and it answered confidently, complete with grounds and a citation of an “Article 87”. The problem? That Article 87 wasn’t in my corpus. The model pulled it from its training memory and dressed it up as if it came from my documents. The answer read like legal information, but it was a hallucination. In exam terms, the student ran out of pages, so she answered from memory and pretended it was in the book.

That shows up as a faithfulness failure, and evals caught it: the judge checked the answer against the retrieved contexts and the citation simply wasn’t there. But the real product failure is abstention. The system answered when it should have stopped.

This matters because a legal RAG shouldn’t confidently answer questions that the corpus can’t actually support. If the retriever didn’t find the right legal text, or if the answer isn’t in the corpus at all, forcing the model to answer is worse than a normal wrong answer. The user might treat the response as legal information even though the system had no evidence for it. Going back to the exam analogy, sometimes the honest move is for the student to write “it’s not in the book” instead of bluffing.

So evals shouldn’t only check whether the system answers well when the answer exists. They should also check whether the system knows when to stop. Your dataset should include unanswerable questions, out-of-scope questions, and questions where the retrieved context is too weak. For those, the right behavior is to abstain instead of giving a polished but unsupported answer.

Wrapping up

Without evals, improving a RAG project is guesswork. You read a bad answer, shrug, and start turning random knobs like swapping the model or rewriting the prompt, hoping something sticks. With evals, the scores narrow down which part of the student’s process needs coaching. Build your answer sheet, hire a judge, and let the grades tell you where to look.