# Find Whether a RAG Answer Failed in Retrieval or Generation

> A wrong RAG answer can come from missing evidence, a zero-overlap query, a chunk ranked below k, a context budget that cut it, or a model that ignored it, and one end-to-end score cannot tell which. This part saves a per-case trace, applies six ordered checks to eight cases on an in-memory BM25 retriever with a deterministic stub generator, and shows why recall@k, faithfulness and a larger k each mislead on their own. It ends with a per-failure decision table and a gold-only probe that separates a generation failure from a retrieval one.

- Author: [Shashank Rawlani](https://shashank.rawlani.com)

- Published: 2026-11-07T13:30:00.000Z

- Updated: 2026-11-07T13:30:00.000Z

- Category: AI in Testing

- Canonical source: https://automationtester.in/blog/ai-in-testing/debug-rag-retrieval-vs-generation-failures

- Tags: rag, llm-evaluation, retrieval, recall-at-k, ragas, error-analysis, ai-testing

**Quick answer:** Localise a wrong RAG answer by walking one saved trace through ordered checks, where each check is reached only if the one above it cleared: is the required chunk in the index, does it score at all, does it rank inside top-k, does it survive context assembly, and only then did the generator use it. Label the required evidence by chunk id first. Count a correct refusal as a pass. Re-label after every fix, because repairing one stage often exposes the next.

A policy assistant tells an employee that flights are economy class only. The employee asks about a 10-hour flight, and the policy has a paragraph that allows business class above 8 hours with a VP's written approval. The assistant never mentions it. The team spends a week rewording the system prompt to "check for exceptions", the score does not move, and nobody opens the saved prompt to see that the exception paragraph was never in it.

A single end-to-end score cannot say which of those two worlds you are in, and the fix for each is different. This article builds a small trace that can, runs it on eight cases, and ends with a decision table you can point at your own failures. Labelling the evidence comes from [the dataset part of this series](/blog/ai-in-testing/build-llm-eval-dataset-product-risks), and trusting whoever applies the labels is the subject of [the judge calibration part](/blog/ai-in-testing/calibrate-llm-as-judge-scores). Whether the exception should have been its own chunk is a chunking question, and the later chunking part of the series owns it.

**What produced the output below, checked 2026-11-02:** Node.js v26.12.1 ran every listing. No model and no API was called. The retriever is a BM25 scorer written for this lab, the corpus is 14 hand-split chunks of a fictional travel policy, and the generator is a deterministic stub that copies one sentence out of the prompt. The questions were written to trigger each failure, so the counts show how the labels behave, not how often real systems fail. Ragas 0.4.3 (the latest on PyPI, uploaded 2026-01-13) is discussed from its documentation and was not run.

## What must one trace hold before a wrong answer can be localised?

Five things, saved together per case: the query, the full ranked list with scores, the chunk ids that actually reached the prompt, the answer with its citations, and the gold evidence, meaning the chunk ids a human marked as required to answer. Without the gold ids you can only say an answer looks wrong. With them, every later question is a lookup.

The ranked list has to be the full ranking, not the top k. A chunk at rank 3 and a chunk with score 0 look identical in a top-2 printout, and they need different repairs. The prompt list has to be what was sent, not what was retrieved, because the stage between the two can drop evidence.

The lab keeps that stage deliberately small. It takes ranked chunks in order until a word budget runs out, and it stops at the first chunk that does not fit:

```
export function assemble(ranked, byId, k, wordBudget) {
  const ctx = [];
  let used = 0;
  for (const r of ranked.slice(0, k)) {
    const text = byId.get(r.id).text;
    const w = text.split(/\s+/).length;
    if (used + w > wordBudget) break;
    ctx.push(r.id);
    used += w;
  }
  return ctx;
}
```

The stub generator answers from the first prompt chunk that shares at least two content words with the question. That anchoring habit is built in so a generation failure can be reproduced on demand. It is a property of this stub, not evidence about any model:

```
export function generate(question, ctxIds, byId, { anchored = true } = {}) {
  const q = new Set(tokens(question));
  const hits = ctxIds.filter((id) => tokens(byId.get(id).text).filter((t) => q.has(t)).length >= 2);
  if (!hits.length) return { answer: REFUSAL, citations: [] };
  const pick = anchored ? hits[0] : hits.find((id) => /^Exception/.test(byId.get(id).text)) ?? hits[0];
  const overlap = (s) => tokens(s).filter((t) => q.has(t)).length;
  const sentences = byId.get(pick).text.split(/(?<=\.)\s/);
  const best = sentences.reduce((a, s) => (overlap(s) > overlap(a) ? s : a));
  return { answer: best, citations: [pick] };
}
```

## In what order do you ask which stage lost the evidence?

Top to bottom, stopping at the first rule that explains the failure. The order is the whole method: a rule below is only meaningful if the ones above have been ruled out.

- No gold evidence exists and the answer is not a refusal. The system answered something it should have declined.

- A gold chunk id is missing from the index. Retrieval never had a chance.

- A gold chunk scores 0 for the query. The query and the chunk share no terms the retriever can see.

- A gold chunk has a score but ranks beyond k. It was found and then cut by the cutoff.

- A gold chunk is inside top-k but absent from the prompt. Something between retrieval and the model dropped it.

- Every gold chunk is in the prompt and the answer is still wrong. Only now is it a generation failure.

The same order, as the labeler:

```
export function label(t, k) {
  if (t.c.expectRefusal) return t.correct ? "pass: correct refusal" : "generation: answered an unanswerable question";
  if (t.goldRanks.some((g) => !g.inIndex)) return "corpus: gold chunk is not in the index";
  if (t.goldRanks.some((g) => g.score === 0)) return "retrieval: no query/chunk term overlap (score 0)";
  if (t.goldRanks.some((g) => g.rank > k)) return `retrieval: gold ranked ${Math.max(...t.goldRanks.map((g) => g.rank))}, outside top-${k}`;
  if (t.goldRanks.some((g) => !t.ctx.includes(g.id))) return "assembly: gold retrieved but cut by the context budget";
  if (!t.correct) return "generation: gold was in the prompt, answer ignored it";
  return "pass";
}
```

Here are two of the eight traces, with k=2 and a 50-word budget. The first is the opening story. The exception chunk `c08` ranks third, so it never reaches the prompt:

```
long-haul-cabin
  query    : Which class must employees book for flights, and is there an exception for long flights?
  ranked   : c01(8.6) c02(5.5) c08(3.6) c03(0.0) ...
  gold     : c08@3
  in prompt: c01 c02
  answer   : Employees book economy class fares for all flights.  cites c01
  label    : retrieval: gold ranked 3, outside top-2
```

The second looks similar from the outside, since the answer is the same economy-class sentence. But here the exception is in the prompt:

```
business-10h
  query    : Can I book business class on a 10-hour flight?
  ranked   : c01(3.2) c08(2.9) c13(1.8) c02(1.8) ...
  gold     : c08@2
  in prompt: c01 c08
  answer   : Employees book economy class fares for all flights.  cites c01
  label    : generation: gold was in the prompt, answer ignored it
```

Rewording the prompt is a plausible fix for the second trace and a useless one for the first. That is the difference a single score hides.

## What does each label tell you to change?

Every label the run produced except a plain pass, with the check that separates each from its neighbours and the change worth trying. The last column is the change teams tend to make instead.

| Label | Trace signature | Discriminating check | Try first | Wasted effort |
| --- | --- | --- | --- | --- |
| Corpus gap (`mileage-rate`) | Gold id absent from the index; system refuses or answers from a neighbour | Look the gold id up in the index, not the retriever output | Ingest the missing section, then re-run the case | Tuning ranking for a chunk that does not exist |
| Vocabulary miss (`lie-flat-seat`) | Gold chunk scores exactly 0; rank is near the bottom | Print the gold chunk's score from the full ranking | Rewrite or expand the query; add terms users actually say; try a retriever that matches meaning | Raising k, which cannot surface a zero score |
| Rank miss (`long-haul-cabin`) | Gold has a positive score at rank 3 with k=2 | Compute recall at the next k and compare | Reranking, or a larger k together with a larger budget | Prompt edits |
| Assembly cut (`taxi-receipt`) | Gold at ranks 1 and 2, prompt holds only one of them | Diff retrieved ids against prompt ids | Fix the budget or the packing rule | Blaming the model for ignoring text it never saw |
| Generation (`business-10h`) | All gold in the prompt; answer cites a different chunk | Gold-only prompt passes, original prompt fails | Order, prompt instructions, a model change, one variable at a time | Re-indexing, which changes nothing the model saw |
| Abstention failure (`zurich-hotel`) | No gold exists; the answer is a confident sentence from a related chunk | Check whether the expected answer is a refusal | Abstention instructions and a relevance floor | Counting it as a retrieval miss |
| Correct refusal (`gym-fee`) | No gold exists; the answer is a refusal | None needed | Leave it, and exclude it from recall | Treating it as a failure to fix |

The failure rows map onto the first four failure points in the experience report on engineering RAG systems by Barnett and colleagues: missing content, an answer that did not rank highly enough, a document that never made it into the context, and an answer present in the context that the model did not extract. Their wording for the missing-content case describes the zurich-hotel trap exactly: a question related to the content, with no answer in it, can fool the system into responding. The report describes case studies from their own deployments rather than measuring frequencies, so treat the seven points as a checklist and not as a prevalence table.

## What do aggregate retrieval scores hide?

The same run, summarised:

```
cases with gold evidence: 6 of 8
recall@1: 0.25
recall@2: 0.50
recall@3: 0.67
recall@5: 0.67
end-to-end correct: 2 of 8
answered (not refused): 6; toy-faithful (sentence copied from a prompt chunk): 6; correct: 1
```

Three readings. First, recall@2 of 0.50 is accurate and still misleading: `taxi-receipt` and `business-10h` each count as retrieved, and both failed downstream. Second, recall@3 and recall@5 plateau at 0.67 because two gold chunks, `c15` and `c08` for the lie-flat question, are out of reach of any k. The first is not indexed and the second has no matching terms. Third, the toy faithfulness check passes all six answered cases, because the stub only ever copies a prompt sentence, and one of the six is correct.

That last result is not a quirk of the stub. The Ragas paper defines faithfulness as whether the claims in the answer can be inferred from the retrieved context, so an answer grounded in the wrong context scores well. It defines context relevance as how focused the retrieved context is, which penalises redundant text rather than missing text, so neither dimension tells you whether the required paragraph was retrieved. The Ragas documentation separates the retrieval questions further: its LLM-based context recall estimates coverage by checking whether claims in a reference answer are supported by the retrieved context, and a non-LLM variant compares retrieved contexts to reference contexts. Neither is the gold-chunk recall computed above, and both depend on whatever reference you supply. Name which recall a dashboard reports before comparing two of them.

## Does raising k fix a retrieval miss?

For one of the two retrieval labels, sometimes, and the run shows what happens next. Re-running `long-haul-cabin` with larger settings:

```
k=2 budget=50: prompt c01,c02 -> retrieval: gold ranked 3, outside top-2
k=3 budget=50: prompt c01,c02 -> assembly: gold retrieved but cut by the context budget
k=3 budget=80: prompt c01,c02,c08 -> generation: gold was in the prompt, answer ignored it
k=14 budget=400: prompt c01,c02,c08,c03,c04,c05,c06,c07,c09,c10,c11,c12,c13,c14 -> generation: gold was in the prompt, answer ignored it
```

Raising k alone changed the label but not the answer, because the new chunk was then cut by the budget. Raising both put the exception in the prompt behind the general rule, and the anchored stub answered from the first chunk. Each change fixed one stage and revealed the next, which is why a fix needs a re-label and not a glance at the headline score.

For the remaining generation failure, two probes separate the causes. Run the generator on the gold chunks alone, then on the same prompt chunks with the gold moved first:

```
const ok = (ctx) => generate(t.c.question, ctx, byId).answer.includes(t.c.mustContain);
const promoted = [...t.c.gold, ...t.ctx.filter((id) => !t.c.gold.includes(id))];
// gold-only prompt:  ok(t.c.gold)
// gold moved first:  ok(promoted)
```

```
business-10h: gold-only prompt PASS | same chunks, gold first PASS | original order c01,c08 FAIL
```

Gold alone passes and the original order fails. The model, or here the stub, can read the exception when it is the only thing in front of it and prefers the general rule when the rule comes first. That points at ordering and competing text, so the next experiments are a reranker, instructions about exceptions, or a different model, changed one at a time. A gold-only prompt that also failed would point at the model or the prompt itself.

This oracle design comes from the "Lost in the Middle" study, which compared multi-document question answering against a closed-book setting with no documents and an oracle setting that gives the model only the single answer-bearing document. For GPT-3.5-Turbo its Table 1 reports 56.1% closed-book and 88.3% oracle. It also found accuracy highest when the relevant passage was at the start or end of the context and lower in the middle, and for GPT-3.5-Turbo the middle placement fell below that closed-book figure. Those are 2023 results on specific models and one task, so use them as a reason to run the reorder probe on your own model, not as a prediction of its result.

## Is a refusal a failure?

It depends on the gold label, which is why the label has to exist before the run. When no gold evidence exists, a refusal is the correct answer and the case drops out of the recall denominator: the lab computes recall over the six cases that have gold, not all eight. When gold does exist, a refusal is a failure of whichever stage lost the evidence. In the run, `lie-flat-seat` ended in a refusal and `mileage-rate` in an irrelevant sentence about personal travel. Both are wrong and carry different labels, because the check that separates them is a lookup in the index and a score, not the shape of the answer.

Whether the system declines gracefully, cites honestly and avoids invented detail is a larger topic that a later part of the series covers. Here the only requirement is that unanswerable cases are present in the set with an expected refusal, so an abstention failure has somewhere to be counted.

## Where can an AI grader apply these labels?

The mechanical rules above need no model: they are comparisons between ids, scores and positions. A model-based grader earns its place on the two judgments those rules cannot make, whether an answer is correct against the gold text and whether a refusal was warranted. Even then, the gold evidence has to come from a person reading the source, and the label categories have to be checked by reading a sample of mislabelled traces by hand. A grader that scores everything "generation" would look healthy on a dashboard and send the team to the wrong stage.

## Apply this now

Pick the ten to twenty wrong answers your team already argues about, and spend one sitting on them.

- For each, write the chunk ids a person says are required to answer. Write "none, expect refusal" where that is the truth.

- Save the query, the full ranked list with scores, the ids that reached the prompt, the answer and the citations for each case. If your pipeline cannot export the prompt ids, add that before anything else.

- Apply the six ordered rules by hand to the first five, then script them.

- Pick the largest label bucket and change one thing for that stage only.

- Re-run and re-label every case. Keep the trace from before and after in the pull request.

The evidence to capture is a table with one row per case, its before and after label, and the gold-only probe result for every case labelled generation. Saving traces as JSON lets the [JSON Diff tool](/tools/json-diff) show exactly which case moved between runs.

## FAQ

### Can I localise failures without labelled gold chunks?

Not reliably. Reference-free metrics exist and the Ragas paper was written around them, but they score the answer against whatever was retrieved, which is the blind spot shown above. Without gold ids a trace shows what the prompt held, not whether the needed evidence was ever available.

### Does the score-0 rule work with an embedding retriever?

No. A dense retriever returns a similarity for every chunk, so the vocabulary-miss rule will not fire. The lab ran only BM25. With embeddings, substitute a floor on the gold chunk's similarity that you calibrate on your own data, or fall back to its rank in the full ranking and inspect the lowest-ranked gold chunks by hand.

### Should I just adopt Ragas?

Its metric definitions are worth reading because they say which question each number answers. Running them calls a model, which brings the calibration problem from the previous part back in. The trace in this article needs no model for retrieval, assembly or corpus labels, so it can run on every commit while the model-graded metrics run less often.

### Is the stub's anchoring a realistic model behaviour?

It is a simplification chosen so the failure reproduces on demand. The "Lost in the Middle" results show position sensitivity in the models it tested, but whether yours behaves this way is exactly what the reorder probe is for.

## Primary references

- [Ragas paper on reference-free RAG evaluation](https://arxiv.org/abs/2309.15217). Faithfulness defined against the retrieved context, answer relevance, and context relevance defined as focus rather than coverage.

- [Ragas documentation for context recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/). LLM-based recall estimated from reference-answer claims, and the non-LLM variant using reference contexts.

- [Ragas documentation for context precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/). A rank-aware measure of whether relevant chunks sit above irrelevant ones.

- [Ragas documentation for faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/). Claims in the response checked against the retrieved context.

- [Barnett et al., seven failure points in a RAG system](https://arxiv.org/abs/2401.05856). The missing-content, missed-ranking, not-in-context and not-extracted failure points, from an experience report of three case studies.

- [Liu et al., how language models use long contexts](https://arxiv.org/abs/2307.03172). The closed-book and oracle settings, the Table 1 figures, and the position effect.

- [Ragas release listing on PyPI](https://pypi.org/project/ragas/). Version 0.4.3 as the latest release on 2026-11-02.

## Continue reading on AutomationTester.in

- [Use AI for Performance Diagnosis Without Inventing Causes](https://automationtester.in/blog/ai-in-testing/ai-performance-diagnosis-evidence-boundaries)
- [Review AI-Generated JUnit Tests Before They Raise Coverage](https://automationtester.in/blog/ai-in-testing/review-ai-generated-junit-tests-before-coverage)
- [Catch promptfoo Config Mistakes That validate Lets Through](https://automationtester.in/blog/tools-reviews/catch-promptfoo-config-mistakes-before-eval)
- [Playwright MCP Is Not Your Regression Test Suite](https://automationtester.in/blog/ai-in-testing/playwright-mcp-vs-regression-test-suite)
- [MCP Observability: Audit Tool Calls Without Logging Secrets](https://automationtester.in/blog/ai-in-testing/mcp-observability-audit-without-secrets)

Source: [Find Whether a RAG Answer Failed in Retrieval or Generation](https://automationtester.in/blog/ai-in-testing/debug-rag-retrieval-vs-generation-failures) by Shashank Rawlani.