The complaint always arrives the same way: the model is hallucinating.
It usually isn’t. When a RAG returns wrong answers, the model is normally doing exactly what it was told — answering from the chunks it was handed. The chunks were wrong. The failure happened in retrieval, several steps upstream, and by the time anyone notices, the only visible symptom is a confident wrong sentence with the model’s name on it.
So I built a labelled test corpus and measured where the failures actually come from. Twenty-eight documents with department metadata, publication years, deliberate near-duplicates and error codes. Twenty queries with known correct answers, three of which the corpus cannot answer at all. Then I ran dense retrieval, BM25 and a hybrid of both, and counted.
Some of the results were worse than I expected:
- Filtering metadata after the search returned zero results for 65% of queries — silently, with no error anywhere in the pipeline.
- Error codes and product SKUs embedded to a literal zero vector, which makes them unmatchable by similarity search at any k.
- The highest-scoring query in the entire experiment was one the corpus has no answer for at all.
- Three of five top-k slots went to near-duplicates of each other.
- A document explicitly marked superseded ranked #1 for the question it was superseded on.
Here is what breaks, in the order you should check it.
What this article covers
- How I measured it
- 1. No score threshold, so the retriever always returns something
- 2. Identifiers and error codes defeat similarity search
- 3. Near-duplicate chunks eat your top-k
- 4. Superseded documents outrank their replacements
- 5. You filter metadata after search instead of before
- 6. Your k is wrong and measuring is the only way to know
- 7. The chunk is in context but incomplete
- 8. Retrieval was fine and the failure is downstream
- The 10-minute diagnosis
- Build the measurement harness first
- Run it yourself: the complete code
- What this test doesn’t tell you
- FAQ
How I measured why a RAG returns wrong answers
The corpus is 28 short documents from a fictional company: platform runbooks, billing policies, security procedures, support notes. Each carries a department, a year and an ID. Four of them are near-identical descriptions of the same key-rotation procedure, because real corpora are full of that. Several contain identifiers like ERR_5521 and BX-9920.
Twenty queries, each labelled with the document IDs that genuinely answer it. Eight are plain questions, four contain identifiers, four are phrased the way a frustrated user would phrase them rather than the way the document is written, one targets the duplicate cluster, and three are out-of-domain — the corpus contains no answer. That leaves 17 answerable queries, which is the denominator in every recall figure below.
Three retrievers: dense (cosine similarity over mean-pooled 300-dimension word vectors), BM25 implemented directly, and a hybrid fusing both with reciprocal rank fusion.
One caveat you should read before the numbers. My dense retriever uses static word vectors, which are considerably weaker than a modern sentence-embedding model. On plain questions and paraphrased queries, a current embedding model would score much higher than mine does. So treat the dense-versus-BM25 comparison as a direction, not a magnitude.
What does transfer regardless of embedding quality is everything structural: how identifiers tokenise, how duplicates consume top-k slots, what happens when you filter after ranking instead of before, and whether similarity has any notion of time. Those are properties of the architecture, not the model, and they are where most of the findings below sit. The code is at the end so you can run it against your own corpus and your own model.
1. You have no score threshold, so the retriever always returns something
A vector search returns your top k. Always. Ask it something your corpus knows nothing about and it will hand back the five least-irrelevant chunks it owns, and the LLM will dutifully try to construct an answer from them.
I gave the retriever three questions the corpus cannot answer — the capital of France, how to bake sourdough, and 2019 revenue figures. Every one returned five chunks. Then I compared the top-1 similarity scores against the legitimate queries:
| top-1 similarity | min | mean | max |
|---|---|---|---|
| in-domain queries | 0.841 | 0.897 | 0.936 |
| out-of-domain queries | 0.553 | 0.781 | 0.948 |
The highest-scoring query in the entire experiment was one the corpus has no answer for. It outscored every legitimate question. The separation gap between the lowest legitimate score and the highest nonsense score is −0.107 — they overlap, so no single threshold cleanly separates them.
That doesn’t mean thresholds are useless. It means a threshold alone is not a safety net, and you cannot pick one from a blog post. You have to fit it to your own corpus and accept that it will be imperfect.
Fixes, in order: log the score distribution for real queries before choosing anything. Set a floor and return an explicit “I don’t have information about that” below it. Then, because the threshold will leak, add a grounding check — ask the model whether the retrieved context actually supports an answer before it writes one. Two imperfect filters in series beat one.
2. Identifiers and error codes defeat similarity search entirely
This one is absolute, and it surprised me even though I expected it.
| string | has a vector? | vector norm |
|---|---|---|
ERR_5521 | no | 0.000 |
ERR_4110 | no | 0.000 |
BX-9920 | no | 0.000 |
INGEST_BATCH_SIZE | no | 0.000 |
| refund | yes | 6.960 |
| token | yes | 6.210 |
| invoice | yes | 7.242 |
A zero vector has no direction. Cosine similarity against it is meaningless. Searching for ERR_5521 in a pure vector system is not “less accurate” — it is undefined, and no amount of raising k will help.
Modern subword tokenisers don’t produce a literal zero here; they will split ERR_5521 into fragments and give you a vector. But that vector encodes the fragments, not the identifier, so ERR_5521 and ERR_5522 land close together while meaning entirely different things. The failure changes shape and doesn’t go away.
BM25 has no such trouble. An identifier is a rare token, rare tokens get high IDF, and the document containing it shoots to the top. On my four identifier queries, BM25 found the right document 4 out of 4 times; the dense retriever managed 2, and both of those were the ones where I had also written normal words into the query.
The fix is hybrid search, and this is the strongest argument for it. Run BM25 and vector search in parallel and fuse the rankings with reciprocal rank fusion. Not because hybrid is fashionable, but because there is an entire class of query — product codes, error numbers, version strings, ticket IDs, people’s names — where similarity search structurally cannot work and keyword search is trivially correct.
3. Near-duplicate chunks eat your top-k
Real corpora repeat themselves. The same procedure appears in a runbook, a wiki page, an onboarding doc and a Slack export somebody indexed.
I put four near-identical descriptions of one key-rotation procedure into the corpus and asked how to rotate the signing key:
d19 0.918 To rotate the signing key, generate a new key pair... <- duplicate
d10 0.895 Customers on legacy plans keep their existing rate...
d22 0.888 Signing key rotation: create a new key pair... <- duplicate
d20 0.886 Key rotation procedure: generate a new key pair... <- duplicate
d01 0.880 The ingest service reads documents from object...Three of five slots went to chunks saying the same thing. The user paid for five chunks of context and received three distinct ideas — and if the answer needed a second fact from elsewhere in the corpus, it was pushed off the list by redundancy. Note the fourth-ranked result, too: a billing document about legacy plan rates, sitting above two of the four documents that actually answer the question.
Deduplicating above cosine 0.97 before truncating to k fixed it cleanly: retrieve 10, drop near-identical neighbours, keep the top 5. Distinct topics in the final context went from 3 to 5.
Fix: retrieve 2–3× your target k, deduplicate by similarity, then truncate. It costs one extra pass over a handful of vectors and it is the cheapest quality win in this entire article.
4. Superseded documents outrank their replacements
Embedding similarity has no concept of time. Worse, an obsolete document is often more similar to a query than the current one, because it was written about exactly that topic and nothing else, while the replacement has moved on to cover more ground.
My corpus contains a 2023 pricing table explicitly marked as superseded, and a 2024 description of a retired pipeline. Both rank #1 of 3 for the obvious query. The retriever confidently surfaces the wrong version, the LLM answers from it, and the user gets last year’s prices with no indication anything is stale.
Applying a recency prior — multiplying each score by a decay factor of 0.5 per two years of age — demoted both out of the top position immediately.
Fixes: store a publication or modification date on every chunk and apply a decay factor tuned to how fast your domain moves. Better still, add an explicit superseded_by field and filter those out at query time. Best of all, delete obsolete documents from the index rather than trusting ranking to bury them. Ranking is a soft signal; deletion is a guarantee.
5. You filter metadata after search instead of before
This produced the single worst number in the experiment.
The scenario is ordinary: a user may only see billing documents, so results must be restricted to that department. There are two ways to implement it. Post-filtering retrieves the top 5, then discards anything outside billing. Pre-filtering restricts the candidate set first, then ranks within it.
| approach | queries returning zero results |
|---|---|
| post-filter | 11 of 17 (65%) |
| pre-filter | 0 of 17 (0%) |
Two thirds of queries returned an empty context. Not wrong results — nothing at all. And the failure is silent: the pipeline runs, the LLM receives an empty context block, and it answers from its own parameters while sounding exactly as confident as it does when properly grounded.
The reason is simple once you see it. Post-filtering asks “of the 5 globally most similar chunks, how many happen to be billing documents?” When billing is a fifth of your corpus, the answer is frequently zero. This gets dramatically worse as the corpus grows and each tenant’s share shrinks.
Fix: pass the metadata filter into the vector search itself. Every serious vector database supports this. If yours is doing it in application code after the fact, that is a bug, not a design choice — and it is the first thing I would check in any multi-tenant RAG system.
6. Your k is wrong, and you can’t know which way without measuring
Recall across all 17 answerable queries:
| k | dense | BM25 | hybrid |
|---|---|---|---|
| 1 | 6/17 | 14/17 | 7/17 |
| 3 | 7/17 | 15/17 | 11/17 |
| 5 | 9/17 | 15/17 | 13/17 |
| 10 | 12/17 | 15/17 | 14/17 |
| 20 | 14/17 | 17/17 | 17/17 |
Read that with the caveat from the method section firmly in mind — a modern embedding model would lift the dense column substantially, and the BM25 column is flattered by my corpus having a lot of identifier queries in it. The shapes are what matter.
Dense recall climbs steadily with k, which is the signature of a retriever that finds the right document but ranks it poorly. That is a reranking problem, not a k problem. BM25 is nearly flat from k=3 onward — when lexical matching works it works immediately, and when it fails, more results don’t rescue it.
Going from k=5 to k=20 costs four times the context tokens on every single request. Sometimes that is the right trade. You cannot know without measuring recall on your own labelled queries, which is why the harness section matters more than any fix above.
7. The chunk is in the context but it’s incomplete
Retrieval found the right chunk, the chunk made it into the prompt, and the answer is still wrong — because the chunk was cut in the wrong place and the half containing the answer is in a different chunk.
I measured this separately and the numbers are stark: fixed-size chunking divided a code block in 64% of boundary alignments and a table in 44%. A chunk holding the first half of a procedure embeds as something vaguely procedure-shaped and retrieves fine, but cannot answer a question about step four.
The full comparison of five chunking strategies is a separate article. The short version: use structure-aware splitting that respects headings, code fences and table boundaries, and prepend the section heading to every chunk so it carries its own context.
8. Retrieval was fine and the failure is downstream
Once the correct chunk is demonstrably in the top 5, the remaining causes are:
Position. Long contexts get uneven attention, with material in the middle receiving the least — the effect documented in Lost in the Middle. If you are sending 20 chunks, the best one should be first or last, not tenth.
No grounding instruction. Without an explicit “answer only from the context provided, and say you don’t know if it isn’t there”, the model blends retrieved facts with its training data, and you cannot tell which sentence came from where.
No citation requirement. If the model isn’t required to cite chunk IDs, you have no way to audit an answer. Requiring citations turns silent wrongness into a visible, checkable mistake — and that is the difference between a bug you can fix and a bug you will never find.
The 10-minute diagnosis when a RAG returns wrong answers
When someone reports a wrong answer, don’t touch the prompt. Do this instead:
- Log the retrieved chunks for that exact query. Most teams cannot do this, and it is the reason RAG debugging takes days instead of minutes. Fix that first if it is missing.
- Read them. Is the correct chunk there at all?
- Absent from the top 20 → retrieval problem. Work down the left column of the flowchart: identifier, duplicates, staleness, chunking.
- Present but ranked 6–20 → ranking problem. A reranker is the highest-value fix here; retrieve 20 and rerank down to 5.
- Present in the top 5 → the failure is downstream. Check metadata filtering first, then position, then the prompt.
The prompt is the last thing to check and the rarest cause, which is precisely the opposite of where most teams start.
Build the measurement harness before you build fixes
Everything above is guesswork until you can measure your own system. The harness is smaller than people expect:
# 20 questions you know the answers to, and the doc that answers each
queries = [
("What causes memory pressure on an ingest worker?", "d02"),
("ERR_5521", "d03"),
# ... 18 more
]
for k in (3, 5, 10, 20):
hits = sum(
expected in [d for d, _ in retrieve(q, k=k)]
for q, expected in queries
)
print(f"recall@{k} = {hits}/{len(queries)}")Thirty to fifty labelled questions is enough to catch a serious regression — a broken filter, a mismatched embedding model, an ingest that silently stopped. Detecting the smaller improvements that real tuning produces needs far more than that. Write them once, run them after every change to chunking, embedding model, k or filtering. Include a few identifier queries and a few out-of-domain ones — those are where systems fail silently, and they are exactly the cases nobody thinks to test.
Run it yourself: the complete code
This is the whole experiment — three files, no framework, no vector database. Drop them in a directory together and run the last one. Every number quoted above comes out of these scripts.
pip install spacy numpy
python -m spacy download en_core_web_md
python rag_failures.py # experiments 1-6
python rag_refined.py # OOV, thresholds, k sweep, dedup, recencyThe dense retriever uses spaCy’s en_core_web_md static vectors because it installs in one line and runs anywhere. Swap in a real sentence-embedding model — sentence-transformers with all-MiniLM-L6-v2 is a fine starting point — by replacing the embed() function, and rerun. Then publish your own numbers rather than mine: measurements from your corpus are the only ones that will help you.
rag_corpus.py — the labelled corpus and queries
Twenty-eight documents and twenty labelled queries, built to contain the specific things that break real systems.
"""Test corpus for RAG failure-mode experiments.
Deliberately built to contain the things that break real systems:
- identifier strings (error codes, SKUs) that embeddings handle badly
- near-duplicate chunks that crowd out top-k
- metadata for the pre- vs post-filtering experiment
- questions phrased differently from the documents that answer them
"""
# (id, department, year, text)
CORPUS = [
("d01", "platform", 2026, "The ingest service reads documents from object storage, normalises them, and writes chunk records to the vector store. It runs as a scheduled job rather than a long-lived process."),
("d02", "platform", 2026, "Setting INGEST_BATCH_SIZE above 128 causes memory pressure on the default worker size. Scale horizontally by adding workers instead of raising the batch size."),
("d03", "platform", 2025, "Error ERR_5521 is raised when the embedding model cache volume cannot be read within the startup timeout. Stagger worker start times to resolve contention."),
("d04", "platform", 2026, "Error ERR_4110 indicates a vector store write conflict when two workers process overlapping document ranges. Idempotent writes keyed on document hash prevent it."),
("d05", "platform", 2026, "Dry run mode performs every step except the final write to the vector store. Use it to inspect chunk counts before committing to a full re-index."),
("d06", "platform", 2024, "The legacy ingest pipeline used a single-threaded parser and was retired in 2024. Do not reference it in new runbooks."),
("d07", "billing", 2026, "Invoices are generated on the first business day of each month and delivered to the billing contact on file. Delivery failures retry for seven days."),
("d08", "billing", 2026, "A refund above 500 dollars requires approval from a finance manager before it is submitted to the payment processor."),
("d09", "billing", 2025, "Product SKU BX-9920 is the annual enterprise tier. Product SKU BX-9910 is the monthly equivalent and cannot be mixed on one invoice."),
("d10", "billing", 2026, "Customers on legacy plans keep their existing rate until renewal. At renewal they are moved to the closest current tier."),
("d11", "billing", 2023, "The 2023 pricing table listed the enterprise tier at a lower rate. It is superseded and should not be quoted to customers."),
("d12", "security", 2026, "Access tokens expire after 60 minutes. Refresh tokens are valid for 30 days and are rotated on every use."),
("d13", "security", 2026, "All production database credentials are stored in the secrets manager and injected at runtime. Credentials must never appear in environment files committed to source control."),
("d14", "security", 2026, "Report a suspected breach to the security on-call rotation immediately. Do not investigate independently or notify the affected customer before triage."),
("d15", "security", 2025, "Two-factor authentication is mandatory for all staff accounts with access to production systems."),
("d16", "support", 2026, "When a customer reports slow search, first check whether their index has finished rebuilding. A rebuild in progress degrades ranking quality until it completes."),
("d17", "support", 2026, "Customers can export their data as CSV or JSON from the account settings page. Exports over one gigabyte are delivered by email link instead of direct download."),
("d18", "support", 2026, "The mobile application caches search results for 15 minutes. Ask customers to pull to refresh before escalating a stale results complaint."),
# --- near-duplicates: four chunks saying nearly the same thing ---
("d19", "platform", 2026, "To rotate the signing key, generate a new key pair, publish the public key, wait for the propagation window, then retire the old key."),
("d20", "platform", 2026, "Key rotation procedure: generate a new key pair, publish the public key, wait for propagation, then retire the previous key."),
("d21", "platform", 2025, "Rotating the signing key requires generating a new key pair, publishing the public key, waiting for the propagation window, and retiring the old key."),
("d22", "security", 2026, "Signing key rotation: create a new key pair, publish the public half, allow the propagation window to elapse, then retire the old key."),
("d23", "platform", 2026, "Monitoring exports four metrics. Alert when the failure rate exceeds one per minute or a run exceeds ninety minutes."),
("d24", "platform", 2026, "Dashboards are defined as code in the platform repository. Changes made directly in the UI are overwritten on the next deploy."),
("d25", "support", 2026, "Escalate to engineering only after confirming the issue reproduces on a second account. Include the request identifier in the escalation."),
("d26", "billing", 2026, "Failed payments are retried three times over ten days before the account is suspended. Customers receive an email before each retry."),
("d27", "security", 2026, "Audit logs are retained for 400 days and are immutable once written. Access to audit logs is itself audited."),
("d28", "support", 2025, "Seat counts update immediately when a user is added but are prorated on the following invoice rather than charged at once."),
]
# (query, set of ids that genuinely answer it, note)
QUERIES = [
("What causes memory pressure on an ingest worker?", {"d02"}, "plain"),
("How do I inspect chunk counts before a full re-index?", {"d05"}, "plain"),
("How long is an access token valid?", {"d12"}, "plain"),
("Who approves a large refund?", {"d08"}, "plain"),
("What should I do first if a customer says search is slow?", {"d16"}, "plain"),
("How are failed payments handled?", {"d26"}, "plain"),
("How long are audit logs kept?", {"d27"}, "plain"),
("Can customers export their data?", {"d17"}, "plain"),
# identifier queries — the embedding blind spot
("ERR_5521", {"d03"}, "identifier"),
("ERR_4110", {"d04"}, "identifier"),
("BX-9920", {"d09"}, "identifier"),
("what is INGEST_BATCH_SIZE", {"d02"}, "identifier"),
# vocabulary mismatch — question words differ from document words
("why is my indexing job running out of RAM", {"d02"}, "mismatch"),
("the app shows old results after I change something", {"d18"}, "mismatch"),
("someone might have gotten into our system", {"d14"}, "mismatch"),
("can I get my money back on a big order", {"d08"}, "mismatch"),
# duplicate-heavy topic
("how do I rotate the signing key", {"d19","d20","d21","d22"}, "duplicates"),
# out of domain — nothing in the corpus answers these
("what is the capital of France", set(), "out-of-domain"),
("how do I bake sourdough bread", set(), "out-of-domain"),
("what were the quarterly revenue figures for 2019", set(), "out-of-domain"),
]rag_failures.py — the three retrievers and experiments 1 to 6
Dense retrieval, BM25 and reciprocal rank fusion in about sixty lines, followed by the six measurements behind the tables above.
import math, re, statistics as st
from collections import Counter
import numpy as np, spacy
from rag_corpus import CORPUS, QUERIES
nlp = spacy.load("en_core_web_md")
IDS = [c[0] for c in CORPUS]
DEPT = {c[0]: c[1] for c in CORPUS}
YEAR = {c[0]: c[2] for c in CORPUS}
TEXT = {c[0]: c[3] for c in CORPUS}
# ---------------------------------------------------------------- retrievers
def embed(s):
v = nlp(s).vector
n = np.linalg.norm(v)
return v / n if n else v
EMB = np.vstack([embed(TEXT[i]) for i in IDS])
def dense(q, k=5, allow=None):
qv = embed(q)
sims = EMB @ qv
order = np.argsort(-sims)
out = []
for j in order:
if allow is not None and IDS[j] not in allow:
continue
out.append((IDS[j], float(sims[j])))
if len(out) == k:
break
return out
def toks(s):
return re.findall(r"[a-z0-9_]+", s.lower())
DOCT = {i: toks(TEXT[i]) for i in IDS}
DF = Counter()
for t in DOCT.values():
DF.update(set(t))
N = len(IDS)
AVGDL= sum(len(t) for t in DOCT.values()) / N
def bm25(q, k=5, allow=None, k1=1.5, b=0.75):
qt = toks(q)
scores = {}
for i in IDS:
if allow is not None and i not in allow:
continue
d = DOCT[i]; dl = len(d); tf = Counter(d); s = 0.0
for w in qt:
if w not in tf:
continue
idf = math.log(1 + (N - DF[w] + 0.5) / (DF[w] + 0.5))
s += idf * (tf[w] * (k1 + 1)) / (tf[w] + k1 * (1 - b + b * dl / AVGDL))
scores[i] = s
return sorted(scores.items(), key=lambda x: -x[1])[:k]
def rrf(q, k=5, allow=None, kc=60):
a = {d: r for r, (d, _) in enumerate(dense(q, k=N, allow=allow))}
c = {d: r for r, (d, _) in enumerate(bm25(q, k=N, allow=allow))}
fused = {}
for d in set(a) | set(c):
fused[d] = 1/(kc + a.get(d, N)) + 1/(kc + c.get(d, N))
return sorted(fused.items(), key=lambda x: -x[1])[:k]
def hit(res, gold, k=5):
return bool(gold & {d for d, _ in res[:k]})
def bar(title):
print("\n" + "=" * 66); print(title); print("=" * 66)
# ---------------------------------------------------------------- 1
bar("1. RECALL@5 BY QUERY TYPE (dense vs BM25 vs hybrid)")
groups = {}
for q, gold, tag in QUERIES:
if not gold:
continue
groups.setdefault(tag, []).append((q, gold))
print(f"{'query type':<15}{'n':>3}{'dense':>9}{'BM25':>9}{'hybrid':>9}")
print("-" * 46)
overall = {"dense": [0, 0], "bm25": [0, 0], "hybrid": [0, 0]}
for tag, items in groups.items():
r = {"dense": 0, "bm25": 0, "hybrid": 0}
for q, gold in items:
r["dense"] += hit(dense(q), gold)
r["bm25"] += hit(bm25(q), gold)
r["hybrid"] += hit(rrf(q), gold)
n = len(items)
for kk in r:
overall[kk][0] += r[kk]; overall[kk][1] += n
print(f"{tag:<15}{n:>3}{r['dense']:>6}/{n}{r['bm25']:>6}/{n}{r['hybrid']:>6}/{n}")
print("-" * 46)
print(f"{'ALL':<15}{overall['dense'][1]:>3}"
+ "".join(f"{overall[k][0]:>6}/{overall[k][1]}" for k in ("dense","bm25","hybrid")))
# ---------------------------------------------------------------- 2
bar("2. NO THRESHOLD: what scores do OUT-OF-DOMAIN queries get?")
ind, ood = [], []
for q, gold, tag in QUERIES:
top = dense(q, k=1)[0][1]
(ood if tag == "out-of-domain" else ind).append(top)
print(f"in-domain top-1 score: min {min(ind):.3f} mean {st.mean(ind):.3f} max {max(ind):.3f}")
print(f"out-of-dom top-1 score: min {min(ood):.3f} mean {st.mean(ood):.3f} max {max(ood):.3f}")
print(f"\nhighest out-of-domain score ({max(ood):.3f}) vs lowest in-domain ({min(ind):.3f})")
print("OVERLAP -> no threshold separates them cleanly"
if max(ood) > min(ind) else "separable by threshold")
n_conf = sum(1 for q, g, t in QUERIES if t == "out-of-domain" and dense(q, k=1)[0][1] > 0.5)
print(f"out-of-domain queries scoring above 0.50: {n_conf}/3")
print("every out-of-domain query still returned 5 chunks the LLM will try to answer from")
# ---------------------------------------------------------------- 3
bar("3. NEAR-DUPLICATES CROWDING OUT TOP-K")
q = "how do I rotate the signing key"
res = dense(q, k=5)
dupes = {"d19", "d20", "d21", "d22"}
print(f"query: {q}")
for d, s in res:
mark = " <- duplicate" if d in dupes else ""
print(f" {d} {s:.3f} {TEXT[d][:52]}...{mark}")
occupied = sum(1 for d, _ in res if d in dupes)
print(f"\n{occupied} of 5 slots consumed by near-duplicates")
print(f"unique distinct answers surfaced: {5 - occupied + 1}")
# ---------------------------------------------------------------- 4
bar("4. POST-FILTERING vs PRE-FILTERING (metadata)")
print("Scenario: user may only see 'billing' documents.\n")
post_empty = pre_empty = 0
rows = []
for q, gold, tag in QUERIES:
if not gold:
continue
allow = {i for i in IDS if DEPT[i] == "billing"}
post = [(d, s) for d, s in dense(q, k=5) if d in allow] # filter AFTER top-5
pre = dense(q, k=5, allow=allow) # filter BEFORE ranking
if not post: post_empty += 1
if not pre: pre_empty += 1
rows.append((q[:44], len(post), len(pre)))
n = len(rows)
print(f"{'query':<46}{'post':>6}{'pre':>6}")
for r in rows[:8]:
print(f"{r[0]:<46}{r[1]:>6}{r[2]:>6}")
print("...")
print(f"\nqueries returning ZERO results after post-filtering: {post_empty}/{n} "
f"({post_empty/n*100:.0f}%)")
print(f"queries returning ZERO results with pre-filtering: {pre_empty}/{n} "
f"({pre_empty/n*100:.0f}%)")
# ---------------------------------------------------------------- 5
bar("5. STALE DOCUMENTS OUTRANKING CURRENT ONES")
pairs = [("what does the enterprise tier cost", "d11", 2023),
("how does the ingest pipeline parse documents", "d06", 2024)]
for q, stale, yr in pairs:
res = dense(q, k=3)
pos = [d for d, _ in res].index(stale) + 1 if stale in [d for d, _ in res] else None
print(f"query: {q}")
print(f" superseded doc {stale} ({yr}) rank: {pos if pos else 'not in top 3'}")
for d, s in res:
flag = " <- SUPERSEDED" if d == stale else ""
print(f" {d} ({YEAR[d]}) {s:.3f}{flag}")
print()
# ---------------------------------------------------------------- 6
bar("6. HOW MUCH DOES k MATTER?")
print(f"{'k':>3}{'dense':>10}{'BM25':>10}{'hybrid':>10}")
labelled = [(q, g) for q, g, t in QUERIES if g]
for k in (1, 3, 5, 10):
d = sum(hit(dense(q, k=k), g, k) for q, g in labelled)
b = sum(hit(bm25(q, k=k), g, k) for q, g in labelled)
h = sum(hit(rrf(q, k=k), g, k) for q, g in labelled)
n = len(labelled)
print(f"{k:>3}{d:>7}/{n}{b:>7}/{n}{h:>7}/{n}")rag_refined.py — vocabulary, thresholds, dedup and recency
The follow-up experiments: the out-of-vocabulary probe that produces the 0.000 norms, the threshold separability check with identifier queries excluded, the full k sweep, and working implementations of the two fixes — deduplication and a recency prior.
import numpy as np, spacy, statistics as st
from rag_failures import (dense, bm25, rrf, hit, IDS, TEXT, DEPT, YEAR, EMB, embed, nlp)
from rag_corpus import QUERIES
def bar(t): print("\n" + "=" * 66); print(t); print("=" * 66)
# ---------------------------------------------------------------- OOV
bar("A. OUT-OF-VOCABULARY: what happens to identifiers?")
probe = ["ERR_5521", "ERR_4110", "BX-9920", "INGEST_BATCH_SIZE",
"refund", "token", "invoice"]
print(f"{'string':<22}{'in vocab':>10}{'vector norm':>14}")
print("-" * 46)
for p in probe:
d = nlp(p)
inv = any(t.has_vector for t in d)
print(f"{p:<22}{str(inv):>10}{float(np.linalg.norm(d.vector)):>14.3f}")
print("\nAn identifier with no vector cannot be matched by similarity at any k.")
print("This is a property of the tokenizer/vocabulary, not of model quality:")
print("subword models produce a vector, but it encodes the fragments, not the ID.")
# ---------------------------------------------------------------- threshold, cleaned
bar("B. THRESHOLD SEPARABILITY (identifier queries excluded)")
ind = [dense(q, k=1)[0][1] for q, g, t in QUERIES if g and t != "identifier"]
ood = [dense(q, k=1)[0][1] for q, g, t in QUERIES if t == "out-of-domain"]
print(f"in-domain n={len(ind)} min {min(ind):.3f} mean {st.mean(ind):.3f} max {max(ind):.3f}")
print(f"out-of-dom n={len(ood)} min {min(ood):.3f} mean {st.mean(ood):.3f} max {max(ood):.3f}")
gap = min(ind) - max(ood)
print(f"\nseparation gap: {gap:+.3f} ->",
"cleanly separable" if gap > 0 else "OVERLAPPING, no single threshold works")
# how many ood beat the median in-domain score?
med = st.median(ind)
print(f"out-of-domain queries scoring above the MEDIAN in-domain score "
f"({med:.3f}): {sum(1 for s in ood if s > med)}/{len(ood)}")
# ---------------------------------------------------------------- k sweep
bar("C. RECALL@k SWEEP")
labelled = [(q, g) for q, g, t in QUERIES if g]
n = len(labelled)
print(f"{'k':>3}{'dense':>10}{'BM25':>10}{'hybrid':>10}")
for k in (1, 3, 5, 10, 20):
d = sum(hit(dense(q, k=k), g, k) for q, g in labelled)
b = sum(hit(bm25(q, k=k), g, k) for q, g in labelled)
h = sum(hit(rrf(q, k=k), g, k) for q, g in labelled)
print(f"{k:>3}{d:>7}/{n}{b:>7}/{n}{h:>7}/{n}")
print("\nGoing from k=5 to k=20 costs 4x the context tokens.")
# ---------------------------------------------------------------- dedup fix
bar("D. DOES DEDUPLICATION RECOVER THE LOST SLOTS?")
def dedup(res, thresh=0.97, k=5):
kept, vecs = [], []
for d, s in res:
v = EMB[IDS.index(d)]
if any(float(v @ u) > thresh for u in vecs):
continue
kept.append((d, s)); vecs.append(v)
if len(kept) == k: break
return kept
q = "how do I rotate the signing key"
raw = dense(q, k=10)
kept = dedup(raw)
dupes = {"d19","d20","d21","d22"}
print(f"query: {q}\n")
print("before dedup (top 5):")
for d, s in raw[:5]:
print(f" {d} {s:.3f} {'DUP' if d in dupes else ' '} {TEXT[d][:44]}...")
print("\nafter dedup at cosine > 0.97 (top 5):")
for d, s in kept:
print(f" {d} {s:.3f} {'DUP' if d in dupes else ' '} {TEXT[d][:44]}...")
print(f"\ndistinct topics before: {len({'dup' if d in dupes else d for d,_ in raw[:5]})}"
f" after: {len({'dup' if d in dupes else d for d,_ in kept})}")
# ---------------------------------------------------------------- recency
bar("E. RECENCY: does the retriever know a document is superseded?")
tests = [("what does the enterprise tier cost", "d11"),
("how does the ingest pipeline parse documents", "d06")]
for q, stale in tests:
res = dense(q, k=3)
ranks = [d for d, _ in res]
print(f"{q}")
print(f" superseded {stale} ({YEAR[stale]}) ranks #{ranks.index(stale)+1} of 3")
print("\nEmbedding similarity has no notion of time. A superseded document is")
print("often MORE similar to the query than its replacement, because it was")
print("written about exactly that topic and nothing else.")
# with a recency prior
bar("F. RECENCY PRIOR AS A FIX")
def with_recency(q, k=3, halflife=2.0, now=2026):
res = dense(q, k=len(IDS))
adj = [(d, s * 0.5 ** ((now - YEAR[d]) / halflife)) for d, s in res]
return sorted(adj, key=lambda x: -x[1])[:k]
for q, stale in tests:
before = [d for d, _ in dense(q, k=3)]
after = [d for d, _ in with_recency(q, k=3)]
print(f"{q}")
print(f" before: {before}")
print(f" after : {after} superseded {stale} "
f"{'still #1' if after[0]==stale else 'demoted'}")What this test doesn’t tell you
Being clear about the limits. This is 28 documents, not 28 million, and some failures only appear at scale — index quantisation error, shard imbalance, ACL checks timing out. My dense baseline is a static-vector model, so its absolute recall understates a modern embedding model, though the structural findings don’t depend on it. Twenty queries is enough to detect large effects and too few to resolve small ones. And I measured retrieval quality, not end-to-end answer quality, which also depends on the model and the prompt.
What generalises: identifiers break similarity search, duplicates consume top-k, post-filtering silently returns nothing, and similarity has no notion of time. Those are architectural, and they will show up in your system too.
FAQ: why a RAG returns wrong answers
Is it hallucination when a RAG returns wrong answers?
Usually not. In most cases the model answered faithfully from the chunks it was given and the chunks were wrong, missing or empty. Log the retrieved context for the failing query before you conclude anything about the model — if the correct chunk was never retrieved, no prompt change will fix it.
Why can’t my vector search find an exact error code?
Because identifiers are out of vocabulary. In this test ERR_5521, ERR_4110, BX-9920 and INGEST_BATCH_SIZE all embedded to a vector with a norm of 0.000, which makes cosine similarity undefined rather than merely inaccurate. Subword models give you a vector, but it encodes the fragments, not the identifier. Add BM25 and fuse the two rankings.
Will increasing k fix my retrieval?
Only if recall actually climbs with k, which means the retriever finds the right document but ranks it badly — a reranker is the cheaper fix there. If recall is flat, more chunks just add noise. Either way k=5 to k=20 costs four times the context tokens on every request, so measure recall@k on your own labelled queries before you change it.
Is hybrid search always better than dense retrieval?
No, and this corpus flatters BM25 because it is deliberately full of identifiers. The honest claim is narrower: there is a class of query — codes, SKUs, version strings, ticket IDs, proper nouns — where similarity search structurally cannot work and keyword search is trivially correct. If your users type strings like that, you need both.
How many labelled queries do I need to catch this?
Thirty to fifty is enough to catch a serious regression such as a broken filter, a mismatched embedding model or an ingest that stopped silently. Resolving the smaller gains from real tuning needs considerably more. Make sure a few of them are identifier queries and a few are out-of-domain — those are the ones that fail without producing an error.
Why does my RAG return an answer for questions the corpus can’t answer?
Because a vector search always returns its top k. There is no “no result” state unless you build one. Set a score floor, return an explicit refusal below it, and add a grounding check on top, because the threshold will leak — in this test an out-of-domain query scored higher than every legitimate one.
Where to go next
Retrieval design is also the part of a RAG system that interviewers probe hardest — the trade-offs here are the same ones behind the RAG-versus-fine-tuning question in the LLM system design interview, and they come up again in machine learning and generative AI interview questions. If you are earlier in the journey, the generative AI guide covers the ground this article assumes.
But the useful next step is not more reading. Take the three files above, point them at fifty of your own documents and twenty questions you already know the answers to, and run them. The numbers that will help you are yours, not mine.
