Shipping RAG That Survives Contact With Real Users

A retrieval-augmented generation demo takes an afternoon. You embed a corpus, wire a vector store to a chat model, ask it three questions you already know the answers to, and it works. The demo is honest. It is also not the thing you are going to operate.

The service version fails in ways the demo cannot show you, because the demo’s question set was written by the person who built the retriever. Real users ask questions the corpus does not answer, ask them in fragments across four turns, and believe the answer.

Below are the failure modes I keep running into, and what I have found actually helps.

Retrieval is wrong quietly

The worst retrieval failure is not an empty result. It is a chunk that is topically adjacent, superficially relevant, and wrong. It mentions the right product, the right policy name, the right version number. It is about a different case.

The generator does not flag this. It reads the chunk as authoritative context and writes a fluent, confident answer that smooths over the mismatch. The output has the texture of a correct answer. Everything downstream, including the user, treats it as one.

This is the core asymmetry of the architecture: retrieval quality degrades gradually, and generation quality hides the degradation. A retriever that is right most of the time produces a system that looks right nearly all of the time and is wrong in a way nobody notices until someone acts on it.

The mitigation is not a better prompt. It is making the retrieval step observable. Log the retrieved chunks with their scores for every request. When someone reports a bad answer, the first question is which chunks were in context, and you should be able to answer it in seconds rather than reconstructing the state.

Chunking beats model choice

I have watched teams spend a week evaluating embedding models and an hour deciding chunk size. The ratio should be inverted.

Chunking determines what a retrievable unit is. Too small and a chunk loses the context that makes it interpretable, so the model sees a sentence with a dangling pronoun and no idea which entity it refers to. Too large and the embedding averages several topics into a vector that is near nothing in particular, and you burn context window on irrelevant text.

The harder issue is that a fixed chunk size assumes a document structure that most corpora do not have. A table split across a boundary becomes two unusable fragments. A section header separated from its body leaves the body unattributed. Whatever your documents actually are, respecting their structure will usually beat tuning the token count.

Two things that consistently help: keep a small overlap between adjacent chunks so a boundary never cleanly severs a claim from its qualifier, and prepend the document title and section path to each chunk before embedding, so a fragment carries its own provenance into the vector.

None of that is exciting. It matters far more than the leaderboard position of your embedding model.

Abstention is a feature

Some questions have no answer in the corpus. The corpus is finite; user curiosity is not. A system that always produces an answer will produce a fabricated one for every question outside its coverage, and those are precisely the questions where a wrong answer costs the most, because the user had no other source to check against.

So the system needs to be able to say it does not know. Not as a fallback, as a designed behavior with its own tests.

In practice this means a retrieval-side check and a generation-side instruction that reinforce each other. If nothing clears a relevance threshold, do not call the generator at all. If something clears it, instruct the generator that answering only from the provided context is mandatory and that declining is an acceptable output.

Both halves are needed. Threshold alone is brittle, because similarity scores are not calibrated and the right cutoff drifts as the corpus grows. Instruction alone is unreliable, because a model handed weak context will still try to be helpful.

And you have to test abstention explicitly. Build a set of questions you know the corpus cannot answer and check that the system declines. Otherwise every change that raises answer coverage silently raises fabrication too, and your aggregate quality number will look better while the system gets worse.

Evaluate retrieval and generation separately

This is the practice I would keep if I could keep only one.

Conflated end-to-end evaluation gives you a single number that goes down without telling you where. Did the right chunk fail to make the top-k? Or was it there and the generator ignored it, or contradicted it, or added a detail it invented? Those are different bugs with different fixes, and one number cannot distinguish them.

So measure two things.

Retrieval: for a question with a known supporting chunk, is that chunk in the top-k? This is a ranking problem and it has standard tooling. It requires no model call and it is cheap enough to run on every change.

Generation, conditioned on correct retrieval: given the right chunk in context, is the answer faithful to it? Every claim in the answer should be traceable to the provided text. Judge this with the correct context supplied deliberately, not with whatever the live retriever happened to return, or you have folded retrieval error back in and lost the separation.

retrieval_score = recall_at_k(gold_questions, retriever, k=5)
# Note the fixed gold context: this isolates generation.
faithfulness = judge(answer=generate(q, gold_context), context=gold_context)

On the evaluation set itself: a few dozen hand-labeled examples, written by someone who knows the domain, will teach you more than a few thousand auto-generated ones. Synthetic questions produced from a chunk tend to be answerable by that chunk, phrased in that chunk’s vocabulary. They test lexical matching. They do not contain the ambiguity, the wrong terminology, or the compound structure of questions real people ask. A large synthetic set gives you a precise measurement of something you do not care about.

Treat prompts like code

RAG systems drift. Someone tweaks a prompt, someone re-embeds after a library upgrade, someone adds documents, and quality moves without any commit that obviously caused it.

Pin what can be pinned: embedding model version, chunking parameters, retriever configuration, the prompt text itself. Put the prompt in version control and require review on changes to it, because a prompt edit is a behavior change with no type checker to catch it.

Snapshot retrieval results for your gold questions. Store the retrieved chunk identifiers and diff them on every change. When the retrieved set moves, you know immediately, and you know whether it was intentional. This catches the class of regression where output text still reads fine but the system is now grounding on different evidence.

Full determinism is not available. Sampling, provider-side model updates, and index rebuilds all introduce variance. The goal is to shrink the unpinned surface until the remaining variance is small enough that a real regression stands out against it.

Latency and cost are design constraints

Users abandon slow answers. Every retrieval hop, every rerank, every additional generation call adds latency, and the sum is what the user experiences.

Cache aggressively where the corpus is stable. Embedding the same query repeatedly is pure waste, and question distributions have a heavy head. Caching common queries and their retrieved sets removes a large slice of load for almost no complexity.

Reranking earns its keep when the first-stage retriever has good recall and poor precision. Pull a wide candidate set cheaply, then let a cross-encoder reorder the top of it. That is a real quality gain for a real latency cost. If the first stage is already returning the right chunk at rank one, a reranker adds latency and nothing else. Measure recall at a wide k first. If it is low, reranking cannot save you, because the correct chunk is not in the candidate set at all.

Multi-turn breaks the assumption of self-contained queries

Follow-up questions are not standalone. “What about the second one?” embeds to nothing useful. The retriever sees a query stripped of every entity that gives it meaning, returns something plausible, and the generator answers the wrong question confidently.

Query rewriting fixes this: use the conversation history to expand the follow-up into a self-contained query before retrieval. Cheap, and it removes a whole category of confusing failures.

Persistent memory across sessions is a further step, and where I have used Mem0 the value was in carrying stable user context rather than recent turns. It also introduces a new failure mode, which is stale or wrong remembered facts contaminating retrieval long after they stopped being true. Memory needs an expiry story and a way for a user to correct it. I do not think I have that fully worked out yet.

A statistician’s habit

My degree is in statistics, and the two habits that transferred best are both about suspicion.

The first is asking what would falsify a claim. “The RAG system works well” is not falsifiable as stated. “The correct chunk appears in the top five for at least this fraction of the gold set” is. Every quality claim about a RAG system should be phrased so that a specific observation could contradict it, and then you should go look for that observation.

The second is distrust of aggregates. A single mean over a heterogeneous evaluation set hides the shape of the distribution underneath it. Improvements that raise the average frequently do so by getting better at the questions that were already easy, while the hard tail stays broken or degrades. Break the set into slices you care about, question type, document source, answerable versus not, and read the slices. The average is where regressions go to hide.

Neither habit is specific to RAG. Both are the difference between a system you have measured and a system you have merely watched succeed.