MutOracle-RAG

The problem

End-to-end RAG scores tell you that a system is wrong. They do not tell you which stage was wrong. A bad answer can come from retrieval returning the wrong chunks, from generation ignoring chunks that were correct, or from the test oracle scoring a correct answer as a failure. These three failures have nothing in common and demand different fixes, but a single aggregate number collapses them into one.

MutOracle-RAG is my attempt to keep the stages separate and localize the fault.

Approach

The pipeline is split into three instrumented stages: a DuckDB-backed retrieval stage, a generation stage, and an oracle-testing stage. Each stage records its inputs and outputs so a failure can be traced backwards rather than inferred from the final answer.

DuckDB is doing real work here rather than acting as a stand-in for a vector database. Corpus, chunks, embeddings, and per-run traces all live in one embedded columnar store, which means an experiment is a single file and joining a run’s retrieval trace against its scores is a query rather than a script.

The fault-localization signal comes from mutation. I implemented mutation operators that perturb the pipeline in controlled ways: corrupting retrieved context, dropping relevant chunks, injecting distractors, and altering the answer under test. If a mutation to the retrieval stage does not change the score, the metric is not sensitive to retrieval and cannot be used to diagnose it. This borrows the logic of mutation testing, where surviving mutants indicate a weak test suite rather than correct code.

Evaluation runs against FITS, and against two families of oracles: local deterministic oracles based on string and structural matching, and LLM-backed oracles. The two disagree in interesting ways. Local oracles are cheap, reproducible, and brittle to paraphrase. LLM oracles handle paraphrase but drift between runs and are themselves a component that can fail. Treating the LLM oracle as trusted ground truth would defeat the purpose, so it is scored as a stage like any other.

Scores are calibrated rather than used raw, so oracle outputs across different judges land on a comparable scale. Experiments are deterministic: fixed seeds, pinned model versions, and recorded configuration, so a run can be reproduced exactly. I hold the codebase at a 95% test-coverage gate. On a project whose subject is the reliability of evaluation, an untested evaluator would be an obvious contradiction.

Where it stands

The work is ongoing. The stage decomposition holds up and the mutation operators do surface metrics that are blind to failures they claim to detect.

The honest limitations: LLM-backed oracles remain nondeterministic in ways that pinning cannot fully remove, so their calibration is a moving target across model releases. Mutation operators encode my own assumptions about how RAG breaks, and failure modes I did not think to mutate are failure modes the pipeline will not find. High test coverage tells me the code runs the paths I wrote; it says nothing about whether those paths are the right ones.