RAG With Evidence

---

Research checked: 20 September 2026.

Retrieval-augmented generation, usually shortened to RAG, connects a language model to information retrieved for the current question. The model’s parameters provide general capabilities, while an external collection supplies relevant evidence. The original RAG paper combined a pretrained generator with a searchable document index, addressing the difficulty of updating and attributing knowledge stored only inside model weights. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.

The original RAG architecture combines a query encoder and document index with a generator.

Source: Figure 1 from Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020).

Imagine asking an assistant which version of a company’s travel policy applies to a trip. A model answering from memory might produce a plausible policy that never existed. A RAG system searches the company’s documents, selects the relevant policy passages, and gives those passages to the generator. The answer can then identify the rule, its effective date, and its source. This example illustrates the intended workflow; retrieving a document does not guarantee that the model interprets it correctly.

An implementation has two connected parts. Before answering questions, it extracts documents, preserves metadata, divides content into retrievable units, and builds an index. At question time, it finds candidates and constructs the model’s evidence context. In this proposed design, keyword retrieval would help with exact identifiers, semantic retrieval with paraphrases, and reranking with selecting the most useful passages from the candidate set.

Chunking is a design decision with consequences. A tiny fragment may lose the heading that explains its scope. A large fragment may contain enough unrelated material to distract the generator. For the travel-policy example, a useful unit might retain a rule together with its exceptions, date, and department. Keeping stable source identifiers makes later citations and debugging easier.

RAG can fail at several boundaries: the collection may be incomplete, retrieval may miss the relevant passage, the context may contain conflicting versions, or generation may overstate what the evidence supports. Long-context research has also found that models can use the same information differently depending on its position in the prompt. Simply adding more retrieved text does not reliably fix an answer. Liu et al., Lost in the Middle.

More adaptive systems decide when retrieval is needed and assess whether retrieved passages support the response. Self-RAG explores this through a trained model that retrieves, generates, and produces reflection signals. It is one research approach to making evidence use more deliberate, rather than proof that all RAG systems can critique themselves reliably. Asai et al., Self-RAG.

For evaluation, I would separate retrieval quality from answer quality. First ask whether the correct passage was available among the retrieved results. Then ask whether the answer used it accurately, cited the right source, and declined unsupported requests. Include questions with no answer in the collection and questions with obsolete documents nearby. This makes failures actionable: an indexing problem needs a different repair from an unsupported inference. RAG becomes useful when evidence selection, attribution, and uncertainty are treated as parts of the product rather than decorations on the final paragraph.