Retrieval augmented generation

Retrieval-augmented generation, RAG, is the pattern of fetching relevant content at request time and putting it in the model's context before it answers. Documents are split into chunks, each chunk is embedded into a vector, and the chunks sit in an index. At request time the question is embedded the same way, the index returns the closest chunks, and the model generates its answer from what came back.

It helps an agent because a model's weights hold only what it was trained on, and context windows are finite, so an agent can't carry its whole knowledge base into every request. RAG hands it exactly the slice a request needs, from internal docs, past tickets, or a product catalog, without retraining anything, and the corpus can be updated at any time.

RAG is bad at questions whose answers aren't localized in a few passages. Retrieval returns a top handful of chunks, so anything that needs aggregating across many documents, counting, or summarizing a whole corpus won't work. It's also weak when the question's wording doesn't resemble the document's wording, since similarity search leans on phrasing. And when the corpus holds stale or contradictory content, the model will faithfully answer from the wrong version.