What is retrieval-augmented generation?

RAG works by splitting documents into chunks, embedding each one into a vector, and storing them in an index a query can search at request time. When a request comes in, the question gets embedded the same way, the index returns the closest chunks, and the model writes its answer from whatever came back.

It exists because a model’s weights only hold what it was trained on, and a context window is finite, so an agent can’t carry its whole knowledge base into every call. RAG hands it exactly the slice a request needs, pulled from internal docs, past tickets, or a product catalog, and you can update the corpus at any time without retraining anything.

The tradeoff is that RAG only answers as well as it retrieves. If the chunk with the answer never comes back, the model has nothing true to generate from, no matter how good the model itself is.