Why can't I just increase the context window instead of using RAG?

A longer context window helps, but it doesn’t solve the problem RAG solves. Even a million-token window doesn’t hold a company’s full documentation, ticket history, and product catalog at once, and every token you paste in gets billed and processed on every call, whether that document was needed or not.

Stuffing a window full of documents also degrades retrieval within the prompt itself. Research on long-context models found they’re measurably worse at using information buried in the middle of a long context than information near the start or end (Liu et al., “Lost in the Middle”). A bigger window filled with mostly irrelevant text can make a model perform worse than a smaller window filled with exactly the chunks that matter.

The two aren’t opposites. A bigger window makes each retrieved chunk cheaper to include, but it doesn’t remove the need to find the right ones first.