How to Improve RAG Retrieval Results (Lessons From Building One)

While building a RAG based application, I realized pretty quickly that the hard part of RAG is not the "generation" half, it's the "retrieval" half. Everyone talks about picking the right LLM or writing a clever system prompt, but if the chunks you feed the model are wrong, irrelevant, or incomplete, no prompt engineering can save the answer. The model will either hallucinate to fill the gap or confidently answer from the wrong source.

I ran into this while building a NotebookLM style research workspace where users upload PDFs, spreadsheets, docs, even YouTube videos, and chat with their own sources. Answers had to be strictly grounded in what was uploaded. That constraint is what forced me to actually fix retrieval instead of papering over it with a bigger model. Here's what actually moved the needle for me, roughly in the order I added them.

The naive setup breaks fast

The first version was the textbook approach: embed the user's query, do a cosine similarity search against the vector store, stuff the top chunks into the prompt. It works fine for the demo. It falls apart the moment:

• The user asks a follow-up like "what about the second one" with no standalone meaning

• The question actually has two or three sub-questions bundled into one sentence

• The query is phrased very differently from how the answer is phrased in the source document

• The user says "hi" or "thanks" and you still burn a vector search on it

Each of these needed a different fix. None of them needed a bigger embedding model.

Query rewriting for conversational context

The first real fix was rewriting the query before it ever hits the vector store. In a multi-turn chat, a raw user message often doesn't mean anything on its own. "What about the second one" only makes sense if you know what "the first one" was three messages ago.

So before retrieval, I run the conversation history plus the latest message through a small, cheap model whose only job is to resolve pronouns and implicit context into one standalone search query. "What about the second one" becomes something like "What is the refund policy mentioned in section 2 of the vendor contract." That single rewrite fixed a huge chunk of "the RAG app feels dumb" complaints, because the retriever was finally searching for what the user actually meant, not the literal string they typed.

I use a small, fast model for this step specifically. It's a narrow task, it doesn't need a frontier model, and keeping it cheap matters because this runs on every single message.

Not every message needs retrieval

Related to that: I added a routing step before retrieval even starts. A lightweight classifier decides whether the message needs a document search at all, or if it's a greeting, a meta-question about the app, or something answerable from the last two messages alone. Skipping retrieval for these isn't just a cost optimization, it also stops the model from awkwardly citing a random chunk in response to "thanks, that helped."

HyDE: search with the answer, not the question

This one felt a little strange the first time I implemented it, but it made a real difference. HyDE stands for Hypothetical Document Embeddings. Instead of embedding the user's question and searching for chunks that look like the question, you first ask the model to write a hypothetical answer to the question, and then embed that hypothetical answer to search the vector store.

The reasoning is simple once you see it: questions and answers are phrased very differently. A user asks "how do I cancel my subscription," but the actual document text says "To terminate your recurring plan, navigate to Account Settings." A plain query embedding might not land close to that chunk. But a hypothetical answer generated by the model tends to be phrased much closer to how the real answer is written, so it retrieves better.

In practice, I run both the raw query search and the HyDE search in parallel and merge the results, keeping whichever hits scored higher. HyDE alone sometimes drifts if the hypothetical answer is confidently wrong, so keeping the direct query search as a fallback avoids that failure mode.

Query decomposition for compound questions

The other pattern that kept breaking naive RAG was compound questions. Something like "what's the refund policy and how does it compare to last year's version" is really two separate questions, but a single embedding search treats it as one blurry vector that partially matches both and fully matches neither.

The fix is to detect when a query is compound and break it into up to two or three sub-queries, each treated as its own retrieval problem: its own rewrite, its own HyDE pass, its own search. The results from every sub-query get merged, deduplicated, and reranked before anything goes to the generation step. This is more expensive since you're running multiple retrieval passes instead of one, but it's the difference between a half-answer and a complete one, and I only trigger it when the routing step actually flags the query as compound, so simple questions stay cheap and fast.

Merge, dedupe, cap

Once you're running multiple retrieval strategies per message, query rewrite, HyDE, and decomposed sub-queries, you end up with overlapping chunks coming back from different angles. I merge everything by highest similarity score per chunk, drop duplicates, and cap the final set to a fixed top-N before it goes into the prompt. Without a cap, compound questions can blow past the context budget and drown the model in tangentially related chunks, which hurts answer quality more than it helps.

Chunking mattered more than I expected

None of the above compensates for bad chunking. I originally used a fixed-size splitter and it produced chunks that cut sentences and tables in half constantly. Switching to a strategy that keeps small pages intact and only splits large pages, with a small overlap and a short prefix carried across page boundaries to preserve context, fixed a category of retrieval failures where the right information existed but was split across two chunks that individually looked irrelevant.

Route by cost, not by fear

One thing I'd tell anyone starting this: not every step needs your best model. Routing, query rewriting, HyDE generation, and decomposition are all narrow, mechanical tasks. I run those on a small, cheap model at low temperature and reserve the larger model for the actual user-facing answer. This keeps latency and cost down without touching answer quality, since retrieval quality is what actually determines the final answer, not which model wrote the search query.

What I'd tell someone starting today

If I were starting over, I'd add these in this order:

1. Fix chunking first. No retrieval trick fixes chunks that split mid-sentence.

2. Add conversational query rewriting. This single change fixes the most visible "why is this app dumb" complaints.

3. Add routing so you're not wasting a vector search on "thanks."

4. Add HyDE once you notice queries and answers are phrased differently, which is almost always in real documents.

5. Add query decomposition last, only once you see compound questions actually failing in your logs. It's the most expensive of the four, so build it when you have evidence you need it, not preemptively.

RAG retrieval quality is not one trick, it's a pipeline of small, cheap decisions made before the vector search ever runs. Get those right and the generation step barely has to work hard.