5 things that break in your retrieval (and how to fix them)

Retrieval-Augmented Generation (RAG) is, in every demo you see, a few lines of code: embed your documents, embed the question, find the top-5 nearest chunks, paste them into the prompt. That works for the demo. And then you go live, get real questions, and it falls apart.

Below are five categories of problems we ran into over 12 months of production, plus how we solved them. None of them is rocket science; all of them require you to go beyond the defaults.

1. Chunking, the mistake starts here

The default in every RAG tutorial is recursive character splitting at 1,000 tokens with 200 overlap. Works for essays. Fails for technical documentation, FAQs and legal texts.

Example: an FAQ page with 30 questions. At 1,000-token splitting you get ~3-5 questions merged per chunk. The answer to question 12 then sits in chunk 4, together with questions 11 and 13, which may be about entirely different topics. The embedding of that chunk is an average of three semantically different things, and ranking suffers as a result.

Our approach

• Structure-aware chunking. For Markdown we split on H2, for HTML on semantic blocks, for FAQ pages per question-answer pair.
• Minimum chunk size. Every chunk at least 300 tokens, otherwise the embedding is too fragmented and you lose context.
• No overlap where you can avoid it. Overlap is a proxy for 'I don't know where the logical boundary is'. If you do know it (header, paragraph break), overlap is unnecessary.

2. Recency, the channel nobody fixes

Pure cosine similarity has no idea that one of two identical documents is from yesterday and the other from 2019. For customer support that's a big problem: old documentation that has been archived internally but is still in the index will beat the fresh FAQ article if it happens to match the question better.

Our approach

A recency boost as a post-processing step. Concretely: after the initial vector search we multiply the similarity score by a decay function based on the last_modified timestamp. Documents older than 18 months get 0.7x, older than 3 years 0.5x. No hard cutoff, sometimes an old doc is relevant, but enough to let fresh content tip the balance when scores are equal.

3. Authority / source boosting

Not all sources are equal. An official return policy page should always beat a blog post about returns. A runbook should beat a Slack thread. Pure similarity doesn't know this.

Our approach

Per knowledge source an authority weight (0.5 to 2.0). Official policy pages: 2.0. Help-center articles: 1.5. Blog: 1.0. Slack archive: 0.7. External sites: 0.5. That weight is multiplied by the similarity score after the vector search.

Important: these weights are configured by the merchant themselves, not by us. We don't know which source is authoritative for their business. We give defaults and they tweak.

4. Deduplication, the hidden problem

A lot of content ends up in your index in multiple forms: an article + a newsletter version + a Slack forward. Vector similarity ranks all three high, and your top-5 retrievals are then really top-2 or top-3 unique documents with variations.

Effect: the LLM gets the same info 5 times and 'thinks' it's important, when what you actually need is the diversity to formulate a good answer.

Our approach

1. At indexing: detect near-duplicates via MinHash / SimHash. Keep one canonical version per cluster.
2. At retrieval: after the initial top-K vector search, run a diversity pass. Known as MMR (Maximal Marginal Relevance). Result: 5 retrievals that each cover a different aspect instead of 5 variations on the same text.

5. Query rewriting, where you get the most return

The user's question is rarely the optimal search query. 'My package is lost' is a bad vector-search query. 'Procedure for lost shipment, compensation, contact carrier' is better.

Our approach

A small LLM call before the vector search that reformulates the user query into 1-3 search terms. We use Claude Haiku, costs ~0.005 cent per call, latency 100-200ms, and the gain in retrieval quality is significant.

const expanded = await haiku.complete({
  prompt: `Rewrite this user message as 1-3 retrieval queries.
User: ${userMessage}
Context: ${conversationHistory}
Queries (one per line):`,
});
const queries = expanded.split("\n").filter(Boolean);
const results = await Promise.all(queries.map(embedAndSearch));
// merge + dedupe + rank → return top 5

Bonus: you now get multiple candidate queries you can use for hybrid search (vector + keyword/BM25), which in practice often delivers another noticeable relevance gain.

What NONE of this replaces

Good knowledge management. If your documentation is outdated, contradictory or incomplete, no RAG technique saves you. The techniques above make a good knowledge base 30-50% more effective. Retrieving from a bad knowledge base more carefully is still retrieving bad content.