Building RAG Systems That Actually Work in Production

Retrieval-Augmented Generation (RAG) is the default pattern for giving LLMs access to private knowledge. In a demo, it looks straightforward: chunk documents, embed them, retrieve similar chunks, stuff them into a prompt.

In production, retrieval quality makes or breaks the product.

Chunking is not one-size-fits-all

Fixed 512-token chunks are a starting point, not a solution. Technical docs benefit from heading-aware splitting. Legal contracts need paragraph boundaries. Code repositories need function-level chunks with file path metadata.

Always store metadata alongside chunks: source URL, section title, last updated date, and access level.

Embedding model selection

Smaller embedding models are faster and cheaper. Larger models retrieve more accurately on nuanced queries. Benchmark on your actual user questions, not on MTEB leaderboard tasks.

Re-embed your corpus when you change models. Version your embedding index.

Reranking improves precision

Bi-encoder retrieval (embed query + embed doc) is fast but approximate. Add a cross-encoder reranker on the top 20–50 results before sending context to the LLM. The latency cost is usually worth the accuracy gain.

Evaluation before launch

Build a golden set of 50–100 question-answer pairs from real user intents. Track:

• Retrieval recall — did the right chunk appear in top-k?
• Answer faithfulness — does the response stick to retrieved context?
• Latency p95 — end-to-end, not just the LLM call

Common failure modes

• Stale data — schedule incremental index updates
• Over-retrieval — too many chunks dilute the context window
• No citation — always return source references so users can verify

RAG is an information retrieval problem wearing an LLM costume. Treat the retrieval layer with the same rigor you'd apply to a search engine.