Production RAG Pipeline Architecture: Hybrid Search, Reranking & Guardrails
How to build production RAG pipelines: hybrid BM25 + dense vector search, Reciprocal Rank Fusion, cross-encoder reranking, and citation guardrails.
Architectural Summary (AEO)
Naive vector-only RAG breaks in production on exact product codes, acronyms, and multi-hop queries. Production RAG combines BM25 lexical search with dense vector retrieval via Reciprocal Rank Fusion (RRF), filters by tenant ACLs before scoring, applies a cross-encoder reranker on the top-30 candidates, and enforces strict inline citation guardrails.
Reference System Flow
User Query ──► Query Rewriter / Intent Classifier
│
┌──────────┴──────────┐
▼ ▼
BM25 Lexical Dense Vector (HNSW)
(Top 30, ACL-filtered) (Top 30, ACL-filtered)
└──────────┬──────────┘
▼
Reciprocal Rank Fusion (RRF) ──► Top 25 Candidates
│
▼
Cross-Encoder Reranker (BGE / Cohere) ──► Top 5 Chunks
│
▼
Grounded LLM Synthesis + Inline Citations [doc_id#page]1. Why Naive Cosine-Only RAG Fails in Production
Dense embeddings compress meaning into a fixed vector, which blurs exact identifiers like invoice numbers (`INV-2026-084`), error codes (`ERR_CONN_RESET`), and legal clause IDs. Running parallel BM25 keyword search alongside dense embeddings ensures both semantic and exact lexical matches surface.
2. Reciprocal Rank Fusion (RRF) & Pre-Filtered ACLs
Combine lexical and semantic result lists without brittle score normalization using RRF (`score = sum(1 / (k + rank_i))`, `k=60`). Crucially, apply tenant and role-based ACL filters *inside* the index query (pre-filtering), never after top-K retrieval, so unauthorized documents never leak or starve results.
3. Cross-Encoder Reranking & Hallucination Guardrails
Bi-encoders encode query and document independently; cross-encoders attend across both tokens simultaneously for far higher precision on the top 25–30 candidates. Finally, require `[source_id]` citations for every factual claim and return an explicit abstain response when top rerank scores fall below threshold.
Reciprocal Rank Fusion (RRF) + Grounded Prompt Pattern
export function reciprocalRankFusion(bm25Hits, vectorHits, k = 60) {
const scores = new Map();
const docs = new Map();
[bm25Hits, vectorHits].forEach((list) => {
list.forEach((doc, index) => {
const prev = scores.get(doc.id) || 0;
scores.set(doc.id, prev + 1 / (k + index + 1));
docs.set(doc.id, doc);
});
});
return Array.from(scores.entries())
.sort((a, b) => b[1] - a[1])
.slice(0, 25)
.map(([id, rrfScore]) => ({ ...docs.get(id), rrfScore }));
}Architectural Trade-Offs
| Stage | Latency Budget | Recall / Precision Gain | When to Skip |
|---|---|---|---|
| BM25 + Dense RRF | 15–35ms | +18–25% recall on exact terms | Never skip for enterprise docs |
| Cross-Encoder Rerank | 60–140ms | +12–20% top-5 precision | Ultra-low-latency autocomplete |
| Semantic Query Cache | < 5ms on hit | Zero LLM cost on repeat FAQs | Highly personalized user queries |
Production Benchmarks & Field Observations
- Hybrid BM25 + HNSW + BGE-Reranker-v2-m3 improves Context Precision@5 from 0.61 (vector-only) to 0.89 on technical SOP datasets.
Related Commercial Services
Engineering FAQs
How many chunks should we pass into the LLM context window?
Pass the top 4 to 8 reranked chunks (roughly 2,000–4,500 tokens). Stuffing 30+ noisy chunks degrades attention ('lost in the middle') and increases latency and cost.
How do we measure if a RAG change actually improved accuracy?
Maintain a 100+ question golden evaluation set and score Faithfulness, Answer Relevance, and Context Recall on every chunking or prompt change.