Hybrid Search for RAG: Combining BM25 and Dense Embeddings

Quick answer: Exact keywords and semantic similarity solve different retrieval problems. The practical approach is to design for correctness first, measure with real workloads, and only then optimize the parts that are proven to matter.

Why this matters in 2026

Retrieval-augmented generation (RAG) connects a language model to external evidence. In 2026, the engineering challenge is not merely finding similar passages; it is delivering current, authorized, attributable answers under production latency and budget constraints. Exact keywords and semantic similarity solve different retrieval problems. For teams shipping SaaS products, AI copilots or internal automation, an architectural choice affects customer trust, future maintenance and recurring infrastructure costs. A proof of concept may tolerate manual fixes; a production platform needs explicit behavior when inputs change or dependencies fail.

The architecture and how it works

Index the same versioned documents for lexical search and embeddings. Retrieve from each independently, fuse ranked results with reciprocal rank fusion, then rerank a shortlist. This should be represented as a documented sequence with clear inputs, outputs and owners. Each boundary needs a testable contract: what is accepted, what happens on partial failure, and what can safely be retried.

A useful implementation sequence is:

1. Define the workload. Collect representative requests, data volume, concurrency, access controls and the maximum acceptable delay or error rate. Avoid choosing a database or framework before defining these.
2. Build a small baseline. Implement the simplest pipeline that solves the core user task. Record its quality, latency and costs before adding complexity.
3. Add operational safeguards. Index the same versioned documents for lexical search and embeddings. Treat access control, retries, timeouts, versioning and monitoring as part of the design rather than a later patch.
4. Test realistic failures. Simulate dependency timeouts, duplicate events, stale data, partial updates and sudden load increases. Check final business outcomes, not only HTTP success codes.
5. Roll out incrementally. Start with a controlled slice of traffic, compare metrics against the baseline and retain a rollback path.

Example: a realistic application

An error-code search such as ERR_CONNECTION_RESET often needs lexical matching; a paraphrased policy question benefits from dense retrieval. In this situation, the team should draw a boundary between trusted application state and external or model-produced material. The implementation should record the identifiers and versions needed to replay an incident, but it should avoid indiscriminate logging of personal or confidential content.

A minimal system diagram for this use case is:

Client / Business Event
        |
        v
Identity + Validation + Policy
        |
        v
Application Orchestrator
        |
        +----> Data / Search / Stream / Tool
        |                 |
        <----- Result + Provenance
        |
        v
Business Rules + Response
        |
        v
Metrics / Tracing / Audit

Not every system needs every component. For example, a small, read-only search app can be simpler than a multi-tenant system that modifies financial records. Complexity should follow proven requirements.

Common failure modes and how to avoid them

The most important trap: Merging raw scores from unrelated ranking systems without normalization creates unstable results. Prevent it with an automated regression test that reproduces the failure under concurrency or bad input. Also verify that access controls are enforced in the service and storage layers, not just through user-interface hints or model instructions.

A second common mistake is optimizing average latency while ignoring rare but severe failures. Capture p95 or p99 performance, retry amplification, stale outputs and the cost of recovery. Finally, document which guarantees are provided by the underlying platform and which must be built in application code; a managed service does not eliminate application-level correctness requirements.

What to measure before launch

Track Recall@10, mean reciprocal rank, zero-result rate and reranker uplift.. Establish a baseline and an explicit acceptance threshold for each metric. Report both normal operation and degraded-mode behavior; the latter often determines whether a production service is trustworthy.

Production readiness checklist

• Define data ownership, refresh and deletion rules before embedding.
• Build an evaluation set that includes no-answer and permission-boundary queries.
• Track source/version metadata all the way to citations.
• Compare accuracy and cost before introducing new pipeline stages.

FAQ

Is this necessary for a small MVP?

Not always. Start with the smallest implementation that meets user needs, but keep security, correctness and recovery constraints from day one. Add advanced components only if measurements justify them.

What is the first step for an existing application?

Instrument the current workflow, identify its highest-impact failure, and run one controlled experiment. Keep the old path available until the new implementation demonstrates a measurable improvement.

How should a team choose between competing tools?

Create a short benchmark using its own requests and data. Compare correctness, operational burden, vendor portability, p95 latency and total monthly cost—not just advertised peak throughput.

Further reading

• LangChain RAG tutorials
• pgvector project documentation
• OWASP LLM Top 10

More from Rohini

• Production RAG Architecture in 2026: From Documents to Reliable Answers
• Agentic RAG vs Traditional RAG: Which Architecture Should You Choose?
• RAG Chunking Strategies: Semantic, Recursive and Parent-Child Retrieval

---

Rohini builds AI-enabled applications, RAG systems, agents and scalable backend infrastructure. [Discuss an architecture project](/contact) if you need help taking an implementation from prototype to production.