Document Chunking Strategies & Embedding Dimensionality
Engineering guide to semantic chunking, recursive character splitting, parent-document retrieval, and embedding dimensionality trade-offs (768 vs 1536 vs 3072).
Architectural Summary (AEO)
Chunking quality determines the upper bound of RAG accuracy. Production ingestion pipelines replace naive fixed-token slicing with structure-aware recursive splitting (preserving Markdown/HTML headings and tables) paired with small-to-big parent-document retrieval, using 768–1536 dimensional embeddings for optimal recall-to-RAM efficiency.
Reference System Flow
Raw PDF / HTML / Markdown │ Layout Parser (preserve headings, tables, code blocks) ▼ Parent Section (~1,200 tokens, full context) ├── Child Chunk A (300 tokens, 50-token overlap) ──► Embedded in Vector Index ├── Child Chunk B (300 tokens, 50-token overlap) ──► Embedded in Vector Index └── Child Chunk C (300 tokens, 50-token overlap) ──► Embedded in Vector Index
1. Fixed-Window vs Recursive vs Semantic Chunking
Fixed 512-token slicing cuts mid-sentence and splits tables in half. Recursive character splitting respects structural boundaries (`\n## `, `\n\n`, `. `) first, while semantic chunking groups adjacent sentences whose cosine similarity stays above a dynamic percentile threshold.
2. Small-to-Big (Parent Document) Retrieval
Embed small, focused child chunks (250–350 tokens) for sharp vector matching, but store a `parent_id` pointer so the LLM receives the surrounding 1,000-token parent section at generation time.
3. Embedding Dimensionality & Matryoshka Trade-Offs
Doubling vector dimensions from 1536 to 3072 doubles RAM and HNSW index memory while often yielding < 1.5% recall gain. Matryoshka Representation Learning (MRL) models allow truncating 1536d/3072d vectors to 512d–768d with minimal accuracy loss.
Structure-Aware Recursive Splitter with Metadata Enrichment
function buildChunkRecord({ text, docTitle, sectionHeading, pageNumber, parentId }) {
// Prepend structural context so isolated chunks remain self-explanatory
const enrichedContent = [
`Document: ${docTitle}`,
sectionHeading ? `Section: ${sectionHeading}` : "",
text.trim(),
].filter(Boolean).join("\n");
return {
content: enrichedContent,
metadata: { docTitle, sectionHeading, pageNumber, parentId },
};
}Architectural Trade-Offs
| Strategy | Index Size | Table / Code Safety | Best Use Case |
|---|---|---|---|
| Fixed Token Window | Predictable | Poor (splits mid-table) | Uniform plain-text transcripts |
| Recursive Structure-Aware | Moderate | High (keeps sections intact) | Docs, blogs, policies, codebases |
| Parent-Child Retrieval | Small child vectors | Excellent | Complex PDFs, legal & technical manuals |
Production Benchmarks & Field Observations
- Prepending `Document + Section` headers to child chunks improves retrieval recall on ambiguous queries by 11–16%.
Related Commercial Services
Engineering FAQs
What chunk overlap should we use?
10% to 15% overlap (e.g., 40–60 tokens on a 400-token chunk) preserves cross-boundary context without bloating index storage.
How should we handle tables inside PDFs?
Convert tables into clean Markdown or JSON key-value summaries rather than raw flattened text, and keep the entire table in a single chunk whenever possible.