Anatomy of a Contextualized Chunk: Before and After

The raw chunk

When a document is split into fragments, the result is plain text without semantic metadata. Let’s look at a real example from an HR guide:

Chunk #47 (original):

“Requests must be submitted at least 15 days in advance. In case of emergency, the direct supervisor can approve exceptions. The form is available on the employee portal, Permissions section.”

What is this chunk about? Vacation? Leave? Special permits? Without context, it’s impossible to tell.

The contextualization process

Step 1: Document summary

The system takes the first ~500 characters as a summary:

“Human Resources Policy Manual - Version 2026. This document describes labor policies including vacation, leave, benefits, code of conduct and disciplinary procedures…”

Step 2: LLM prompt

The system sends the summary + chunk and asks for a context paragraph:

• Don’t repeat chunk content
• Only contextualize (section, topic, entities)
• 2-3 sentences max

Step 3: Generated context

“This fragment belongs to the Leaves and Permissions section of the HR Manual. It describes the standard procedure for requesting work permits, including advance deadlines and emergency exceptions.”

Step 4: Final composition

The indexed contextualized chunk combines context + original.

Embedding impact

The contextualized embedding now captures:

• “Leaves and Permissions” → matches leave queries
• “HR Manual” → associates with human resources
• “work permits” → matches work permission queries
• “advance deadlines” → still matches original content

Dual storage: why it matters

The system stores two versions:

Why not use contextualized for everything? Because generated context could introduce subtle inaccuracies. The final LLM always works with original document text.

Document types and context quality

Token consumption

Each contextualized chunk requires one LLM call:

• Input: ~100-200 tokens (summary + chunk)
• Output: ~50-80 tokens (context paragraph)
• Total: ~150-280 tokens per chunk

For a 50-page document (~200 chunks), total cost: ~40,000-56,000 tokens — a one-time indexing cost.

Overlap Strategies and Their Impact

The overlap between chunks determines how much text adjacent fragments share:

How overlap affects reranking

With overlap, the cross-encoder receives partially overlapping chunks. This is an advantage: if the answer is at the “edge” of a chunk, the adjacent fragment also contains it, giving the reranker more opportunities to find the best match.

Metadata Enrichment

Beyond LLM contextualization, chunks benefit from additional metadata:

• Page number — For citing the exact source
• Position in document — Allows reconstructing original order
• File name — Identifies where each chunk comes from
• Processing date — Useful for evaluating freshness
• Detected language — For multilingual search

The embedding combines contextualized text + metadata for a richer, more precise vector.

Chunk Quality Metrics

How do you know if your chunks are good?

• Average confidence per document — If < 60%, there are chunking problems (too small, too large, or poorly fragmented tabular content)
• Orphan chunk rate — Chunks that never appear in search results. If > 30%, chunking isn’t capturing content correctly
• Average reranking score — Top chunks should have scores > 0.5 after the cross-encoder

Debugging: Good Chunks vs. Bad Chunks

Bad chunk example

“See table 3.2 on the next page for more details about the applicable requirements per the regulations described above.”

Problems: Unresolved internal references, no substantive information, depends on content not included.

Good chunk example (contextualized)

“This fragment belongs to the Security Requirements section of the Operations Manual v2026. Production servers must use AES-256 encryption at rest and TLS 1.3 in transit. Security audits are conducted quarterly by an ISO 27001 certified external team.”

Strengths: Clear context, self-contained information, specific entities that improve matching.

Smarter chunks = more precise answers. Try Contextual Retrieval →