Contextual Retrieval: What It Is and Why It Improves RAG

The “orphan chunks” problem

When a RAG system splits a document into chunks, each fragment loses its original context. A chunk saying “The deadline is 30 business days” doesn’t indicate what deadline it’s about, which section it was in, or what process it refers to.

This creates a critical problem: semantic search finds the chunk, but the LLM doesn’t have enough context to give a precise answer.

What is Contextual Retrieval?

Contextual Retrieval enriches each chunk with a context paragraph generated by an LLM before generating the embedding. The LLM reads the full document and generates a brief description that situates each fragment:

• Which section or topic it comes from
• What key entities or concepts it mentions
• How it relates to the rest of the document

How the pipeline works

1. Normal parsing and chunking

The document is split into fragments using standard parameters (size, overlap). Nothing different here.

2. LLM contextualization

For each chunk, the LLM receives:

• A document summary (~first 500 characters)
• The chunk content

The LLM generates a brief context paragraph, which is prepended to the original chunk.

3. Enriched content embedding

The embedding is generated on context + original chunk, not just the chunk. This makes the vector capture much more semantic information.

4. Dual storage

Both original and contextualized content are stored. Search uses the contextualized embedding; for the final answer, the LLM sees original content (preventing generated context from “contaminating” the response).

Concrete example

Original chunk:

“The maximum deadline is 30 business days from the date of request.”

Generated context:

“This fragment belongs to the ‘Returns Policy’ section of the customer service manual. It describes the refund deadline for defective products.”

Contextualized chunk (what gets indexed):

"This fragment belongs to the ‘Returns Policy’ section of the customer service manual. It describes the refund deadline for defective products.

The maximum deadline is 30 business days from the date of request."

Now when someone asks “How long does a refund take?”, the contextualized embedding matches much better.

When to enable Contextual Retrieval?

Enable it if:

• Your KB has multiple documents on different topics
• Documents are long or poorly structured
• Chunks tend to be ambiguous out of context
• You see low confidence on queries that should have good answers

Skip it if:

• Your KB has few short, well-titled documents
• FAQs already cover the most frequent questions
• Chunk volume is low (<50)

Cost impact

Contextualization consumes LLM tokens per chunk processed. For a 100-chunk document at ~200 tokens per call, that’s ~20,000 total tokens. This is an indexing investment that pays off with every subsequent query returning better results.

Contextual Retrieval is one of the most effective techniques for improving RAG without changing the model. Try it →