Cross-Encoder Reranking: How to Improve Your Chatbot's Accuracy
Citai Team
March 6, 2026 · 10 min read
The problem: initial search isn’t perfect
Embedding + BM25 search is fast but imprecise. It retrieves ~20 candidates, but not all are relevant.
What is a cross-encoder?
Unlike embeddings (which process query and document separately), a cross-encoder processes both together. Much more precise rankings.
MMR Diversity
After reranking, Citai applies MMR to avoid redundancy and bring complementary information.
Practical benchmarks: latency vs accuracy
| Configuration | Latency | nDCG@5 | Tokens to LLM |
|---|---|---|---|
| Embeddings only | ~80ms | 0.62 | ~4,000 |
| Hybrid (embed + BM25) | ~100ms | 0.68 | ~4,000 |
| Hybrid + Cross-encoder | ~300ms | 0.85 | ~1,000 |
| Hybrid + CE + MMR | ~320ms | 0.83 | ~1,000 (diverse) |
The extra ~200ms is imperceptible to users (LLM generation takes 2-4s), but context tokens drop by 75%.
When NOT to use reranking
- Very small KBs (< 50 chunks): Initial search is already good enough
- Exact term queries: BM25 already matches well on specific technical jargon
- Critical latency (< 100ms): Reranking may be a bottleneck
- FAQ-only mode: If 95% of queries resolve via FAQ matching, reranking rarely runs
Comparison of popular cross-encoder models
| Model | Params | Languages | Latency (20 docs) | GPU |
|---|---|---|---|---|
| ms-marco-MiniLM-L-6-v2 | 22M | EN | ~150ms | No |
| mmarco-mMiniLMv2-L12 | 118M | 100+ | ~300ms | No |
| bge-reranker-large | 560M | EN/ZH | ~800ms | Recommended |
| Cohere Rerank v3 | API | 100+ | ~200ms | Cloud |
Citai uses mmarco-mMiniLMv2 because it is natively multilingual (100+ languages), CPU-friendly, and has a good latency-accuracy balance.
Fine-tuning for specific domains
For specialized terminology (medical, legal), fine-tuning improves nDCG@5 by +8% to +15%. You need 1,000+ labeled query-document pairs and can use sentence-transformers with CrossEncoderTrainer.
Integration patterns
Sequential (Citai): Hybrid Search → Cross-encoder → MMR → LLM
Cascade: Embeddings → top 100 → Light CE → top 20 → Heavy CE → top 5
Conditional: Only activate reranking when the top result has score < 0.9
Reranking is enabled by default in Citai. Try it →
Try Citai for free
Create your intelligent knowledge base in minutes. No credit card required.
Create free accountRelated articles
What is RAG? Retrieval-Augmented Generation Explained
Discover how RAG combines intelligent search with generative AI to deliver accurate answers with verifiable sources.
Read article → Practical GuideConfidence Scoring: How to Interpret and Improve Answer Quality
Understand what each confidence level means and what actions to take to go from amber to green.
Read article → TechnologyEmbeddings and Semantic Search: How Context-Aware AI Works
Understand how embeddings transform text into vectors to find information by meaning, not exact words.
Read article →