CitaiCitai
Back to blogCreate free account
Practical GuideMarch 8, 2026·7 min read

Confidence Scoring: How to Interpret and Improve Answer Quality

Citai Team

March 8, 2026 · 7 min read

What is the confidence score?

Every RAG response includes a confidence score (0.0 to 1.0) indicating how certain the system is that the answer is correct and relevant.

It’s not “the probability of being correct” — it’s a composite metric reflecting the quality of found chunks and how well they answer the question.

≥ 0.7🟢 High — reliable answer
0.4 – 0.7🟡 Medium — review content
< 0.4🔴 Low — possible hallucination

How it’s calculated

The score combines two signals:

1. Rerank score (50%)

The cross-encoder evaluates how relevant the best chunk is to the question.

2. Vector similarity (50%)

Cosine distance between query embedding and retrieved chunks.

Both signals pass through a sigmoid function normalizing to 0-1.

Why sigmoid? Raw reranking and similarity scores have different ranges. Sigmoid normalizes them to a comparable scale.

Diagnosis: why is my confidence low?

Case 1: Information DOESN’T exist in KB

Signals: Low rerank + Low vector Action: Add documents covering the topic. Create FAQs.

Case 2: Information exists but is “diluted”

Signals: Medium vector + Low rerank Action: Reduce chunk_size or enable contextualization.

Case 3: Information exists but uses different vocabulary

Signals: BM25 finds nothing + Medium vector Action: Raise semantic_weight or add FAQs with user vocabulary.

Case 4: Too many irrelevant chunks competing

Signals: High vector but Low rerank Action: Raise MMR diversity or reduce top_k.

Quick method: Run the query in the Playground and look at breakdown: vector, BM25, merged, reranked. The weakest link tells you where to act.

Actions to improve confidence

Content actions (most effective)

  1. Add documents covering low-confidence topics
  2. Create FAQs for frequent questions (FAQ match = maximum confidence)
  3. Improve document quality: clear titles, H1/H2 structure

Configuration actions

  1. Enable contextualization for ambiguous chunks
  2. Adjust chunk_size per document type
  3. Calibrate semantic_weight
  4. Raise rerank_top_n if best chunks don’t reach final top

Advanced actions

  1. Review confidence_threshold: if set to 0.7 and most queries hit 0.5-0.65, content may need time to mature
  2. Resolve knowledge gaps: the gaps section shows exactly which questions lack good answers

The continuous improvement cycle

Query → Low confidence → Knowledge Gap detected
    ↓
Add FAQ / Document → Reprocess
    ↓
Re-test → Confidence rises → Gap resolved
+30%typical improvement adding FAQs
+15%improvement with contextualization
2-3 wkstypical maturation cycle

Monitor confidence from Analytics and use the Playground to diagnose specific queries.

Try Citai for free

Create your intelligent knowledge base in minutes. No credit card required.

Create free account

Related articles

Technology

Cross-Encoder Reranking: How to Improve Your Chatbot's Accuracy

Learn how cross-encoder reranking dramatically improves RAG response relevance.

Read article → Practical Guide

FAQ vs RAG: When to Use Each Strategy and How to Combine Them

Guide to deciding when to answer with direct FAQ vs full RAG search, and how to optimize the combined flow.

Read article → Practical Guide

RAG Playground: how to diagnose and fine-tune your knowledge base

Practical guide to using Citai RAG Playground: pipeline diagnostics, parameter tuning, A/B comparison and batch testing.

Read article →
Hybrid Search: How to Balance Semantic a... FAQ vs RAG: When to Use Each Strategy an...