RAG Playground: how to diagnose and fine-tune your knowledge base

What is the RAG Playground?

The RAG Playground is Citai’s diagnostic and tuning tool. It lets you test queries against your knowledge bases and see exactly what happens at each stage of the search pipeline — from vector search to the LLM response.

Unlike normal chat, the Playground doesn’t consume your query quota and shows information that’s normally hidden: internal scores, intermediate stages, execution times, and smart suggestions.

The three Playground modes

The Playground has three operating modes, accessible from the buttons in the top-right corner:

1. Single Mode

The default mode. Enter a query, select one or more KBs, and the system shows the complete pipeline:

• Diagnostics panel: Confidence, detected language, cache hit/miss, and timings per stage
• Content alerts: Whether your content rules would block or filter this query
• KB Health: Health score for each selected KB (0-100)
• Smart Routing: With multiple KBs, shows which ones would be auto-routed to
• Smart suggestions: Tips with “Apply” button to adjust parameters
• LLM preview: Optional — generates the actual response with token count
• 5 pipeline stages: Vector, BM25, Merge RRF, Reranking, Final MMR

2. Batch Mode

Test up to 10 queries simultaneously for an aggregate view of your KB:

• 5 summary metrics: Total, average confidence, min-max range, low-confidence queries, FAQ matches
• Detailed table: Each query with its confidence, sources found, FAQ match, and status (color indicators)
• Best use: Before putting a KB in production, test the 10 most frequent user questions

3. A/B Comparison Mode

Test the same query with two different configurations side by side:

• Config A / Config B: 8 independent sliders for each configuration
• Parallel results: Confidence, per-stage counts, timings, and suggestions for each config
• Best use: “What happens if I increase candidate_k from 30 to 60?” — see it instantly without modifying your KB

Diagnostics panel: what each data point means

When you run a test in Single mode, the top panel shows:

Confidence

The confidence score (0-100%) indicates how well your documents cover the query. It’s calculated by combining:

• 50% reranking score: How relevant the chunks are according to the cross-encoder
• 50% vector similarity: How close the query embedding is to the chunks

Detected language

The system automatically detects whether the query is in Spanish, English, or Portuguese. If you have a forced language configured in the KB, it’s shown alongside the detected one. Language affects the LLM system prompt.

Cache

Shows whether this exact query (or a very similar one, >= 95% similarity) already has a cached response. If it says “HIT”, the system in production would respond instantly without using the LLM — saving tokens and time.

Timings per stage

Each pipeline stage shows its time in milliseconds:

• embedding: Time to generate the query vector (~10-30ms)
• cache_check: Semantic cache lookup (~5-15ms)
• faq_check: FAQ match search (~10-20ms)
• pipeline: Full search pipeline (~50-200ms)
• llm: LLM response generation (~500-3000ms)

Content alerts

If your KB has content rules configured (BLOCK, REDIRECT, or FILTER), the Playground shows exactly what would happen:

• BLOCK/REDIRECT: Red alert indicating the query would be blocked and what response would be sent
• FILTER: Amber note indicating how many chunks were removed for containing blocked terms

This lets you test your content rules without affecting real users.

KB Health and Smart Routing

Health Score

For each selected KB, the Playground shows its health score (0-100) with a grade (A-F). The score combines 5 components:

Component	Weight	What it evaluates
Documents	25%	Number of processed docs
FAQs	15%	FAQ coverage
RAG Quality	30%	Average confidence of recent queries
Gaps	15%	Pending unanswered queries
Freshness	15%	Document age

If a KB has a low score, the Playground helps identify why.

Smart Routing

When you select multiple KBs, the Playground shows the relevance score Smart Routing would assign to each one. This lets you verify queries go to the correct KB.

How to read the 5 pipeline stages

Stage 1a: Vector Search

Finds the most semantically similar chunks using embeddings (pgvector). Score ranges 0 to 1 — higher means more similar in meaning.

What to look for: If the query is semantically clear but has no results, documents may not be processed or content is very different from query language.

Stage 1b: BM25 Search

Finds exact keyword matches using PostgreSQL full-text search. Useful for technical terms, error codes, and proper nouns.

What to look for: If BM25 finds nothing, try rephrasing the query with exact words from the document. If your content is in Spanish but the query is in English, BM25 won’t find anything (but vector search will).

Stage 2: Merge RRF

Reciprocal Rank Fusion combines rankings from both searches. The bm25_weight parameter controls how much weight each gets (0.3 = 30% BM25, 70% vector).

Stage 3: Reranking

A cross-encoder (NLP model) evaluates each query+chunk pair together. Slower but much more accurate than vector search alone. The reranking score is the most reliable.

What to look for: If the reranker significantly reorders results (merge tops drop and others rise), it’s a sign it’s refining relevance well.

Stage 4: Final (MMR)

Maximum Marginal Relevance selects final chunks balancing relevance and diversity. The lambda_param parameter controls this balance.

What to look for: If all results come from the same document, lower max_per_doc or lambda_param. If there are few results, increase candidate_k or top_k.

Smart suggestions

The Playground generates automatic suggestions based on results:

Indicator	Meaning	Suggested action
Green	High confidence	All OK, no changes needed
Amber	Medium confidence / few results	Adjust parameters or add docs
Red	Low confidence / no results	Add documents or create FAQ
Blue	Informational	Reranker reordered results (normal)

Each suggestion with parameters shows the current and recommended value, with an “Apply” button that saves the change directly to the KB. You can also use “Apply all” to apply multiple changes at once.

Practical flow: “My KB gives low confidence”

1. Open the Playground and select the problematic KB
2. Test the query that’s causing problems and observe:
   • Any vector results? If not → docs aren’t processed
   • Any BM25 results? If not → query doesn’t have words from the document
   • Does the reranker lower scores significantly? → Chunks aren’t truly relevant
   • Few final results? → Increase candidate_k
3. Review suggestions and apply the ones that make sense
4. Use Compare mode to verify impact: Config A (current) vs Config B (with changes)
5. Test in batch with the 5-10 most common user questions
6. If average confidence goes up → apply changes to the KB

Key parameters and when to adjust them

candidate_k (default: 30)

How many initial candidates to retrieve. Increase if you’re not finding enough results. Decrease if responses are slow.

bm25_weight (default: 0.3)

Keywords vs semantics weight. Increase for technical content with exact terms. Decrease if BM25 introduces noise.

lambda_param (default: 0.6)

Relevance vs diversity balance in MMR. Increase (toward 1.0) to prioritize precision. Decrease (toward 0.3) for more document diversity.

min_confidence (default: 0.25)

Minimum threshold for the LLM to use sources. If confidence is below this, the LLM says “I don’t have information.” Decrease if the system rejects queries it should answer.

temperature (default: 0.3)

LLM creativity. For technical documentation, keep low (0.1-0.3). For conversational content, raise slightly (0.4-0.6).

Pre-production checklist

Before putting a KB in production, use the Playground to verify:

• [ ] The 10 most frequent questions give confidence >= 60%
• [ ] No false positives in content rules
• [ ] Cache is enabled and works for repeated queries
• [ ] Smart Routing sends queries to the correct KB
• [ ] KB has Health Score >= 60
• [ ] Batch mode with real questions gives average confidence >= 50%
• [ ] Most important FAQs give match (score >= 0.75)
• [ ] Pipeline times are below 500ms

Common mistakes

“I uploaded 100 PDFs but confidence is low”

Probably a chunking issue. Very large documents with 1024-token chunks may lose context. Solutions:

1. Enable chunk contextualization in the KB config
2. Lower chunk_size to 512 for dense documents
3. Increase chunk_overlap to 200 to avoid losing info at boundaries

“BM25 finds nothing”

BM25 uses exact keyword search. If your documents are in Spanish but the user asks in English, BM25 won’t find anything. Vector search does understand cross-language, so lower bm25_weight.

“Cache never hits”

Semantic cache requires >= 95% similarity between queries. If every query is very different, there won’t be hits. This is normal for KBs with varied queries. If you want more hits, the threshold can be adjusted in code.

“Smart Routing sends to the wrong KB”

Routing is based on centroids (average of KB embeddings). If a KB has very varied content, its centroid is diffuse. Solution: split the KB into more specific topics.

---

The Playground is your primary tool for understanding how Citai processes each question. Use it before going to production and whenever you detect quality issues.