A RAG quality checklist before you publish a document chatbot
A research note on retrieval quality, citation behavior, freshness, and evaluation signals for teams shipping RAG workflows.
A document chatbot can look polished while still retrieving the wrong evidence. Before publishing a RAG workflow, test retrieval quality separately from answer fluency.
Check retrieval before generation
Run representative questions and inspect the retrieved chunks before the model writes an answer. If the source passages are weak, the final response will be weak even when the prose sounds confident.
Watch freshness and scope
Record when documents were indexed, which sources are included, and which sources are out of scope. Users trust a RAG system more when it can say what it knows and what it does not know.
Evaluate citation behavior
Ask the system questions where the answer should cite a specific policy, changelog, or technical page. Then verify whether the citation points to the right evidence rather than a loosely related document.
Keep a failure set
Save bad queries. A small failure set is one of the cheapest ways to improve a RAG system because it turns vague complaints into repeatable tests.