How does a chatbot answer questions about your documents?
A useful document chatbot first finds the right information, then writes an answer from it. This guide explains the common approaches without database jargon.
You ask, How many leave days do I have? The answer sits on page 18 of an employee handbook. A useful chatbot needs to find that page before it starts writing.
There are two common ways to help it. You can give the model the documents with the question, or search the documents first and send only the useful parts.
- Small document sets can go straight to the model
- Large collections usually need a search step
- Show the reader where the answer came from
Small document sets can go straight to the model
If you have a short handbook or a handful of stable files, the application can include them with each question. The model reads that material and writes an answer.
This is often the easiest prototype. It becomes slower and more expensive as the pile grows, and a large pile can make the right detail harder to notice.
Large collections usually need a search step
Retrieval augmented generation, or RAG, means the application searches first. It sends the model a few passages that appear relevant to the question.
The model then writes from those passages. If the search picks the wrong passage, the final answer may also be wrong.
| Option | Good fit | Trade-off |
|---|---|---|
| Send the documents | Small, stable collections | More text is sent with every question |
| Search first | Large or changing collections | Search may miss the answer |
| Combine both | Important questions with several sources | Takes more work to test |
Show the reader where the answer came from
Put the document name and page or section beside the answer. Let the reader open it.
Do not add a source link only for decoration. Check that the linked passage supports the actual sentence.
Respect the same permissions as the source
A chatbot should not reveal a salary file to someone who cannot open that file normally. Apply access rules during search, before text reaches the model.
Keep sensitive documents out of logs and test collections. Use sample files while building the first version.
Test questions with known answers
Write questions that have one clear answer, questions that need two documents, and questions with no answer in the collection.
A trustworthy bot should find the right passage, quote or summarise it accurately, and say when the documents do not contain an answer.
How we prepared this guide
This guide explains the topic in plain language using current primary documentation and practical examples. It does not claim hands-on benchmarking. Check the linked sources for current product details.
Facts reviewed
