"We want a chatbot that reads our company policies and answers questions." The standard way to build this is RAG. Getting started is easy, but reaching usable quality hinges on a few points. Here are five.
Short for Retrieval-Augmented Generation. When a question comes in, it first finds the relevant parts of your documents (retrieval), then gives them to the AI as evidence to write the answer (generation). It keeps the AI from inventing things it doesn't know about your company.
1. Are the documents up to date?
AI is only as smart as the documents it gets. If the old and revised versions of a policy are both in there, it may answer with the old one. Start by assigning an owner and revision date to each document and setting up a flow to re-ingest changes.
2. How do you split documents?
Long documents are stored as small chunks. Too small and context breaks; too large and irrelevant parts come along. Splitting along the document's own structure — articles and headings — usually works best.
3. Do answers cite sources?
- Show which document and article each answer referenced
- When nothing is found, answer "I couldn't find this in the documents"
- Let users open the source text right away
When sources are visible, people can check correctness immediately, which makes a huge difference in whether AI answers can be trusted.
Want to check this against your own data?AI Readiness Assessment · we review your material and map candidates and orderContact us4. Who can see which documents?
Some documents — HR policies, payroll — shouldn't be visible to everyone. Search itself must only find documents the asker is allowed to see. Adding this later means restructuring, so design it in from the start.
5. How do you measure quality?
"It seems to work" isn't something you can improve. Collect questions employees actually ask along with correct answers to build a test set, and compare scores on the same test whenever documents or settings change.
| What to check | Question |
|---|---|
| Retrieval | Did it fetch the document containing the answer? |
| Answering | Did the answer match the retrieved document? |
| Unknowns | Did it avoid inventing what isn't in the documents? |
If your documents can't leave the company, retrieval and the AI model can both run on your own servers with the same architecture.
In a one-hour AX consultation we map AI candidates and similar cases for you.
We build web apps, mobile apps and business systems, then add and run the AI they need. We build and operate our own meeting notes, contract and company analysis AI.
See our projects