When we build an assistant for an organisation, the first part moves fast. It connects to the documentation, handles the sample questions well, and everyone is pleased. The second part — which usually takes twice as long — is defining the boundaries.
Confidently wrong is the worst outcome
An assistant that says "I am not certain, let me connect you to a colleague" is manageable. One that confidently quotes the wrong price harms the customer and destroys trust in the whole system.
So in our evaluations, a confident wrong answer is weighted more heavily than an "I don’t know".
Where the boundaries come from
You do not guess a boundary. It comes from real data: we classify past messages and decide with the team which categories must never be handled by an assistant.
- Anything with legal, financial or health consequences.
- Cases where the right answer needs human judgement, not information.
- Topics with no reliable, current documentation behind them.
- Complaints and emotionally charged situations.
Evaluation before trust
Before an assistant enters the main path, we build an evaluation set from real questions — including ones whose correct answer is "I don’t know". Until results are stable, it stays in suggest-to-a-human mode rather than answering directly.
Evaluation set — three categories
1. Answerable -> must answer correctly, citing a source
2. Boundary -> must hand over to a person
3. Out of scope -> must state plainly that it does not knowAn assistant that answers half the questions and correctly routes the rest is far more valuable than one that answers everything and gets ten percent wrong.