Practical guide · AI Integration for SMEs
How to evaluate an internal knowledge assistant
Evaluate an internal knowledge assistant against the tasks people actually need to complete. A fluent answer is not enough: check whether the right material was retrieved, the claims are supported, permissions are respected and the assistant handles missing or conflicting information appropriately.
Leap Dynamics GmbHUpdated:
1. Define the task and a comparison point
Choose a bounded use case, intended users and approved information sources. Consider whether existing search or a conventional workflow already solves it adequately. Establish a baseline for the current process before selecting a bespoke system.
Create representative questions with domain reviewers. Include common questions, difficult cases, outdated documents, conflicting instructions and questions the available material cannot answer. Keep evaluation cases separate from examples repeatedly used to tune the system.
2. Inspect retrieval and source support
Inspect the passages as well as the final answer. A clickable citation is useful only if its contents support the statement and the user is authorised to read it.
| Check | Evidence to examine |
|---|---|
| Retrieval | Are the relevant passages found for different wording and document formats? |
| Grounding | Does each material claim follow from the cited passage, rather than merely sharing its topic? |
| Currency | Can the system distinguish current guidance from superseded versions? |
| Uncertainty | Does it state missing evidence and request clarification where needed? |
3. Test permissions and hostile content
Test each user role against material it may and may not access. Apply access control during retrieval and when opening source documents. Include attempts to obtain restricted information through paraphrases or follow-up questions.
Treat retrieved text as untrusted data. Evaluate prompt-injection examples and attempts to turn document instructions into system actions. Limit available tools and permissions, and define a human route for consequential decisions or exceptions.
4. Agree acceptance measures
Set thresholds with the business owner before judging a candidate. Record the question set, source versions, configuration, model/provider and review method so comparisons can be repeated. These are proposed evaluation measures, not achieved results for Leap projects.
- Supported-answer and retrieval quality on the agreed question set.
- Appropriate handling of unanswerable or conflicting questions.
- Access-boundary failures and unsafe actions.
- Reviewer correction effort, task completion, latency and operating cost.
5. Make evaluation part of delivery
Begin with a bounded user group, clear feedback and an escalation path. Changes to documents, permissions, retrieval, prompts or models can alter behaviour; repeat the relevant checks after those changes.
Select cloud, private-cloud or on-premises deployment against data and operating requirements. Agree who maintains sources, reviews failures and owns updates. A pilot can establish whether the workflow is useful before a wider integration is commissioned.