A colleague asks the new AI search tool, “Can I approve this purchase myself?” It replies confidently and links to a policy. Everyone moves on.
But which policy? Was it approved, or still a draft? Did it apply to that colleague? And did the linked passage actually support the answer?
Those questions belong in acceptance testing, before people start relying on the tool.
They also give business analysts and delivery teams a practical way to turn AI requirements into observable behaviour.
The starting point is a business decision with an agreed answer, a known set of documents and clear access rules.
Agree what a correct answer means
Choose a task that matters to the people who will use the service. Ask the relevant business owner to identify the authoritative sources, explain any exceptions and agree what the tool should do when evidence is missing.
Record this before running the demonstration. Otherwise, a convincing answer can quietly become the benchmark against which it is judged.
For each test, capture the question, user role, relevant document versions, expected answer and acceptable fallback. Leave space for the actual response, retrieved passages, citations and reviewer’s decision.
Include everyday wording. Someone might ask, “Who signs this off?” rather than use the terminology in the policy.
Put the wrong document within reach
Here is a fictional exercise, not a customer example or a reported test result.
A purchasing team has three documents. The current, approved policy requires manager approval for purchases over $2,000. A superseded policy sets the threshold at $5,000. A draft proposes $3,000 but has not been approved.
Ask: “I need to purchase equipment costing $2,500 today. Do I need manager approval?”
The expected answer is yes, with evidence from the current policy. Using either of the other thresholds is a failure, even if the answer is well written and includes a working link.
Do not resolve this by assuming the most recently uploaded file must be correct. Approval status, effective dates and the business owner’s rules determine authority.
Then ask a historical question with an explicit date. The superseded policy may be the right source for that question. A useful test checks whether the system understands which policy applies, rather than simply suppressing every older document.
Ask the same question as someone else
Repeat the exercise using separate test accounts with different permissions. Use synthetic documents in a controlled environment, not confidential records introduced merely to make a demonstration realistic.
For this exercise, give one account access to the purchasing policy and deny another account access to all three documents. The second account should not receive the restricted thresholds through an answer, preview, citation or conversation summary.
Checking whether the source link opens is insufficient. The answer itself may already have disclosed the information.
Next, remove access from the authorised account and repeat the tests. Agree beforehand how quickly that change must take effect, then record what happens within and after that period.
Test new questions and follow-up questions in an existing conversation. Review how saved responses are handled under the organisation’s retention and access rules. Removing permission cannot make someone forget information they have already received.
Read the evidence, not just the answer
Open the cited passage and compare it with each important claim. Does it state the approval threshold? Does it establish who approves the purchase? Is the response adding an exception that the document never mentions?
A citation is a route to evidence, not proof that the answer is supported.
Add a question the documents cannot answer: “Who approves emergency purchases when my manager is away?” In this fictional set, no document defines that exception.
The expected response should acknowledge the missing evidence and direct the user to an agreed escalation point. Inventing a substitute approver is not helpful completion.
Where diagnostic evidence is available, separate retrieval from generation. Did the system find the wrong passage, fail to retrieve the right one, or misinterpret evidence it had already found? These failures need different remedies.
Make the result useful to a release decision
Have business reviewers assess relevance and correctness, and security reviewers assess access behaviour. Repeat important cases with different wording. Keep some questions out of tuning so the final check is not limited to examples the team has repeatedly adjusted the system to answer.
Compare useful task completion with the existing search process too. An AI answer that needs more checking than a conventional result may not improve the work.
Report failure types separately. A high overall score must not conceal a restricted disclosure or an unsupported approval decision. Agree which failures block a pilot, which require remediation and which can be managed through a clearly limited scope.
NIST’s Generative AI Profile cautions that pre-deployment tests may not reflect real deployment conditions. This small exercise is a starting point, not a security assessment or certification.
Keep the tests after the demonstration
Save the reviewed test pack with the source versions and system configuration. Run it again when policies, permissions or the search configuration change, and add failures reported during any controlled pilot.
Record the time, document versions and active permissions for each run, so another reviewer can reproduce the conditions and understand any delay before updates appear.
The most useful outcome is a record that lets someone explain why a particular answer was acceptable, where the system fell short and what must happen before more people depend on it.