What a test case consists of
In classic software, a test case is an input and an expected output. A chatbot holds a conversation over several steps and asks questions back. The test case therefore also has to define how the user answers those questions.
| Part | Answers the question | Example |
|---|---|---|
| Initial message | How does the user start? | “I’d like to reschedule my appointment.” |
| Context | Who is the user, what do they know? | Name, customer number, old and preferred date |
| End | When is the request done? | The bot has confirmed the new appointment. |
| Expected statement | What must have been said by then? | The bot states the new appointment, date and time. |
An example from start to finish
Request: a customer wants to reschedule a garage appointment.
- Initial message: “Hi, I need to move my appointment on Tuesday.”
- Context: You are Anna Example, customer number 48213. Your appointment is on Tuesday, May 12, at 9 a.m. You want to switch to Thursday, May 14, in the morning. If Thursday is not possible, you take Friday.
- End: The bot has confirmed a new appointment or explained that it cannot offer one.
- Expected statements:
- The bot asks for the customer number or another detail to identify the customer.
- The bot confirms a new appointment on Thursday or Friday, with a time.
- The bot says that the Tuesday appointment is cancelled as a result.
- The bot does not mention any cost for rescheduling.
The context also contains the fallback. Without it, the simulated user does not know what to say when the bot turns down Thursday, and the conversation ends somewhere.
How to word the expected statements is covered in the article How to test an LLM chatbot.
Which test cases to write first
The best test cases come from real conversations, not from the specification.
- The most frequent requests. Chat logs or customer service reports show how most conversations start. Ten to twenty requests often cover the largest share.
- One deviation per request. The customer corrects herself, does not know her customer number or changes her goal halfway through.
- The limits. Questions the bot must not answer, and cases in which it has to hand over to a person.
- Errors from production. Every complaint about the bot becomes a test case before the error is fixed.
How this turns into a suite that runs before every release is described in the article Regression testing for chatbots and voicebots.
Five mistakes that make test cases useless
- Several requests in one test case. If it fails, you do not know which request caused it.
- Too little context. The bot asks for the date of birth, the test case does not have one, the conversation gets stuck.
- No clear end. Without a finish criterion the conversation goes on until a limit kicks in, and the test fails even though the bot did everything right.
- Test data the bot does not know. A customer or contract that does not exist in the test environment leads to an error message instead of a conversation.
- The wording as the expectation. A bot that phrases freely never hits the same sentence twice.
Maintaining test cases
A collection of test cases is only as good as its order. Give every case a name that states the request and the variant, for example “Reschedule appointment: preferred day taken”. Group cases by topic, so that after a change to the appointment logic you can run exactly those cases. And delete cases when the request no longer exists: a test that has been ignored for months devalues all the others.
Test cases in gentesty
In gentesty, a test case consists of the initial message, the context, the finish criteria and the expected outputs. gentesty holds the conversation with your bot itself and answers its questions from the context. Tags group test cases, and many cases can be created and changed at once through Excel or CSV.
The getting started guide shows how to create your first test case. Continue to chatbot testing.
Frequently asked questions
How long should a test case for a chatbot be?
As long as a single request. One test case covers one goal of the user, for example rescheduling an appointment. If the user also wants to change their address in the same conversation, that is a second test case.
Do I need real customer data for the tests?
No. Use test customers that exist in your bot’s test environment, and made-up names. What matters is that the bot knows the data: a customer number that does not exist in the system makes every test fail.
Who should write the test cases?
Ideally the people who know the requests: the business team, customer service or the product owner. A test case made of a scenario and expected statements needs no programming skills.