Articles Fundamentals

Regression testing for chatbots and voicebots

Why a bot can get worse without anyone touching its code, and how to notice before your customers do.

By b-nova · As of November 2026

Why bots get worse without a code change

Traditional software behaves the same as long as nobody changes the code. A bot built on a language model has more moving parts:

  • The language model. The provider releases a new version or retires the old one.
  • The prompt. An addition for a new case shifts the behavior in three old ones.
  • The knowledge base. A new document contradicts an existing one.
  • Connected systems. An interface returns a field differently, and the bot suddenly answers wrong.
  • For voicebots: speech recognition, voice and telephony have their own versions and their own sources of error.

None of these changes shows up in a code review. They only become visible in the conversation.

What a regression suite contains

A regression suite is a collection of test cases that together cover what the bot has to be able to do.

Type of test caseExample
Main requestsReport a claim, reschedule an appointment, reset a password
DeviationsThe user corrects themselves or gives an invalid answer
LimitsThe user asks something the bot is not allowed to answer
Past errorsEvery error from production, once fixed, stays on as a test

Each test case describes a scenario and the statements that must have been made by the end. It’s better not to fix the exact wording: a bot that phrases things freely would otherwise never pass the same test twice. More on this in the article Testing LLM chatbots.

When the suite should run

  1. Before every release. The suite is the gate: if cases that passed before fail, the change doesn’t go live.
  2. After a model or prompt change. Run the suite with the old and the new setting and compare the results.
  3. Regularly in production. A daily or weekly run catches changes that come from the outside.

Reading the results correctly

A single failure is not yet a regression. Language models don’t answer the same way every time, so a second look is worthwhile:

  • Does the case fail again on a repeat? Then it’s a real error.
  • Does it fail only sometimes? Then the behavior is unstable, and that is a finding too.
  • Do several cases on the same topic fail? Then the cause is usually a single change.

An evaluation that gives a verdict with a reason for each expected statement helps. That way you see not only that a case failed, but why.

Regression tests with gentesty

In gentesty, you create test cases as a scenario with expected statements. gentesty runs the conversations itself, by chat or by phone call, and evaluates each statement. Scheduled test runs and notifications in Slack, Microsoft Teams or Rocket.Chat are available from the Business plan, and the API for CI pipelines on the Enterprise plan.

Continue to chatbot testing or voicebot testing.

Frequently asked questions

What is a regression test for a chatbot?

A regression test repeats conversations that used to work and checks whether they still reach the expected outcome after a change. A regression is a case that passed before and now fails.

How many test cases does a regression suite need?

To start, the ten to twenty requests that make up most of your conversations are enough. After that, the suite grows with every error you spot in production: each one becomes a new test case.

How often should regression tests run?

Before every release and after every change to the prompt, language model or knowledge base. Because the environment can change too, a regular run is worth adding, for example daily.