Regression testing is the practice of re-checking the parts of a system that were not meant to change, after something else has changed, to confirm they still work. A regression is a feature that worked yesterday and is broken today because of a change made somewhere else. The change might be a new feature, a bug fix, a database upgrade or a Windows update on the server.
It exists for a simple reason. Whoever makes a change tests the thing they changed. The damage usually appears somewhere they did not look.
What a regression looks like
Business software is full of shared code and shared data, so a change in one place can alter behaviour in another. Some typical cases:
- A developer changes how VAT is rounded on invoices. Invoices are now right. Credit notes use the same calculation, are now a penny out, and nobody notices until month end.
- A new field is added to the customer record. The nightly export to the accounts package expected the old layout and starts failing without telling anyone.
- The database server is upgraded. A report that depended on a quirk of the old version now takes ten minutes to run.
- A fix for one customer’s problem removes a workaround that another customer relied on.
In every case the change itself was tested and worked. The failure was in something nobody thought to check, which is the gap regression testing is meant to close.
How regression testing is done
There are two ways to do it, and most teams use both.
By hand. Someone works through a written list of checks before each release: sign in as each type of user, raise an order, post an invoice, run the month-end report. This costs nothing to set up. Its weakness is that it is slow and repetitive, so as the list grows it gets done less often, and steps are skipped when a release is urgent.
Automatically. The same checks are written as code and run by a machine. They come in different sizes. A unit test exercises one calculation or rule in isolation. An integration test checks that the application and its database, or an outside system, work together. An end-to-end test drives the real application through a browser the way a user would, using a tool such as Playwright or Selenium. Our overview of software testing strategies explains how the sizes fit together.
Automated tests take effort to write and almost none to run, so they can be run on every change. Most teams arrange for this to happen whenever a developer submits work, a practice called continuous integration, and set a failing test to block the change from being released.
Automation does not make the manual pass redundant. An automated test checks only what somebody thought to write down. A person using the system notices things nobody anticipated, such as a screen that has become confusing or a total that looks wrong.
Deciding what to test
Checking everything after every change is called full regression testing. Done by hand it is rarely affordable, so teams choose what to check by risk. In a business system the first candidates are:
- anything involving money, such as prices, tax, invoices, payments and payroll;
- who can see and do what, especially where several customers share one system;
- connections to other systems, such as imports, exports, payment providers and accounting packages;
- the few workflows the business cannot go a day without;
- whatever shares code or data with the thing that was changed.
One habit is worth insisting on. Each time a bug is fixed, a test is added that would have caught it. Fixed bugs have a way of returning, and over a few years this builds a set of tests concentrated on the places where your particular system breaks.
Regression testing a system that has no tests
Many business systems that have been in use for years have no automated tests at all. We see this often, because much of our work is taking over software that someone else built. There is frequently no document saying what the system should do either, and the person who knew has left. If you are not sure which position you are in, whether tests exist is one of the things a code audit reports on.
You cannot test a system against a specification that does not exist. You can still protect it, in four steps.
- Record what it does now. A characterisation test, a term from Michael Feathers’s book Working Effectively with Legacy Code, captures the system’s current behaviour without judging whether that behaviour is right. Feed in last month’s real orders, save the invoices that come out, and after each change run the same orders again and compare. Any difference is either intended or a regression, and someone has to say which.
- Name a person for each workflow. For every workflow that matters, find someone in the business who uses it and can confirm it still behaves correctly. Write their name next to it. Without that person, “correct” is a guess.
- Rehearse on a copy. Restore a recent backup into a separate environment, apply the change there and run the checks before touching the live system. This also proves the backup can be restored, which is worth knowing in its own right.
- Start where a mistake would cost most. Put checks around the calculations, permissions and integrations before any of them are changed, and extend the coverage with each later piece of work.
Characterisation tests have one awkward property: they preserve existing bugs along with everything else. That is deliberate. Staff may have built their routines around an odd behaviour, and it should be changed by decision, with the named person agreeing, and not by accident.
What it costs and what it saves
Tests are not free. They take time to write, and end-to-end tests in particular need upkeep, because they break when screens are redesigned. A suite that nobody maintains starts to fail for reasons unrelated to real faults, the team learns to ignore it, and at that point it gives false comfort.
What you get in return is the ability to change the system without holding your breath. Releases can be small and frequent. An upgrade away from an unsupported database or framework version can be verified, where otherwise it would be hoped for. Systems without regression tests tend to freeze, because every change feels dangerous, and a frozen system gathers technical debt until it has to be replaced.
Questions to ask whoever maintains your system
- What is checked before each release, and is that list written down?
- Which of those checks are automated, and do they run on every change?
- When a bug is fixed, is a test added so it cannot come back unnoticed?
- Is there a test environment with a recent copy of the data where a change can be rehearsed?
- Who in our business confirms that each key workflow still works?
- When did a test last fail and stop a release? If the answer is never, the tests may not be checking much.
If the honest answer to most of these is that nothing is in place, begin with the list. Write down the ten things the system must do correctly and who can confirm each one. That list is a regression test in its simplest form, it takes an afternoon to produce, and everything else in this article builds on it.