You should start testing early in a software project because a mistake is cheapest to correct at the moment it is made. Every week it goes unnoticed, more work is built on top of it, more people come to depend on it and the person who made it remembers less about why. The point of testing early is less to find more faults than to find them while they are still small.
How much more a late fault costs
You will often read that a defect costs 100 times more to fix in production than at the requirements stage. The figure is usually credited to the “IBM Systems Sciences Institute”. In July 2021 The Register reported the work of Laurent Bossavit, who traced that citation back through a 1987 textbook to course notes from an internal IBM training programme. No underlying data is available to check, and any projects behind the figures date from 1981 or earlier.
A better-sourced version comes from Barry Boehm and Victor Basili, in an article called “Software Defect Reduction Top 10 List” published in the journal IEEE Computer in January 2001. Its first item says that finding and fixing a problem after delivery is “often 100 times more expensive” than doing so during requirements and design. The paragraph that follows is quoted far less often. The authors explain that they added the word “often” on purpose, that for small, noncritical systems the factor is “more like 5:1 than 100:1”, and that good design can reduce it even on large systems by confining most fixes to small, self-contained parts of the code.
So the honest position is this. Late faults generally cost more to fix, the size of the difference depends on the size of the system and how well it is built, and nobody can give you a single multiplier. The Register’s article adds that later studies are mixed on how large the effect is. The argument for testing early does not need a number, because the reasons are plain enough.
Why a late fault costs more
More has been built on it. A wrong assumption about how prices are calculated, found in the second week, changes one piece of code. Found after the system goes live, it has shaped the database, the reports and the invoices already sent to customers.
The context has gone. A developer who wrote something this morning can correct it in minutes. Six months later, somebody has to work out what the code was meant to do before they can change it.
Live data needs repairing too. Once real records have passed through faulty code, fixing the code is half the job. The invoices have to be reissued and the stock figures recounted.
More people are involved. A fault in a live system is reported by a user, investigated by support, scheduled into a release and explained to whoever was affected.
The most expensive faults of all are misunderstandings about what was wanted. The software does exactly what the specification said, and the specification was wrong. Testing the code never reveals this. Showing the work to the people who will use it does.
What testing early means in practice
- Test the requirements before any code exists. Write each requirement with worked examples: “a trade customer orders 40 units at the bulk rate and pays by account”. Walk through the examples with the people who do the job. A clickable mock-up of the screens serves the same purpose. Agreeing these checks in advance is also what makes a fixed price workable.
- Write automated tests alongside the code. Test-driven development is the strict form of this, where the developer writes the test first and then the code that makes it pass. Strict or not, the tests should exist from the first week and not be promised for later.
- Run every test on every change. With continuous integration, the tests run automatically when a developer submits work. A fault is reported within minutes to the person who caused it, while they still have the details in mind.
- Put working software in front of users every couple of weeks. This is the main purpose of the short cycles in agile development, described in our article on integrating quality assurance with agile development.
- Start with the awkward parts. Moving data from the old system and connecting to other systems are where the surprises are. Try the data migration with a full copy of real data early, and repeat it as the project goes on.
- Rehearse the release. Deploy to a test environment from the start, and restore a backup at least once before go-live.
The different kinds of test mentioned here are explained in our overview of software testing strategies.
When the project is a system you already have
“Early” means something different when the software already exists. Many older business systems have no automated tests and no document describing what they should do. For those, early means before the first change.
This is the order we follow when taking over a system someone else built. No real work on the system begins until the application has been built from its source code, a backup has been restored and one small, low-risk change has been released in a controlled way. Then automated checks are placed around the calculations, permissions and integrations that matter most, recording what the system does today. These are called characterisation tests. For each important workflow, a named person in the business confirms the recorded behaviour is right. Changes are rehearsed on a copy.
The reasoning is the same as for a new build. The cheapest time to learn that a backup will not restore is before you need it, and the cheapest time to learn that a change breaks the month-end report is before the change reaches the people who run it.
What early testing does not do
It is not free. Tests take time to write and to keep up to date, and a mock-up takes time to build. Boehm and Basili’s point about small systems cuts both ways: on a modest project, a light approach with frequent demonstrations may be all that is justified, and heavy process would cost more than the faults it prevents.
It does not replace a final check. The finished system still needs a full pass and acceptance by the people who will use it. The difference is that the final pass should find very little.
It does not mean testing everything equally. Effort belongs where a mistake would cost most, which in a business system usually means money, access to data and connections to other systems.
Checks to make on your own project
- Can you see the acceptance checks for the next piece of work before it is built?
- Has anyone in your business used working software from the project in the last month?
- Are there automated tests, and do they run on every change?
- Has the data migration been tried with a full copy of real data?
- Has a backup been restored?
If a project is several months in and most of these answers are no, the testing has been left for the end, and the faults are accumulating unseen. Assessing a project that has drifted, as in a project rescue, starts with the same step you can take yourself: ask for a demonstration this week of whatever exists, on real data, and judge the project by what you see.