Test Automation ROI: 5 Ways to Show the Business Benefits
Originally Published on TechBeacon 6/23/2017
Paul Merrill
CEO, Test Automation Consultant, Beaufort Fairmont
Done well, test automation can return multiples of the amount of time and effort invested. Setting expectations with stakeholders and business owners is key to helping ensure that everyone is moving toward the same vision of success. Just as important is setting expectations of returns and communicating with them after.
After working for years to help clients across the US develop great test automation, I have come up with five ways of defining test automation returns.
The business benefits of test automation
1. Faster delivery
One of the largest unaccounted costs in software development is the fixing of defects. Long delays between the development of code and the finding of a defect increase the time and effort it takes to fix the defect. Test automation can help. Creating a tighter feedback loop between the development of code and the testing of whether the code does what was intended is key. What used to take developers days to fix because they had to relearn their own code written months ago takes minutes when test automation is ready to run the moment code is complete.
Decreasing defect costs is one of many outcomes with test automation. Consider the cost of missing market opportunities because testing held up development. The opportunity cost there can be big enough to sink a company.
One way to measure faster delivery is by calculating total time of delivery before test automation and after. The total time must include all the time developers, product owners, customer service, testers, and anyone else spend on identifying, prioritizing, and fixing defects before test automation is created. Then compare that result to the result after test automation is used. Take the total number of hours and multiply by a reasonable average hourly rate, including benefits (usually 40% or more above salary).
In this and each item below, return on investment (ROI) will be the total return (the number we just found) minus the total invested, divided by the total invested. If you’re getting a number greater than 1, you’ve made money. If not, you’ve lost money so far.
2. Find more regressions
Finding regressions more frequently after the introduction of test automation is typical. The difference is that regressions are generally found so quickly and fixed so easily that they may not be reported as they would have been in the past. A developer may see a failed test in the continuous integration (CI) environment, recognize the issue, and fix it quickly, rather than documenting it. So counting found regressions may take a bit more effort.
Account for number of regressions before and after introducing test automation and assign a value to them. Averages are helpful. If you take the average cost of a regression to fix before automation and multiply by the count of regressions, then do the same for regressions after the introduction of test automation; the difference is the gain. The hope is that the average time to fix a defect will trend downward after some time and that the number of regressions found will trend up for a period of time.
While finding more regressions may be a higher cost to you now, the contrast in cost compared with users finding them is stark.
3. A more testable product
A funny thing happens when you commit to test automation across your product development process: People start creating more testable products. Imagine!
Organizations can define benefits both qualitatively and quantitatively. Ask seasoned test automation engineers if your product is highly testable. They’ll be able to respond right away. Many times in working with clients at Beaufort Fairmont, I get a sense of the product’s testability in a short phone call. Other times, I need to dive into the codebase. This is the qualitative aspect of this return.
The quantitative side can be found by answering the question, “How long does your product take to create a test for a feature?” If this time is trending downward over significant periods of time, you’re experiencing a gain in testability. When a product is more testable, you can go to market faster, test in more ways, and have less chance of missing use cases in your testing. Stakeholders and product owners end up loving more testable products because they have confidence in what the product does.
Generally, teams also get better test coverage and code coverage by supplementing testing efforts with test automation and eventually having a more testable product.
4. Shift left
When are test cases written? When were they written prior to introducing test automation? The closer they are to being created at the same time as when tasks are defined, the farther “left” your testing has shifted. Moving the activity of creating test automation and planning for it to the left also forces the clarification of features and user paths. Test automation written earlier means earlier detection of defects, which means faster delivery and delivery of a better product.
5. Less painful regression
For many organizations, regression is painful. It’s slow and it rarely finds significant issues. Test automation can help with this. Replacing manual checks with machine checks provides us with several benefits. First, we know the check was performed as it’s always performed. Software doesn’t vary its procedures. Second, we can run it more often for lower cost. Third, we can easily check whether a regression has happened at any time.
What’s the return on testers’ morale?
Repeating the same test cases over and over is mind-numbing for testers. Removing this can energize testers, fueling them to find the really costly defects.
Before and after test automation is in full swing, survey developers, testers, and product owners about how they feel about regression testing. How confident are they that regression testing is identifying regressions? How do they feel about the time they spent on regression? How much time did they spend on regression?
Additionally, the ability to run regression anytime with the press of a button and not having to wait days or weeks for results is a major time-saver and speeds delivery.
Measuring the ROI
The ROI in test automation can be huge. Most teams only consider the amount of time it takes to run a suite of tests in comparison to how long it took before test automation. There are many other benefits, both qualitative and quantitative. When making a case for test automation, keep in mind the many ways test automation can return on your investment.