December 27, 2011
How to calculate ROI for automation testing? Is this a question that crosses your head now and then? Well, here is the detailed explanation on the various ROI calculation formulas on test automation.
Test Automation is attractive, necessary and expensive too. The decision in front of the test manager is whether to invest in automated testing tools or continue with manual testing. In this article, let us look at a couple of ways of looking at ROI on test automation.
The most basic form of automation ROI calculation is Simple ROI calculation; however in this article we will discuss slightly advanced techniques – Efficiency ROI calculation and Risk Reduction ROI calculation.
The ROI calculation for automation testing needs to be done in terms of days since automated tests can be run for 24 hours continuously; as opposed to 8 hours that is considered for manual testing. However, 18 hours seems more reasonable since sometimes the test cases are interrupted and do not run for 24 hours.
The formula, which forms the basis of ROI calculation, is :
(a) Automated test script development time = (Hourly automation time per test * Number of automated test cases) / 8
(b) Automated test script execution time = (Automated test execution time per test * Number of automated test cases*Period of ROI) / 18
(c)Automated test analysis time = (Test Analysis time * Period of ROI) / 8
(d) Automated test maintenance time = (Maintenance time * Period of ROI) / 8
(e) Manual Execution Time = (Manual test execution time * Number of manual test cases * Period of ROI) / 8
Notes: Period of ROI is the number of weeks for which ROI is to be calculated. Divided by 8 is done wherever manual effort is needed. Divided by 18 is done wherever automation is done.
In this method, time investment gains are considered rather than monetary gains. Once gains and investment costs are calculated, we can insert the values of these variables into the formula and get the total Efficiency ROI.
In the Efficiency ROI calculation method in test automation, focus is on the actual efficiency of the automation and not just the money factor. This method does not require any sensitive information such as the hourly rate of tester, etc. However, this method makes several assumptions such as automated test cases fully replace the manual test cases (but in reality some automated test cases may need manual intervention) and that the manual testing counts only one manual tester.
In the Risk Reduction ROI calculation method for automation testing, automation benefits are independently calculated thus addressing the ROI concerns that were not addressed in the earlier approach. When automated testing is adopted, resources can be freed to do more productive tasks such as time in-depth analysis of application, exclusive random and negative testing, test design/development etc.
So, when test automation is adopted, increased coverage can be attained. However, when manual testing is solely relied upon it, can lead to undiscovered application bugs and post delivery bug fixes. Because of this, the quality of testing and the product is reduced, thus putting the organization at a risk of having to pay the penalty or undergo some loss. This loss needs to be considered in the ROI formula for test automation so as to calculate the gain in terms of money if a bug is discovered post delivery/implementation.
Thus Investment cost remains the same, but the gain is the monetary loss that the company may have to face if automation testing is not implemented. These values are then to be inserted in the same formula which will then provide you the ROI.The advantage of calculating ROI for test automation is that the problems in the above method are addressed and it deals with the positive effect of test coverage. However, there is some subjectivity in this method and high level of risk analysis is required along with difficulty in calculating loss of not doing test automation.
Since there are several factors that affect the ROI calculation on test automation, it is advisable to introduce improvements to the overall automation testing by introducing efficient testing automation tools, investing more time in the initial phase itself, using better reporting mechanisms, etc.
Also, there is no single accurate mechanism to calculate ROI for automation testing. It is up to the tester’s/management’s discretion to use the method that suits their needs and situation the best.
To Improve the ROI on Test Automation,