In recent years, automated testing has become a trend in software development. In some cases its implementation has become a “tribute to fashion.” However, the introduction and support of automated tests is a very resource-intensive, and therefore an expensive procedure. The widespread use of this tool often leads to significant financial losses without any significant result.
How can one simple tool help evaluate the possible effectiveness of using autotests on a project?
WHAT IS DETERMINED AS A “EFFICIENCY” OF TESTING AUTOMATION?
The most common way to evaluate performance (primarily economic) is to calculate return on investment (ROI). As soon as the value of the ROI goes over 1 – the solution returns the funds invested in it and begins to bring profit.
In the case of automation, profit refers to savings on manual testing. In addition, the profit in this case may not be obvious – for example, the results of finding defects in the process of ad-hoc testing by engineers, whose time was freed up by automation. Such profit is quite difficult to calculate, so you can either make an assumption (for example + 10%) or omit it.
However, saving is not always the goal of automation. For a number of reasons, the speed of testing (both in terms of the speed of one test and the frequency of testing) can be critical for business.
Or the elimination of the “human factor” in the testing process. This is important when the accuracy and correctness of operations is critical for business. The cost of such an error can be significantly higher than the cost of developing and maintaining an autotest.
Why measure efficiency?
Measuring efficiency helps answer the questions: “should we implement automation on a project?”, “When will the implementation bring us a significant result?”, “How many hours of manual testing will we replace?”, “Can we replace 3 engineers of manual testing with 1 automated testing engineer ?” and etc.
These calculations can help formulate goals (or metrics) for the automated testing team. For example, saving X hours per month of manual testing, reducing the cost of a testing team for Y conventional units.
Help is needed? Request Gersis expert advice on test automation for your software product.
In order to form the data for evaluation, it is necessary to determine what will be covered by the automated tests. To highlight it, you should follow the next two simple rules:
- the covered functionality should be fairly stable (the more stable the functionality, the lower the costs of supporting the tests);
- covered functionality should be tested regularly and often (the more frequently the functionality is tested, the faster its automation pays off; rarely and irregularly tested functionality is not effective for automation – it is more profitable to test it manually);
- ability to test using autotests.
To calculate the automation efficiency of a project, a set of input data is required.
- First, the cost (in man-hours) of manual testing within the selected coverage.
- Secondly, the cost of developing autotests (in man-hours) autotests under this coverage.
- Third, data on the frequency of testing.
Usually the interval for analysis is a month. But it is quite possible to adapt it for a specific project – for example, take a sprint or weekly interval.
To begin, calculate the cost of manual testing for each interval. Please note that costs are reduced (replaced by autotests). Then we calculate the costs of developing tests.
It is necessary to take into account the estimated size of the automation team on the project – this will show how quickly we can implement autotests, replacing manual testing.
Then we calculate the sum of these costs. In fact – this is the total cost of testing. Add for comparison the cost of manual testing (without automation). Next, we calculate the accumulated testing costs with and without automation.
The difference between these values will show us the effectiveness of the automation process (saving labor costs) for a given period of time.
Let’s try to build a graph of this data.
Obviously, the test automation of these scenarios has a significant economic effect, which occurs quite quickly and surely increases with the time.
In general, this graph is practically canonical to demonstrate the effect of automation: the rapid increase in labor costs for manual testing with a slight increase in labor costs for automated testing.