In the past two parts of our QA automation strategy discussion, we explained organization tradeoff, technology stack, defining your vision, and determining your tools. In this last part, we will be focusing on defining your automation ROI and implementation of test cases.

It is crucial to define your automation ROI to build a positive case for transitioning your team to QA automation.

Transitioning to automation is a large investment, and to effectively achieve it, you will need a set of metrics and considerations to calculate your ROI. These may include:

Determine business requirements/user stories to be implemented
Understand test cases for each requirement
Determine percentage of test cases that can be automated
Asses test case complexity
Measure test cases from prior requirements needed for regression testing
Calculate total number of configurations that need to be tested.

The calculation of your ROI is based on the investments that you are making when building your automation sets. This will be a long process, but you might want to consider a proof of concept (POC) experimentation phase to prove a minimum ROI. This will build your team confidence in the success of this endeavor. Consider the followings in your ROI calculation:

The capability and knowledge of each automation engineer. How long it might take different engineers to understand and script test cases can vary. It is good to define an average when considering the time and the complexity of the test cases.
Define a baseline for how long it takes to execute your test cases manually
Once you define the set of test cases, your simple ROI can be calculated with the following: ROI = Gains – Investments / Investments

It is usually recommended to start your automation build up with your regression test cases. Regression testing is the process of running older tests to ensure that new updates to a piece of software have not introduced or re-introduced previously eradicated bugs. For this reason, regression is important to the success of your software delivery. It makes sense to focus your automation on your regression suites, as those repeatable test cases that you need to run all the time. You usually know how long they will take you to execute them successfully. Spend your time automating those first. Then, your gain will be calculated by how long it takes to run your automation for that suite. For example, if your regressions suites take you 16 hours manually to run successfully, and your successful automation now lasts 16 min. But what if it took your QA engineer about 60 hours to implement the automation? Then, your ROI is about 73%.

Secondly, you have to consider calculating your ROI for new test cases. For new test cases, you have to ensure that the QA automation engineer has a great understanding of the feature in the test to build out the automation immediately. In this case, you may not have any baseline of manual test cases to calculate any gain. Your gain is the speed of adding these new sets to your regression suites. I usually don’t encourage this for a new team experimenting with QA automation, but for mature teams, this may work after you have successfully tackled all of your regression test cases, sanity tests, and smoke tests.

Most of the organizations that transition into QA automation usually set a grand vision and goal for automation. It is important to have a goal the team can work toward. Manual testing will continue to occur and should be your first attempt in testing for your application, though some scenarios might not be able to be automated. Therefore, an automation goal of 60 – 80% is recommended.

Another consideration is test case complexity. It can be discouraging to focus on more complex test cases at the beginning. It is not recommended to explore automation with test cases that are manually complex first. They are prone to error and require a huge amount of maintenance for the future.

With automation, some test cases can be reusable. Therefore, the ROI can increase as your suites grow since you will be using some of your old test cases to execute new ones.

Now consider the cross-browser testing when you have multiple platforms to test one. With an automation set, you will be able to run your tests on many platforms in parallel. There can also be a massive gain to coordinating all your tests (smoke, sanity, manual, automated, and load testing) in parallel, increasing your ROI as well.

Defining your QA automation strategy is an integral part of your automation transition. This process requires various mindset shifts, practices, and considerations into organization tradeoffs, your technology stack, their automation vision, your tools, your ROI, and finally, your implementation. Calculating your ROI will help others in your organization and your team to build confidence in the automation implementation to understand potential benefits and gain. Gain will continue as the implementation evolves, as well as the investment to maintain test cases.

This concludes our series on QA automation. We can’t wait to bring you more information on how to think of quality differently with agility.

Feel free to comment and give your feedback below. Zsquare Solutions Inc specializes in supporting you and your team to define all the important pieces needed to craft your QA automation strategy. Schedule a 30 min consultation with us today so we can learn more about what you do and how we can support you in your goal.

Resources :

https://smartbear.com/resources/ebooks/6-ways-to-measure-the-roi-of-automated-testing/