QA KPI Metrics

I’ve been talking at several conferences and one of the questions that keep coming back when I talk about QA Automation are the metrics, specifically the QA KPI metrics. Everyone is interested in the kind of metrics to look at when measuring automation.

What kind of metrics are you considering in your automation? Let discuss a little bit about QA KPI metrics.

In general KPI metrics software quality is defined as followed:

      • the number of defects,
      • the number of blast rate,
      • or the number of failed tests cases,
      • the code coverage

Although this above information gives some important data, all related to testing activity, my approach to quality KPI really depends on the following questions:

    • Who is looking for the data?
    • Who is looking at the data?
    • What is the data going to be used for?
    • How are you able to make a decision about the data?

Based on the answers to each above question, I create a dashboard to make data accessible to everyone in the organization.

When talking about who? –

From my personal experience, there are three groups of people who are interested in the data for software quality to determine the health of the application that is under test.

The first group, usually the tester, is actually the one doing the testing work; the QA tester, the QA engineer, and the Quality engineer.

They just want to determine where they are at during the testing so they can see the progress and have a great confidence level about the testing.

The second group, usually the middle management, the direct team member manager (software quality manager or Quality leader, Dev manager, DevOps manager or Ops manager), and anybody that is supporting the application and managing a team that is supporting the application.

This second group is more interested in the progress of their application under test.

Finally, the third group, the upper management (your C suite, your VP of Technology, VP of product, CTO, CIO, and CISSO) are the one that really needs to look at the high level on a high-level view of the data.

This third group needs to know the following:

How is the application going? Is it healthy? can I sleep better at night and know that my application is bug-free. That nothing is going to happen in production,

especially around this time, the holiday, what is the performance of my application?

So, each of these groups mentioned above has different interests in the application. They have different interests in the quality of the application.

They also have different interests in the data they’re looking at.

So, you can’t mash up every data and give it to them.

You have to make sure that at each level, you provide some type of data that is really useful, usable for them. Obviously, if you are the VP of product and you want to dive in and really look at the bugs, you can feel free to do that, or if you are a Dev manager and you want to look at the bugs report you are free to do that as well. So that’s how I usually try to frame my quality KPI metrics. I also like to consider a different approach and promote the following as part of your Quality KPI metrics

    • Better Visibility
    • Clear transparency into the data
    • Direct accountability
    • Guidance for continuous improvement

For each audience member here are the possible QA KPI metrics to consider and the threshold to evaluate:

 For the QA team member, consider creating an internal QA scorecard

      • Active defects(new, open or fixed) — trending low
      • Authored Test cases: Number of test cases created – Threshold should be 1-1 to the number of user’s stories or 1-2
      • Covered Requirements: Percentage of test cases mapped to requirements – Threshold 80-100%
      • Defect fixed per day /per week: How fast is development fixing bug
      • Passed Requirements: Number of test cases passed for a requirement
      • Passed Tests: 80% threshold
      • Rejected Defects:  No more than 5%
      • Reviewed Requirements:  100% threshold
      • Critical Defects:  should not exceed 10

For Team Manager, the metrics should provide current progress so consider the following KPI metrics:

      • # USERS STORIES
      • #USERS STORIES PASSED
      • % of Test Passed
      • #Number of test cases
      • #Number of Passed
      • #Number of Failed
      • #Number of Test cases not Executed
      • Defects Metrics
      • #Open Defects
      • #CLosed Defects

      For the Leadership Team, the metrics should allow for a quick decision at first glance, so consider the following KPI metrics:

      • 80 % of test cases PASSED
      • % of user stories PASSED
      • Number of P1 and P2 bugs
      • Escape bugs in production
      • Product quality health (red, yellow or green)

We also define the product quality health; I have defined a KPI framework to determine the quality of your product using the following data and threshold to evaluate.

%Passed Test cases

% Escape bugs aka bugs found in production

% Passed User Stories (Pass Rate to stories Ratios).

% PASSED TEST CASES> 80% – GREEN EXCELLENT

% PASSED TEST CASES < 80 – 50% –YELLOW Satisfactory

% PASSED TEST CASES < 50% – RED Unsatisfactory

%Escape Bugs = #escape bugs / #functional bugs

% Escape bugs < 10 % – GREEN EXCELLENT

% Escape bugs > 10 -30 % – YELLOW – Satisfactory

% Escape bugs > 30 % – RED – Unsatisfactory

In summary, it’s important to determine who you deliver this KPI to; in order to make the best decision for your team. In essence to determine the quality and give you that high level of confidence that software has the best quality.

Comment below and tell me about how you determine KPI in your organization or what kind of challenges are you facing in terms of KPI metrics.