Testing forms the eyes and ears of a project – providing invaluable information to inform decisions and deliver high-quality software. Unfortunately, this rich vein of information often lies untapped – instead, poor testing metrics are shared that only serve to bamboozle or waste time.
Different metrics are usable by different people for different purposes. Identifying which information is helpful and worth sharing can be daunting for test professionals, who are focused on getting through their tasks and perhaps don’t look at things from a Project Manager’s or Stakeholder’s perspective.
To make matters worse, many Stakeholders and PMs don’t know which information is accessible or possible or understand how long it takes to produce useful metrics. This means great information is being lost in the middle – testers don’t know it’s useful, and management doesn’t know it’s available.
This insight delves into testing metrics, why they’re important, why they can be less relevant, and provides appendices full of good and bad testing metrics to pull from.
Why Effective Testing Metrics Are The Key to Creating High-Quality Software
The landscape of software testing is complex and diverse. There is no one-size-fits-all approach to selecting the best metrics for your project.
By understanding the implications and value offered by different metrics, you can make informed decisions that align with your project goals and contribute to the overall success of your software product.
I’ve listed several useful metrics in the appendices at the bottom of this article. These will help you deliver helpful test management, coverage, testing quality, and testing efficiency and cost metrics.
These metrics are more than just numbers – they are valuable insights that can help you pinpoint areas of improvement, optimise your testing process, and deliver a high-quality product.
I also list a few poor metrics less likely to improve testing quality directly. While these metrics provide some insight in some edge cases, focusing on them will divert attention from the more actionable and impactful metrics.
As you progress in your software testing journey, take these metrics as guiding lights, adapt per your project needs, and remember – the ultimate goal is delivering software that meets your users’ needs and expectations while ensuring reliability and performance.
Easy Access to Effective Metrics: Why You Need Test Management Tools
Effective testing metrics are undoubtedly important to software projects, but they need to be easily accessible to anyone who needs them.
Software testing metrics can technically be managed without test management tools, but it’s highly inefficient and prone to errors.
Software implementations are already highly pressured and up against tight timelines. No project can afford to waste days of effort collating test data, especially not when modern test management tools provide instant access to this information.
Automated collection and reporting
Test management tools allow for automated and real-time collection and reporting of testing metrics. Without these tools, you would need to gather and compile all the relevant data manually, which is prohibitively time-consuming and prone to mistakes.
Standardisation
Test management tools enforce standardised metrics, which is important for consistency, especially when different testers or teams are involved.
Traceability
Test management tools provide traceability by linking requirements, test cases, and defects, making tracking the coverage and effectiveness of tests easier. You just cannot achieve this with spreadsheets.
Access to Advanced Metrics
Modern test management tools include more advanced metrics, such as prediction and trends analysis, which can be difficult, if not impossible, to calculate manually.
Centralisation
By providing a centralised place for managing all aspects of testing, these tools significantly increase efficiency and reduce the risk of missing or duplicating work. This is increasingly important in hybrid/WFH project environments. Also, if team members roll-off, their work is readily accessible to the rest of the team and new team members.
Scalability
As the size of a project increases, manually handling testing metrics becomes an absolute nightmare. If you’ve tried it, you know how many headaches it causes. Test management tools allow projects to scale smoothly without an increase in complexity or workload for the testing team.
Give Your Software The Best Chance of Success
When selecting test management tools for accessing effective test metrics, OpenText Quality Center (Waterfall/traditional projects) and Value Edge Quality (Previously ALM Octane, and used for Agile/DevOps) are great options and should always be on your shortlist.
These class-leading test tools give your software the best possible chance of success and provide detailed insights and comprehensive dashboards to track your testing metrics.
These tools provide instant access to the status and progress of your testing activities, making them powerful tools for managing your testing processes.
By leveraging these tools, you will gain deeper insights, foster better collaboration, and make better-informed decisions throughout the SDLC.
Appendix 1 – Useful Testing Metrics
Effective testing metrics provide a quantitative way of measuring various aspects of your software product or process. They offer invaluable insights that guide project management decisions and support successful outcomes.
The importance of these metrics depends on the specific context and goals of your project. Therefore, it’s essential to choose and track the ones that provide the most value to your testing processes.
Useful Metrics for Effective Test Management
| 1. Test Execution Progress | The number of planned versus executed test cases gives a clear picture of the progress of the testing process. This metric can help identify any delays or blockers in the testing phase. |
| 2. Test Run Repeat Rate | A high repeat rate could indicate issues with test design or execution and might suggest you are wasting resources on ineffective testing. |
| 3. Failed Deployments | Tracking unsuccessful deployments and understanding their reasons can help improve future deployment strategies and reduce downtime. |
| 4. Test Environment Stability | Stability issues in the test environment can lead to unreliable test results. This metric allows you to identify and rectify such cases to ensure more reliable and efficient testing. |
| 5. Percentage of Automated Test Cases | It clearly shows how much of the testing process is automated, which can provide insights into potential areas for increasing efficiency and coverage through automation. |
| 6. Requirement Traceability Matrix (RTM) Coverage | Helps to ensure that all requirements have been thoroughly tested and can identify gaps in the testing process. |
Useful Metrics for Optimising Test Coverage
| 1. Test Coverage Percentage | Low coverage might indicate areas of your software at risk due to insufficient testing. |
| 2. Requirements Coverage | Low coverage could indicate potential functional gaps in your software. |
Useful Metrics for Driving Testing Quality
| 1. Defect Density | It measures the number of confirmed defects per code unit. A high defect density can indicate problem areas in your code that require additional focus and testing. |
| 2. Test Case Success Rate | This shows the percentage of passes. A low success rate might indicate software functionality or quality issues. |
| 3. Defect Acceptance Rate | The rate of acknowledged and fixed defects to total reported defects. A low acceptance rate might indicate miscommunication or disagreement between the testing and development teams. |
| 4. Defect Status and Details | Tracking defect severity and status gives a detailed overview of software quality and areas that require attention. |
| 5. Defect Age | How long a defect remains open in the system. Long-standing defects might indicate issues with the defect remediation process. |
| 6. Defect Trend Over Time | Observing the rate at which defects are found and fixed can provide insights into the effectiveness of your testing and remediation processes. |
| 7. Percentage of Critical Defects | Shows the severity and impact of the bugs found. A high rate of critical defects might indicate severe issues with the software. |
| 8. Defect Leakage | Measures the number of defects found post-release. High defect leakage might indicate problems in your testing or release processes. |
Good Metrics for Testing Efficiency and Budget
| 1. Mean Time to Detect (MTTD) | How long it takes to find a defect on average. A long MTTD might indicate inefficiencies in the testing process. |
| 2. Mean Time to Repair (MTTR) | How long it takes to fix a defect on average. A long MTTR might suggest issues in the defect remediation process. |
| 3. Test Efficiency | Number of defects found per hour of testing. Low efficiency could indicate issues with your testing process or tools. |
| 4. Cost of Finding and Fixing Defects | Helps in budgeting and optimizing the testing process. High costs could indicate inefficiencies in your testing or remediation processes. |
| 5. Code Complexity | High code complexity can make testing and remediation more difficult and time-consuming. Understanding and managing code complexity is essential to ensure effective testing and maintainability. |
Appendix 2 – Poor Testing Metrics
While tracking metrics is critical, it’s equally important to recognise which metrics may not provide significant value.
It’s important to understand that while all metrics can provide some insight, the key to effective testing is to focus on metrics that provide actionable insights to improve your testing processes and software quality.
Poor Metrics for Tracking Test Process
| 1. Number of Test Cases Written | Simply counting test cases doesn’t tell you anything about the quality or effectiveness of the tests. It’s better to focus on test coverage and results. |
| 2. Time Spent on Testing | The time spent on testing does not necessarily equate to thorough or effective testing. Focusing on the quality of the tests conducted is more critical. |
| 3. Number of Test Runs | Simply running tests multiple times is not beneficial if the quality or coverage of the test is poor. The outcomes and improvements made based on those test runs matter more. |
| 4. Number of Test Environments | Having multiple test environments does not ensure thorough testing if those environments are not stable or efficient. The focus should be on the reliability and relevance of these environments. |
| 5. Time Taken to Write Test Cases | The time taken to write test cases does not equate to their quality. Focusing on how comprehensive and effective these test cases are is better. |
| 6. Number of Test Tools Used | Test tools are important, but you using increasing amounts of them does not necessarily equate to effective testing. More tools can muddy the water and lead to worse quality. The suitability and effectiveness of these tools are more important. |
| 7. Amount of Test Code Written | Writing more test code doesn’t guarantee better testing. It’s the quality and coverage of the test code that matters. |
Poor Metrics for Assessing Software Quality
| 1. Total Number of Defects Logged | Without considering the severity or status of the defects, the total number alone does not provide a clear picture of quality. |
| 2. Lines of Code (LOC) | The number of lines of code is not a direct measure of software quality. More lines could mean verbose or redundant coding. |
| 3. Number of Code Commits | The frequency of code commits does not necessarily reflect improvements in quality. It’s the quality of the changes that matters more. |
Poor Metrics for Monitoring Effort and Resources
| 1. Size of the Testing Team | A larger team does not guarantee better testing. The skills, experience, and efficiency of the team are what matters. |
| 2. Hours Spent in Training | Time spent in training does not necessarily translate into better testing skills or more effective testing. |
| 3. Amount of Test Data | Large amounts do not ensure comprehensive testing. Instead, focus on the relevance and variety of the test data. |
Poor Metrics that Can’t be Neatly Categorised
| 1. Tester Experience | Experience is essential but, on its own, doesn’t always equate to better testing. Testers need the right skills and knowledge and should stay up-to-date with the latest testing methodologies. |
| 2. Number of Test Meetings | The frequency of meetings does not reflect the quality or outcomes of the sessions. What’s important is the effectiveness of communication and decision-making in these meetings. |
| 3. Number of Test Documents Written | Just because you have numerous test documents, it doesn’t mean they are helpful, comprehensive, or accurate. It’s the quality of these documents that’s more important. If a document isn’t going to add value, don’t create it. |
| 4. Number of Test Reports Generated | Generating multiple reports doesn’t ensure accuracy or comprehensiveness. It’s the insights drawn from these reports that matter. |
| 5. Number of Browsers Tested | Testing the application on numerous browsers doesn’t ensure thorough testing if these browsers are irrelevant to your user base. It’s more important to focus on the browsers most used by your audience. |
