Accounting for Defect Characteristics in Empirical Studies of Software Testing

Software testing is an indispensable activity in quality assurance andan enduring topic of research. For decades, researchers have beeninventing new and better techniques to test software. However, notesting technique will ever be a panacea for all softwaredefects. Therefore, while researchers should continue to develop newtesting techniques, they also need to deeply understand the abilitiesand limitations of existing techniques, the ways they complement eachother, and the trade-offs involved in using different techniques. Thiswork contends that researchers cannot sufficiently understand softwaretesting without also understanding software defects.This work is the first to show that simple, automatically-measurablecharacteristics of defects affect their susceptibility to detection bysoftware testing. Unlike previous attempts to characterize defects,this work offers a characterization that is objective, practical, andproven to help explain why some defects and not others are detected bytesting.More importantly, this work shows that researchers can and shouldaccount for defect characteristics when they study the effectivenessof software-testing techniques. An experiment methodology is presentedthat enables experimenters to compare the effectiveness of differenttechniques and, at the same time, to measure the influence of defectcharacteristics and other factors on the results. The methodology isdemonstrated in a large experiment in the domain ofgraphical-user-interface testing. As the experiment shows, researchers who use the methodology willunderstand what kinds of defects tend to be detected by testing andwhat testing techniques are better at detecting certain kinds ofdefects. This information can help researchers develop more effectivetesting techniques, and it can help software testers make betterchoices about the testing techniques to use on their projects. As thiswork explains, it also has the potential to help testers detect moredefects, and more important defects, during regression testing.