Replication and a Multi-Method Approach to Empirical Software Engineering Research

Empirical research is vital if software engineering technology is to be meaningfully evaluated --- without it only guesses can be made at the competing merits of different approaches. Unfortunately, as this thesis demonstrates, much of existing empirical work contains some kind of weakness because the approaches used to conduct the research are inadequate in many ways. To combat this problem, a more methodical approach to empirical software engineering research is required. This thesis examines replication and a multi-method approach which, it is argued, can improve the quality of results achieved by performing empirical research. External replication, where independent researchers seek to check and improve the findings of an experiment, is discussed. The results of an external replication which did not repeat the results of the original study are presented; they demonstrate the importance of replicating experiments to gain confidence in the findings. As a result, a set of general recommendations are made for researchers conducting external replications, and conducting and reporting subject-based experiments. A multi-method approach to empirical research is proposed which comprises of a series of empirical studies being evolved from an initial exploratory study through to laboratory-based studies involving internal replications. The multi-method approach allows attention to be paid to important matters and the results from each study in the series can be confirmatory. Results are presented of an application of the multi-method approach to the object-oriented paradigm. A set of general recommendations are then made for applying the multi-method approach which should enable researchers achieve quality empirical results in which they can have confidence. It is concluded that if researchers adopt a methodology for empirical software engineering research which integrates the multi-method approach with the technique of replication then, over time, more reliable and generalisable results will be realised.