Simulating Statistical Power Curves with the Bootstrap and Robust Estimation

Power and effect size analysis are important methods in the psychological sciences. It is well known that classical statistical tests are not robust with respect to power and type II error. However, relatively little attention has been paid in the psychological literature to the effect that non-normality and outliers have on the power of a given statistical test (Wilcox, 1998). Robust measures of location exist that provide much more powerful tests of statistical hypotheses, but their usefulness in power estimation for sample size selection, with real data, is largely unknown. Furthermore, practical approaches to power planning (Cohen, 1988) usually focus on normal theory settings and in general do not make available nonparametric approaches to power and effect size estimation. Beran (1986) proved that it is possible to nonparametrically estimate power for a given statistical test using bootstrap methods (Efron, 1993). However, this method is not widely known or utilized in data analysis settings. This research study examined the practical importance of combining robust measures of location with nonparametric power analysis. Simulation and analysis of real world data sets are used. The present study found that: 1) bootstrap confidence intervals using Mestimators gave shorter confidence intervals than the normal theory counterpart whenever the data had heavy tailed distributions; 2) bootstrap empirical power is higher for Mestimators than the normal theory counterpart when the data had heavy tailed distributions; 3) the smoothed bootstrap controls type I error rate (less than 6%) under the null hypothesis for small sample sizes; and 4) Robust effect sizes can be used in conjuction with Cohen's (1988) power tables to get more realistic sample sizes given that the data distribution has heavy tails.