Augmentation Procedures for Control of the Generalized Family-Wise Error Rate and Tail Probabilities for the Proportion of False Positives

This article shows that any single-step or stepwise multiple testing procedure (asymptotically) controlling the family-wise error rate (FWER) can be augmented into procedures that (asymptotically) control tail probabilities for the number of false positives and the proportion of false positives among the rejected hypotheses. Specifically, given any procedure that (asymptotically) controls the FWER at level alpha, we propose simple augmentation procedures that provide (asymptotic) level-alpha control of: (i) the generalized family-wise error rate, i.e., the tail probability, gFWER(k), that the number of Type I errors exceeds a user-supplied integer k, and (ii) the tail probability, TPPFP(q), that the proportion of Type I errors among the rejected hypotheses exceeds a user-supplied value 0<q<1. Existing approaches for control of the proportion of false positives typically rely on the assumption that the test statistics are independent, while our proposed augmentation procedures control the gFWER and TPPFP for general data generating distributions, with arbitrary dependence structures among variables. Applying the augmentation methods to step-down multiple testing procedures that control the FWER asymptotically exactly at level alpha (van der Laan et al., 2004), yields procedures that also provide exact asymptotic control of the gFWER and TPPFP at level alpha. The adjusted p-values for the gFWER and TPPFP-controlling augmentation procedures are shown to be simple functions of the adjusted p-values for the original FWER-controlling procedure. Finally, two simple conservative procedures are proposed for controlling the false discovery rate.