Multiple Testing. Part III. Procedures for Control of the Generalized Family-Wise Error Rate and Proportion of False Positives

The accompanying articles by Dudoit et al. (2003b) and van der Laan et al. (2003) provide single-step and step-down resampling-based multiple testing procedures that asymptotically control the family-wise error rate (FWER) for general null hypotheses and test statistics. The proposed procedures fundamentally differ from existing approaches in the choice of null distribution for deriving cut-offs for the test statistics and are shown to provide asymptotic control of the FWER under general data generating distributions, without the need for conditions such as subset pivotality. In this article, we show that any multiple testing procedure (asymptotically) controlling the FWER at level alpha can be augmented into: (i) a multiple testing procedure (asymptotically) controlling the generalized family-wise error rate (i.e., the probability, gFWER(k), of having more than k false positives) at level alpha and (ii) a multiple testing procedure (asymptotically) controlling the probability, PFP(q), that the proportion of false positives among the rejected hypotheses exceeds a user-supplied value q in (0,1) at level alpha. Existing procedures for control of the proportion of false positives typically rely on the assumption that the test statistics are independent, while our proposed augmentation procedures control the PFP and gFWER for general data generating distributions, with arbitrary dependence structures among variables. Applying our augmentation methods to step-down multiple testing procedures that asymptotically control the FWER at exact level alpha (van der Laan et al., 2003), yields multiple testing procedures that also asymptotically control the gFWER and PFP at exact level alpha. Finally, the adjusted p-values for the gFWER and PFP-controlling augmentation procedures are shown to be simple functions of the adjusted p-values for the original FWER-controlling procedure.