A Generalized Sidak-Holm Procedure and Control of Generalized Error Rates under Independence

For testing multiple null hypotheses, the classical approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of even one false rejection. In many applications, one might be willing to tolerate more than one false rejection provided the number of such cases is controlled, thereby increasing the ability of the procedure to detect false null hypotheses. This suggests replacing control of the FWER by controlling the probability of k or more false rejections, which is called the k-FWER. In Hommel and Hoffmann (1987) and Lehmann and Romano (2005a), single step and stepdown procedures are derived that control the k-FWER, without making any assumptions concerning the dependence structure of the p-values of the individual tests. However, if the p-values are mutually independent, one can improve the procedures. In fact, Sarkar (2005) provided such an improvement. However, we show other improvements are possible which appear to be generally much better, and are sometimes unimprovable. When k=1, the procedure reduces to the classical method of Sidak, and the stepdown procedure is unimprovable and strictly dominates that of Sarkar. Under a monotonicity condition, an unimprovable procedure is obtained. In the case k=2, the monotonicity condition is satisfied, and the condition can be checked numerically in general. We then develop a stepdown method that controls the false discovery proportion. Except for the case of k-FWER control with k=1, the gains are surprisingly dramatic, and theoretical and numerical evidence is given.