Controlled multistage selection procedures for comparison with a standard

Comparison with a standard is a general multiple comparison problem, where each system is required to be compared to a single system, referred to as a “standard”, as well as to other alternative systems. The goal is to determine the best system among a number of systems that are better than the standard, or to select the standard when it is equal to or better than the other alternatives. Kim (2005) proposed an efficient fully sequential procedure for comparison with a standard, that obtains a single observation at each stage from the surviving systems, and is one of the most efficient existing procedures. We develop two provably valid multistage selection procedures that take a number of observations from each system and update the variance estimators at each stage. We also employ appropriate control variate technique for each procedure to further improve the efficiency. Empirical results are provided to demonstrate that the proposed procedures are statistically and computationally more efficient than existing fully sequential procedures.