ELEVEN - Tests needed for a Recommendation

A decision maker has to recommend a treatment, knows that any outcome will be in [0; 1] but only has minimal information about the likelihood of outcomes (there is no prior). The decision maker can design a finite number of experiments in which treatments are tested. For the case of two treatments we present a rule for designing experiments and making a recommendation that attains minimax regret and can thus ensure a given maximal error with the minimal number of tests. 11 tests are needed under the so-called binomial average rule to limit the error to 5%. We also consider the setting where there is covariate information to then identify minimax regret behavior and drastically reduce the number of tests needed to attain a given maximal error as compared to the literature (over 200 to 22 given two covariates). We extend the binomial average rule to more than two treatments and use it to derive a bound on minimax regret.