The design of replicated difference tests

We show that adding replications in replicated difference test results in larger power and smaller variance when the number of assessors is fixed. On the other hand, when the number of total assessments is fixed, the power usually decreases and the variability increases whenever replications are considered instead of different assessors. The appropriate numbers of assessor needed to gain the same power respectively variability when replications are used will be given. It is shown that the number of assessors might indeed be reduced, but this has to be paid for by an increasing total number of assessments. We show that two key models, namely the mixture binomial and a corrected version of the Beta-binomial model, are quite similar with respect to the properties of interest. We provide tables from which, according to her/his requirements, the investigator might find an appropriate setting with respect to the number of assessors and replications.