Conjugate Priors Represent Strong Pre-Experimental Assumptions

It is well known that Jeffreys' prior is asymptotically least favorable under the entropy risk, i.e. it asymptotically maximizes the mutual information between the sample and the parameter. However, in this paper we show that the prior that minimizes (subject to certain constraints) the mutual information between the sample and the parameter is natural conjugate when the model belongs to a natural exponential family. A conjugate prior can thus be regarded as maximally informative in the sense that it minimizes the weight of the observations on inferences about the parameter; in other words, the expected relative entropy between prior and posterior is minimized when a conjugate prior is used. Copyright 2004 Board of the Foundation of the Scandinavian Journal of Statistics..