Bayes Estimate and Inference for Entropy and Information Index of Fit

This article defines a quantized entropy and develops Bayes estimates and inference for the entropy and a Kullback-Leibler information index of the model fit. We use a Dirichlet process prior for the unknown data-generating distribution with a maximum entropy candidate model as the expected distribution. This formulation produces prior and posterior distributions for the quantized entropy, the information index of fit, the moments, and the model parameters. The posterior mean of the quantized entropy provides a Bayes estimate of entropy under quadratic loss. The consistency of the Bayes estimates and the information index are shown. The implementation and the performances of the Bayes estimates are illustrated using data simulated from exponential, gamma, and lognormal distributions.