Information Measures for Nonparametric Kernel Estimation

This paper addresses the following question: How much information do the kernel function and the bandwidth provide for nonparametric kernel estimation? The question is addressed by showing that kernel estimation of a cumulative distribution function (CDF) is an information transmission procedure for transforming the empirical cumulative distribution function into a smooth estimate. The transmission channel is the kernel function itself, which is a conditional distribution with a data point as its location parameter and a bandwidth as its scale parameter. The output of the information transmission procedure is the kernel estimate of the CDF which is a marginal distribution constructed as the sample average of the kernel functions centered at each data point. This framework provides a lower bound for the entropy of the kernel estimate of the distribution in terms of the entropy of the kernel function and the bandwidth, an input information measure for kernel smoothing, and a measure of information for kernel estimation. A family of maximum entropy kernels that includes several well-known kernel functions is identified. These kernels are compared according to the information measures developed herein.