This paper is about constructing confidence bands around ROC curves. Wefirst introduce to the machine learning community three band-generatingmethods from the medical field, and evaluate how well they perform. Suchconfidence bands represent the region where the “true” ROCcurve is expected to reside, with the designated confidence level. Toassess the containment of the bands we begin with a synthetic worldwhere we know the true ROC curve—specifically, where theclass-conditional model scores are normally distributed. The only methodthat attains reasonable containment out-of-the-box producesnon-parametric, “fixed-width” bands (FWBs). Next we move toa context more appropriate for machine learning evaluations: bands thatwith a certain confidence level will bound the performance of the modelon future data. We introduce a correction to account for the largeruncertainty, and the widened FWBs continue to have reasonablecontainment. Finally, we assess the bands on 10 relatively largebenchmark data sets. We conclude by recommending these FWBs, noting thatbeing non-parametric they are especially attractive for machine learningstudies, where the score distributions (1) clearly are not normal, and(2) even for the same data set vary substantially from learning methodto learning method.
