Testing the difference between two Kolmogorov--Smirnov values in the context of receiver operating characteristic curves

The maximum vertical distance between a receiver operating characteristic (ROC) curve and its chance diagonal is a common measure of effectiveness of the classifier that gives rise to this curve. This measure is known to be equivalent to a two-sample Kolmogorov--Smirnov statistic; so the absolute difference <italic>D</italic> between two such statistics is often used informally as a measure of difference between the corresponding classifiers. A significance test of <italic>D</italic> is of great practical interest, but the available Kolmogorov--Smirnov distribution theory precludes easy analytical construction of such a significance test. We, therefore, propose a Monte Carlo procedure for conducting the test, using the binormal model for the underlying ROC curves. We provide Splus/R routines for the computation, tabulate the results for a number of illustrative cases, apply the methods to some practical examples and discuss some implications.