Assessing Dependence: Some Experimental Results

Constructing decision- and risk-analysis probability models often requires measures of dependence among variables. Although data are sometimes available to estimate such measures, in many applications they must be obtained by means of subjective judgment by experts. We discuss two experimental studies that compare the accuracy of six different methods for assessing dependence. Our results lead to several conclusions: First, simply asking experts to report a correlation is a reasonable approach. Direct estimation is more accurate than the other methods studied, is not prone to mathematically inconsistent responses (as are some other measures), and is judged to be less difficult than alternate methods. In addition, directly assessed correlations showed less variability than the correlations derived from other assessment methods. Our results also show that experience with the variables can improve performance somewhat, as can training in a given assessment method. Finally, if a judge uses several different assessment methods, an average of the resulting estimates can also lead to better performance.