Characterizing disagreement and inconsistency in experts' judgments in the analytic hierarchy process

Purpose – Methods and techniques of aggregating preferences or priorities in the analytic hierarchy process (AHP) usually ignore variation or dispersion among experts and are vulnerable to extreme values (generated by particular viewpoints or experts trying to distort the final ranking). The purpose of this paper is to propose a modelling approach and a graphical representation to characterize inconsistency and disagreement in the group decision making in the AHP. Design/methodology/approach – The authors apply a regression approach for estimating the decision weights of the AHP using linear mixed models (LMM). They also test the linear mixed model and the multi‐dimensional scaling graphical display using a case of strategic performance management in education. Findings – In addition to determining the weight vectors, this model also allows the authors to decompose the variation or uncertainty in experts' judgment. Well‐known statistical theories can estimate and rigorously test disagreement among experts, the residual uncertainty due to rounding errors in AHP scale, and the inconsistency within individual experts' judgments. Other than characterizing different sources of uncertainty, this model allows the authors to rigorously test other factors that might significantly affect weight assessments. Originality/value – This study provides a model to better characterize different sources of uncertainty. This approach can improve decision quality by allowing analysts to view the aggregated judgments in a proper context and pinpoint the uncertain component that significantly affects decisions.