A Variant Q-Sorting Methodology for Building Diagnostic Trees

Diagnostic theories are fundamental to information system (IS) practice and are represented as trees. While there are approaches for validating diagnostic trees, these validate the overall performance of the tree rather than identifying ways incorrect diagnoses can occur. It is important to fully validate diagnostic trees because even if the tree gives the correct decision “most of the time,” it is possible for incorrect decisions traveling down little-used branches of the tree to result in catastrophic decisions. In this article, we describe the process of using a variant of q-sorting to validate diagnostic trees. In this methodology, diagnostic trees that independent experts develop are transformed into a quantitative form, and that quantitative form is tested to determine the inter-rater reliability of the individual branches in the tree. The trees are then successively transformed to incrementally test if they branch in the same way. The results help researchers not only identify quality items for use in a diagnostic tree but also facilitate diagnoses of problems with those items and facilitate the reconciliation of discrepant trees by experts. The methodology validates not only the whole tree but also its subparts. Full paper available at 10.1109/TEM.2021.3078582