Testing the Effectiveness of Various Commonly Used Fit Indices for Detecting Misspecifications in Multilevel Structure Equation Models

Two Monte Carlo studies were conducted to investigate the sensitivity of fit indices in detecting model misspecification in multilevel structural equation models (MSEM) with normally distributed or dichotomous outcome variables separately under various conditions. Simulation results showed that RMSEA and CFI only reflected within-model fit. In addition, SRMR for within-model (SRMR-W) was more sensitive to within-model misspecifications in factor covariances than pattern coefficients regardless of the impact of other design factors. Researchers should use SRMR-W in combination with RMSEA and CFI to evaluate the within-mode. On the other hand, SRMR for between-model (SRMR-B) was less likely to detect between-model misspecifications when ICC decreased. Lastly, the performance of WRMR was dominated by the misfit of within-model. In addition, WRMR was less likely to detect the misspecified between-models when ICC was relative low. Therefore, WRMR can be used to evaluate the between-model fit when the within-models were correctly specified and the ICC was not too small.