Detecting treatment-subgroup interactions in clustered data with generalized linear mixed-effects model trees

Identification of subgroups of patients for which treatment A is more effective than treatment B, and vice versa, is of key importance to the development of personalized medicine. Several tree-based algorithms have been developed for the detection of such treatment-subgroup interactions. In many instances, however, datasets may have a clustered structure, where observations are clustered within, for example, research centers, studies or persons. In the current paper we propose a new algorithm, generalized linear mixed-effects model (GLMM) trees, that allows for detection of treatment-subgroup interactions, as well as estimation of cluster-specific random effects. The algorithm uses model-based recursive partitioning (MOB) to detect treatment-subgroup interactions, and a GLMM for the estimation of random-effects parameters. In a simulation study, we evaluate the performance of GLMM tree and compare it with that of MOB without random-effects estimation. GLMM tree was found to have a much lower Type I error rate than MOB trees without random effects (4% and 33%, respectively). Furthermore, in datasets with treatment-subgroup interactions, GLMM tree recovered the true treatment subgroups much more often than MOB without random effects (in 90% and 61% of the datasets, respectively). Also, GLMM tree predicted treatment outcome differences more accurately than MOB without random effects (average predictive accuracy of .94 and .88, respectively). We illustrate the application of GLMM tree on a patient-level dataset of a meta-analysis on the effects of psycho- and pharmacotherapy for depression. We conclude that GLMM tree is a promising algorithm for the detection of treatment-subgroup interactions in clustered datasets.