Predicting Compliance : Leveraging Chat Data for Supervised Classification in Experimental Research

Many experimental research designs offer participants the opportunity to chat with each other. While experimental research has traditionally treated text as process data, this paper proposes a novel approach to interpret and use chat data in a structured supervised classification task. Specifically, we provide a framework for supervised classification where chat text serves as input and a related numeric variable serves as the target to be predicted. We show that the combination of chat data and decision data is an ideal precondition for supervised classification, but that experimental data present unique challenges, such as being small or heavily skewed. To overcome these challenges, we propose a methodological framework that systematically investigates various classification setups to maximize predictive performance in small datasets from experimental behavioral research. We demonstrate the framework's utility in predicting honest decisions in a tax evasion game. To ensure the wider accessibility and usability of our proposed framework, we have created an open-access GitHub repository with clear instructions on how to use the codebase and available data. Additionally, the repository includes slides with theoretical background and quizzes, making it a valuable educational resource for researchers and students alike. Our results show that our best model performs better than random chance, despite the challenges presented by the small, skewed dataset. We also demonstrate that dictionary-based approaches can be used to gain insight into the data even in the absence of raw text. Overall, this paper provides a valuable toolbox for the research community to build upon and encourages further exploration of chat data in supervised classification tasks in experimental research