Content analysis with language models: Exploring a zero-shot learning approach

In this paper, we discuss whether and how Natural Language Processing (NLP) can be integrated into the workflows of conventional qualitative research to support the researcher. We focus on textual data which the researcher wants to analyse using predefined categories. We explore the possibility of car- rying out the coding step using NLP instead of assigning codes manually. Integrating such models into the qualitative research process makes the research more accessible and comprehensible for third parties and can contribute to moving research more into the direction of open science. Our study in- dicates that the procedure is potentially able to identify the core results. However, the findings also indicate that there are weaknesses which strongly depend on the specific text and the research ques- tion under investigation. We find that off-the-shelf language models are not able to distinguish be- tween related topics as clearly as humans can. Moreover, humans are able to understand and classify passages which only implicitly refer to a topic. Findings suggest that off-the-shelf language models frequently fail to identify such passages. We conclude that an importantstep to reliably apply language models in qualitative research consists of improving the models. Nevertheless, in their current state, off-the shelf language models can be used to validate the results obtained by manual coding. This could make qualitative research more traceable and fully reproducible.