No more cost in translation: Validating open-source machine translation for quantitative text analysis

As more and more scholars apply computational text analysis methods to multilingual corpora, machine translation has become an indispensable tool. However, relying on commercial services for machine translation, such as Google Translate or DeepL, limits reproducibility and can be expensive. This paper assesses the viability of a reproducible and affordable alternative: free and open-source machine translation models. We ask whether researchers who use an open-source model instead of a commercial service for machine translation would obtain substantially different measurements from their multilingual corpora. We address this question by replicating and extending an influential study by de Vries et al. (2018) on the use of machine translation in cross-lingual topic modeling, and an original study of its use in supervised text classification with Transformer-based classifiers. We find only minor differences between the measurements generated by these methods when applied to corpora translated with open-source models and commercial services, respectively. We conclude that "free" machine translation is a very valuable addition to researchers' multilingual text analysis toolkit. Our study adds to a growing body of work on multilingual text analysis methods and has direct practical implications for applied researchers.