Optimizing Distance-Based Methods for Big Data Analysis

Distance-based methods for measuring spatial concentration such as the Duranton-Overman index undergo an increasing popularity in the spatial econometrics community. However, a limiting factor for their usage is their computational complexity since both their memory requirements and running-time are in O(n2). In this paper, we present an algorithm with constant memory requirements and an improved running time, enabling the Duranton-Overman index and related distance-based methods to run big data analysis. Furthermore, we discuss the index by Scholl and Brenner (2012) whose mathematical concept allows an even faster computation for large datasets than the improved algorithm does.