Large Scale, Complex Processing of Health Data with MapReduce

The article describes a solution to process large volumes of unstructured health social media data in a scalable fashion using the MapReduce framework. Our work is in the context of health informatics applications involving complex text and language processing as well as large resources such as ontologies, due to which the text processing of a single unit of text takes time. Even with a throughput of an order processing time of one second per unit, it takes over a week to process a million units, which is unacceptable. We present a solution where we take the processing to a MapReduce framework and achieve significant improvement in processing performance by dividing the processing across a cluster of processors. This paper describes the technical details of our work in terms of the design, modeling, and implementation of such an approach. We also present experimental results demonstrating the effectiveness of our approach.