Big Data Analysis, from Cloud to Crowd

Big Data is characterized by its large volumes, the velocity at which it is created and its variety. These characteristics imply that the technology to analyze it must be appropriately equipped in order to process it effectively. The technologies connote a considerable investment in hardware and software for companies and contain the risk of investing in the wrong technology. Also data analysis is often only performed occasionally, for instance quarterly or at the end of the year and the procurement of data analysis equipment is not practicable for many organizations. Cloud based data analysis solutions have proven to be more economic in many situations, mitigating the risk of malinvestment, lowering the maintenance costs and, with its on-demand model, offer a more cost transparent software delivery model (Software as a Service) than an on premises solution.Recently there has been a new development towards a crowd-based data analysis paradigm. Crowd-based approaches can overcome some of the problems inherent in data mining by harnessing the wisdom of the crowd. Some data analysis tasks are still handled poorly by automated, computer-based solutions and human judgement often out-performs machine-based methods. A hybrid approach can mitigate some of the shortcomings of an approach using just a cloud- or a crowd-based solution.This paper describes the Big Data analysis paradigm shift from cloud-based to crowd-based solutions and explores the advantages of such an approach. The paper explores a hybrid, machine- and human-based approach using Entity Resolution as a sample Big Data analysis task