The emergence of online labor markets makes it far easier to use individual human raters to evaluate materials for data collection and analysis in the social sciences. In this paper, we report the results of an experiment - conducted in an online labor market - that measured the effectiveness of...

This paper explores three important but interrelated issues: The power of example; the fragment as evidence; and finally, the field experience and the possibility of generalisation. These issues are central in the current discourse of social science practice in which serious doubts are being raised...