Slipping Anchor? Testing the Vignettes Approach to Identification and Correction of Reporting Heterogeneity

Anchoring vignettes are increasingly used to identify and correct heterogeneity in the reporting of health, work disability, life satisfaction, political efficacy, etc. with the aim of improving interpersonal comparability of subjective indicators of these constructs. The method relies on two assumptions: vignette equivalence – the vignette description is perceived by all to correspond to the same state; and, response consistency - individuals use the same response scales to rate the vignettes and their own situation. We propose tests of these assumptions. For vignette equivalence, we test a necessary condition of no systematic variation with observed characteristics in the perceived difference in states corresponding to any two vignettes. To test response consistency we rely on the assumption that objective indicators fully capture the covariation between the construct of interest and observed individual characteristics, and so offer an alternative way to identify response scales, which can then be compared with those identified from the vignettes. We also introduce a weaker test that is valid under a less stringent assumption. We apply these tests to cognitive functioning and mobility related health problems using data from the English Longitudinal Survey of Ageing. Response consistency is rejected for both health domains according to the first test, but the weaker test does not reject for cognitive functioning. The necessary condition for vignette equivalence is rejected for both health domains. These results cast some doubt on the validity of the vignettes approach, at least as applied to these health domains.