What Personal Information Can a Consumer Facial Image Reveal? Implications for Marketing ROI and Consumer Privacy

Massive availability of individuals' facial images through online uploads and CCTV surveillance, combined with lacking regulation, presents potential for companies to obtain richer consumer data -- at the risk of privacy violations through exposure of personal information in such images. Building on rich survey data and supermarket video and receipt information, we provide systematic evaluation of relative predictability of different types of personal information -- through deep learning on facial images. We find facial images provide versatile statistical signals on individual's behaviors, preferences, character, personality, and beliefs. A large part of such predictive power is attributable to basic demographics extracted from the face. However, image artifacts, observable facial features, and deep image features extracted by a neural net all contribute to prediction accuracy beyond demographics. Using empirical evidence, a decision theory proof, and a simulation study, we argue such incremental information can meaningfully increase return on investment (ROI) when used for targeting. The limited accuracy of predictions from facial images attenuates the privacy risks to some degree, however, the variety of signals from a facial image is surprising. Prejudice caused by human decision makers' exposure to such statistical information is a substantial concern. We hope our results and the novel data we release with this work inspire research in the emerging area of facial analysis