Why is Evaluating Usability of Privacy Designs So Hard? Lessons Learned from a User Study of PRISM

Privacy is a thorny issue that affects all information systems designedfor interpersonal interaction and awareness. Theoretical insightsregarding privacy and user experience in a variety of systemshave produced numerous design principles and guidelines forbuilding systems sensitive to privacy issues. In order to truly improvesupport for privacy management, the usability of systems thatimplement these principles is critical. Yet, usability evaluation ofprivacy designs is a relatively unexplored area. In this paper, wedescribe our experience of conducting a longitudinal user studyto evaluate the effectiveness of the privacy management enhancementsoffered by PRISM, a plugin for Instant Messaging (IM). Althoughthe study was unable to achieve its intended objective, thelessons learned highlight the difficulties faced by evaluators of privacymanagement mechanisms. We hope that our experience willbenefit future evaluations of privacy management mechanisms, andwill initiate discussions on overcoming some of these challenges.