Deep Learning in the Service of Customer Service : The Value of AI-Enabled Timely Performance Feedback

Customer service agents serve a critical role towards post-purchase customer satisfaction, retention, and loyalty. However, properly evaluating their performance involves several challenges: commonly used customer survey measures suffer from scarce, biased, and delayed responses. This study addresses these challenges by presenting a novel approach, which we term Conversational Hierarchical Attention Network (C-HAN), a deep learning algorithm that predicts agents’ performance for each customer service transaction in real-time. The algorithm leverages the hierarchical structure of document classification and incorporates various conversational features present in a customer-agent interaction—e.g., who the speaker is; when and how long each person spoke; and the total duration of the conversation—to better predict transaction quality. To examine the proposed algorithm’s effectiveness and understand the implications of providing timely feedback in practice, we conduct a controlled field experiment that presents algorithm-derived performance metrics to customer service agents. The results demonstrate how the ability to provide timely feedback improves overall agent performance. Yet, when relative feedback, i.e., the average team measures, is additionally presented, the performance gain diminishes. Furthermore, the effects of AI-derived performance feedback on agents of different initial abilities, call-related behavior, as well as the short- and long-term implications are also studied