Large-scale demand estimation with search data

Traditional methods for estimating demand are not always well-suited to online markets, where individual products are sold infrequently, unobserved factors such as webpage layout drive substitution, and often only a limited set of product characteristics is observed. We propose a demand model where browsing data—which is abundant in many online settings— is used to infer individual consumers' consideration sets. In our model, the underlying variables which drive consideration can be correlated arbitrarily across products. We estimate the model through a constraint maximization approach, based on the insight that these correlations should rationalize the product-pair co-search frequencies that are observed in the data. In turn, these correlations make it possible to estimate more flexible substitution patterns. We apply the model to data from an online retailer, recover the elasticity matrix, and solve for optimal prices