Machine Learning and Big Data to Identify Market Expectations and Surprises : Evidence from Scheduled USDA Reports

Traditionally, a predefined surprise proxy (such as the consensus errors of analyst forecasts) is used to estimate the market impact of public announcements. We instead use the post-event price movements to tease out what the market consensus must have been, and to estimate the event-day surprises. Our empirical analysis focuses on the USDA’s weekly Crop Progress and Condition reports (CPCRs), which we show can be forecasted using weather “big” data. Departing from conventional machine learning (ML) approaches, we create a new ML routine to incorporate important features of a market expectation model under the Efficient Market Hypothesis’ semi-strong form. We find that the market often overestimates the condition of both crops by about 5-6%, with occasional spikes up to 22%. Moreover, these surprise estimates suggest that the reports still cause statistically significant post-release market reactions, though of small magnitudes