Computing the accuracy of complex non-random sampling methods: The case of the Bank of Canada's Business Outlook Survey

A number of central banks publish their own business conditions survey based on non-random sampling methods. The results of these surveys influence monetary policy decisions and thus affect expectations in financial markets. To date, however, no one has computed the statistical accuracy of these surveys because their respective non-random sampling method renders this assessment non-trivial. This paper describes a methodology for modeling complex non-random sampling behaviour, and computing relevant measures of statistical confidence, based on a given survey's historical sample selection practice. We apply this framework to the Bank of Canada's Business Outlook Survey by describing the sampling method in terms of historical practices and Bayesian probabilities. This allows us to replicate the firm selection process using Monte Carlo simulations on a comprehensive micro-dataset of Canadian firms. We find, under certain assumptions, no evidence that the Bank's firm selection process results in biased estimates and/or wider confidence intervals.