A True Expert Knows which Question Should be Asked.

We suggest a test for discovering whether a potential expert is informed of the distribution of a stochastic process. In a non-Bayesian non-parametric setting, the expert is asked to make a prediction which is tested against a single realization of the stochastic process. It is shown that by asking the expert to predict a “small” set of sequences, the test will assure that any informed expert can pass the test with probability one with respect to the actual distribution. Moreover, for the uninformed non-expert it is impossible to pass this test, in the sense that for any choice of a “small” set of sequences, only a “small” set of measures will assign a positive probability to the given set. Hence for “most” measures, the non-expert will surely fail the test. We define small as category 1 sets, described in more detail in the paper.