Know Your System! – Turning Data Mining from Bias to Benefit Through System Parameter Permutation

The goal of this paper is to assist the trader in answering two questions: 1) "What is a reasonable performance estimate of the long-run edge of the trading system?" and, 2) "What worst-case contingencies must be tolerated in short-run performance in order to achieve the long-run expectation?" With this information, the trader can make probabilistic, data-driven decisions on whether to allocate capital to the system and once actively trading, whether the system is "broken" and should cease trading.Traditional trading system development leads not only to positively biased performance estimates due to the data mining bias, but much valuable information is lost in the process. The result for many traders is frustration due to poor realized trading system performance that does not live up to expectations. This paper explores how to leverage the optimization inherent in typical system development via a method named System Parameter Permutation (SPP) and to extract information that enables realistic contingency planning based on probabilities. Many traders and system developers go to great lengths to avoid the effects of randomness in trading results, knowing the large impact it may cause. In contrast, SPP embraces randomness as a tool to help uncover what may probabilistically be expected from a trading system in the future. The method is simple to apply yet very effective. The method is applied to an example rotational trading system based on relative momentum and the results are compared to traditional out-of-sample testing. The example shows how SPP fully leverages available historical data to enable deep understanding of potential risks and rewards prior to allocating capital to a trading system