Discovering Interesting Patterns for Investment Decision Making with Glower C - a Genetic Learner Overlaid with Entropy Reduction

Prediction in financial domains is notoriously difficult for a number of reasons. First, theories tend to beweak or non-existent, which makes problem formulation open-ended by forcing us to consider a largenumber of independent variables and thereby increasing the dimensionality of the search space. Second, theweak relationships among variables tend to be nonlinear, and may hold only in limited areas of the searchspace. Third, in financial practice, where analysts conduct extensive manual analysis of historically wellperforming indicators, a key is to find the hidden interactions among variables that perform well incombination. Unfortunately, these are exactly the patterns that the greedy search biases incorporated bymany standard rule algorithms will miss. In this paper, we describe and evaluate several variations of a newgenetic learning algorithm (GLOWER) on a variety of data sets. The design of GLOWER has been motivatedby financial prediction problems, but incorporates successful ideas from tree induction and rule learning.We examine the performance of several GLOWER variants on two UCI data sets as well as on a standardfinancial prediction problem (Samp;P500 stock returns), using the results to identify and use one of the bettervariants for further comparisons. We introduce a new (to KDD) financial prediction problem (predictingpositive and negative earnings surprises), and experiment withGLOWER, contrasting it with tree- and rule-inductionapproaches. Our results are encouraging, showing that GLOWER has the ability to uncovereffective patterns for difficult problems that have weak structure and significant nonlinearities