Pls - a Special Solution of Generalized Delta Rule in Multilayer Linear Neural Networks

It is well known that multilayer neural networks with generalized delta rule (GDR) can map any arbitrary functions. The relationship between statistical linear regression and network learning has been attributed to the principal components which can be extracted from the input patterns along. Partial least squares (PLS) regression, another popular multivariate analysis method, has the advantage of resultant spectral vectors that are directly correlated to the constituents of interest rather than the principal components. Even though, this algorithm has been applied as a architecture of three-layer neural network, it has never identified as a special solution of GDR learning in linear multilayer neural networks. The mapping of PLS regression to a general three-layer architecture of artificial neural networks allows the implementation of PLS algorithm in the parallel weight matrices. Due to the fact that both PLS and GDR use the input-output pair (cross-correlation) for error correction, it is thus interesting to find out the relationship between these two methods, especially for the contents of convergent weight matrices. With orthogonal input-output pair and pure linear activation function, we have proved and demonstrated that PLS network can decompose and identify the optimal solution as in the convergent state of GDR. It can thus provide a statistical way to interpret the extracted features in GDR method. It can also help to determine the optimal numbers of hidden node, which is still empirically decided in BP