Correlation Based Node Partitioning to Sparse Multilayer Perceptron

A signal-dependent, correlation-based pruning algorithm is proposed to sparsify inter-layer weight matrices of a Multilayer Perceptron (MLP). The method measures correlations of node outputs for an input or hidden layer. The nodes are partitioned, accordingly. The nodes of a partition with relatively higher correlations are bundled to be the inputs of a node in the next layer. Such partitioning improves subspace representation of nodes in the network. The numerical performances for various MLP architectures and input (training) signal statistics for the two-class classification problem are presented. The results provide insights on the relationships between signal statistics, node and layer behavior, network dimension, depth, sparsity, and system performance. We show convincing evidence in the paper that the model design should track input statistics and transformations through the building blocks to sparsify the network for improved performance and computational efficiency. The proposed pruning method may also be used to design a self-reconfiguring network architecture with weight and node sparsities.