TechRxiv
tcas1-draft.pdf (456.94 kB)

A Smoothed LASSO Based DNN Sparsification Technique

Download (456.94 kB)
preprint
posted on 29.12.2020, 00:32 by Nitin Chandrachoodan, Basava Naga Girish Koneru, Vinita Vasudevan
Deep Neural Networks (DNNs) are increasingly being used in a variety of applications. However, DNNs have huge computational and memory requirements. One way to reduce these requirements is to sparsify DNNs by using smoothed LASSO (Least Absolute Shrinkage and Selection Operator) functions. In this paper, we show that for the same maximum error with respect to the LASSO function, the sparsity values obtained using various smoothed LASSO functions are similar. We also propose a layer-wise DNN pruning algorithm, where the layers are pruned based on their individual allocated accuracy loss budget determined by estimates of the reduction in number of multiply-accumulate operations (in convolutional layers) and weights (in fully connected layers). Further, the structured LASSO variants in both convolutional and fully connected layers are explored within the smoothed LASSO framework and the tradeoffs involved are discussed. The efficacy of proposed algorithm in enhancing the sparsity within the allowed degradation in DNN accuracy and results obtained on structured LASSO variants are shown on MNIST, SVHN, CIFAR-10, and Imagenette datasets.

History

Email Address of Submitting Author

nitin@ee.iitm.ac.in

ORCID of Submitting Author

0000-0002-9258-7317

Submitting Author's Institution

Indian Institute of Technology Madras

Submitting Author's Country

India

Licence

Exports

Licence

Exports