Feather: An Elegant Solution to Effective DNN Sparsification
Athanasios Glentis Georgoulakis (National Technical University of Athens),* George Retsinas (National Technical University of Athens), Petros Maragos (National Technical University of Athens)
The 34th British Machine Vision Conference
Abstract
Neural Network pruning is an increasingly popular way for producing compact and efficient models, suitable for resource-limited environments, while preserving high performance. While the pruning can be performed using a multi-cycle training and fine-tuning process, the recent trend is to encompass the sparsification process during the standard course of training. To this end, we introduce Feather, an efficient sparse training module utilizing the powerful Straight Through Estimator as its core, coupled with a new thresholding operator and a gradient scaling technique, enabling robust, out-of-the-box sparsification performance. Feather’s effectiveness and adaptability is demonstrated using various architectures on the CIFAR dataset, while on ImageNet it achieves state-of-the-art Top-1 validation accuracy using the ResNet-50 architecture, surpassing existing methods, including more complex and computationally heavy ones, by a considerable margin.Video
Citation
@inproceedings{Georgoulakis_2023_BMVC,
author = {Athanasios Glentis Georgoulakis and George Retsinas and Petros Maragos},
title = {Feather: An Elegant Solution to Effective DNN Sparsification},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0832.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by