FLRKD: Relational Knowledge Distillation Based on Channel-wise Feature Quality Assessment
Zeyu An (University of Electronic Science and Technology of China),* Changjian Deng (University of Electronic Science and Technology of China), Wanli Dang (University of Electronic Science and Technology of China;The Second Research Institute of the Civil Aviation Administration of China), Zhicheng Dong (Tibet university), 谦 罗 (中国民用航空总局第二研究所), Jian Cheng (University of Electronic Science and Technology of China)
The 34th British Machine Vision Conference
Abstract
With the increasing computational power of computing devices, the pre-training of large deep-learning models has become prevalent. However, deploying such models on edge devices with limited memory and computing power remains a significant challenge. To address this issue, this study proposes a novel knowledge distillation approach called Feature-level Relationship-based Knowledge Distillation (FLRKD). The proposed approach employs image quality similarity assessment to distill knowledge from a pre-trained model into smaller models that are suitable for deployment on edge devices. FLRKD utilizes peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) between feature maps of different hidden layers as relational knowledge to enhance the classification accuracy of student models. Moreover, the proposed approach includes an effective loss function that accelerates the convergence of the knowledge distillation algorithm. Additionally, a regressor is introduced to address the issue of inconsistent feature map spatial size between teacher and student models in heterogeneous scenarios. Comparative and ablation experiments demonstrate the superiority of FLRKD over mainstream knowledge distillation methods in terms of higher classification accuracy (up to 4%) and faster convergence rates. Notably, the proposed approach achieves significant improvement in classification accuracy (up to 3%) even in heterogeneous scenarios compared to existing state-of-the-art methods.Video
Citation
@inproceedings{An_2023_BMVC,
author = {Zeyu An and Changjian Deng and Wanli Dang andThe Second Research Institute of the Civil Aviation Administration of Chin and Zhicheng Dong and 谦 罗 and Jian Cheng},
title = {FLRKD: Relational Knowledge Distillation Based on Channel-wise Feature Quality Assessment},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0360.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by