Distillation for High-Quality Knowledge Extraction via Explainable Oracle Approach
MyungHak Lee (Kookmin University),* Wooseong Syz Cho (Kookmin University), Sungsik Kim (Kookmin University), Jinkyu Kim (Korea University), Jaekoo Lee (Kookmin University)
The 34th British Machine Vision Conference
Abstract
Recent successes suggest that knowledge distillation techniques can usefully trans- fer knowledge between deep neural networks as compression and acceleration tech- niques, e.g., effectively and reliably compress a large teacher model into a smaller stu- dent model with limited resources. However, knowledge distillation performance is de- graded when the model compression rate becomes excessively high due to the size of the teacher model. To address this, we advocate for improving the teacher-to-student knowledge transfer by identifying and reinforcing input-level signals of substantial con- tributions for a final verdict, e.g., signals for a long trunk of elephants are strengthened and transferred to the student model. To this end, we adopt gradient-based explainable AI techniques for extracting output-relevant input-level features. Then, we strengthen and transfer these signals to improve the knowledge distillation performance. Our ex- periments on public datasets (i.e., CIFAR-10, CIFAR-100, Tiny-ImageNet, and Ima- geNet) show that our method clearly outperforms existing knowledge distillation ap- proaches, especially in the case of using a small teacher model. Our code is available at https://github.com/myunghakLee/Distillation-for-High-Quality-Knowledge-ExtractionVideo
Citation
@inproceedings{Lee_2023_BMVC,
author = {MyungHak Lee and Wooseong Syz Cho and Sungsik Kim and Jinkyu Kim and Jaekoo Lee},
title = {Distillation for High-Quality Knowledge Extraction via Explainable Oracle Approach},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0665.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by