Distillation for High-Quality Knowledge Extraction via Explainable Oracle Approach

MyungHak Lee (Kookmin University),* Wooseong Syz Cho (Kookmin University), Sungsik Kim (Kookmin University), Jinkyu Kim (Korea University), Jaekoo Lee (Kookmin University)
The 34th British Machine Vision Conference


Recent successes suggest that knowledge distillation techniques can usefully trans- fer knowledge between deep neural networks as compression and acceleration tech- niques, e.g., effectively and reliably compress a large teacher model into a smaller stu- dent model with limited resources. However, knowledge distillation performance is de- graded when the model compression rate becomes excessively high due to the size of the teacher model. To address this, we advocate for improving the teacher-to-student knowledge transfer by identifying and reinforcing input-level signals of substantial con- tributions for a final verdict, e.g., signals for a long trunk of elephants are strengthened and transferred to the student model. To this end, we adopt gradient-based explainable AI techniques for extracting output-relevant input-level features. Then, we strengthen and transfer these signals to improve the knowledge distillation performance. Our ex- periments on public datasets (i.e., CIFAR-10, CIFAR-100, Tiny-ImageNet, and Ima- geNet) show that our method clearly outperforms existing knowledge distillation ap- proaches, especially in the case of using a small teacher model. Our code is available at https://github.com/myunghakLee/Distillation-for-High-Quality-Knowledge-Extraction



author    = {MyungHak Lee and Wooseong Syz Cho and Sungsik  Kim and Jinkyu Kim and Jaekoo Lee},
title     = {Distillation for High-Quality Knowledge Extraction via Explainable Oracle Approach},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year      = {2023},
url       = {https://papers.bmvc2023.org/0665.pdf}

Copyright © 2023 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection