Embedding Human Knowledge into Spatio-Temproal Attention Branch Network in Video Recognition via Temporal attention
Saki Noguchi (Chubu University),* Yuzhi Shi ( Chubu University), Tsubasa Hirakawa (Chubu University), Takayoshi Yamashita (Chubu University), Hironobu Fujiyoshi (Chubu University)
The 34th British Machine Vision Conference
Abstract
When recognizing object or motion, humans can accurately judge the necessary areas for recognition. In contrast, recognition using a deep learning model is based on its training data and may fail to focus on the correct regions. In image recognition, it has been shown that visualizing the basis of decisions and embedding human knowledge into deep neural networks are effective in addressing this issue. However, in video recognition, there is no visualization method enabling us to embed human knowledge. We propose spatio-temporal attention branch network (ST-ABN) for video recognition, which provides visual explanations for both spatial and temporal attentions. One of the features of the ST-ABN is that its attention output can be modified on the basis of human knowledge and used for recognition. However, since a video consists of a large number of frame images, modifying spatial attentions similar to image recognition is costly. Therefore, we manually modify temporal attentions to embed human knowledge into the ST-ABN. Experimental results with Something-Something v.2 indicate that the ST-ABN provides visual explanation for both spatial and temporal information and improves recognition performance. The results also indicate the effectiveness of embedding human knowledge into the ST-ABN and the positive changes in spatial attentions by modifying temporal attentions.Video
Citation
@inproceedings{Noguchi_2023_BMVC,
author = {Saki Noguchi and Yuzhi Shi and Tsubasa Hirakawa and Takayoshi Yamashita and Hironobu Fujiyoshi},
title = {Embedding Human Knowledge into Spatio-Temproal Attention Branch Network in Video Recognition via Temporal attention},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://papers.bmvc2023.org/0207.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by