A Study on the Use of Attention for Explaining Video Summarization
Pages 41 - 49
Abstract
In this paper we present our study on the use of attention for explaining video summarization. We build on a recent work that formulates the task, called XAI-SUM, and we extend it by: a) taking into account two additional network architectures and b) introducing two novel explanation signals that relate to the entropy and diversity of attention weights. In total, we examine the effectiveness of seven types of explanation, using three state-of-the-art attention-based network architectures (CA-SUM, VASNet, SUM-GDA) and two datasets (SumMe, TVSum) for video summarization. The conducted evaluations show that the inherent attention weights are more suitable for explaining network architectures which integrate mechanisms for estimating attentive diversity (SUM-GDA) and uniqueness (CA-SUM). The explanation of simpler architectures (VASNet) can benefit from taking into account estimates about the strength of the input vectors, while another option is to consider the entropy of attention weights.
Supplementary Material
In this video, we present our study on the use of attention for explaining the output of network architectures for video summarization. We start by discussing why explainable video summarization is important, and continue by briefly describing existing approaches for explaining network architectures dealing with the analysis of video data. Following, we present the applied methodology for evaluating the use of attention as explanation for video summarization, and report on the used network architectures, explanation signals and evaluation measures. Finally, we discuss the results of the conducted quantitative and qualitative evaluations, and outline the conclusions of our study.
- Download
- 31.07 MB
References
[1]
Sathyanarayanan N. Aakur, Fillipe D. M. de Souza, and Sudeep Sarkar. 2018. An Inherently Explainable Model for Video Activity Interpretation. In The Workshops of the 32nd AAAI Conf. on Artificial Intelligence.
[2]
Evlampios Apostolidis, Eleni Adamantidou, Alexandros I. Metsai, Vasileios Mezaris, and Ioannis Patras. 2021a. Video Summarization Using Deep Neural Networks: A Survey. Proc. IEEE, Vol. 109, 11 (2021), 1838--1863. https://doi.org/10.1109/JPROC.2021.3117472
[3]
Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and Ioannis Patras. 2021b. Combining Global and Local Attention with Positional Encoding for Video Summarization. In 2021 IEEE International Symposium on Multimedia (ISM). 226--234. https://doi.org/10.1109/ISM52913.2021.00045
[4]
Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and Ioannis Patras. 2022a. Explaining video summarization based on the focus of attention. In 2022 IEEE Int. Symposium on Multimedia (ISM). 146--150. https://doi.org/10.1109/ISM55400.2022.00029
[5]
Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and Ioannis Patras. 2022b. Summarizing Videos Using Concentrated Attention and Considering the Uniqueness and Diversity of the Video Frames. In Proc. of the 2022 Int. Conf. on Multimedia Retrieval (Newark, NJ, USA) (ICMR '22). Association for Computing Machinery, New York, NY, USA, 407--415. https://doi.org/10.1145/3512527.3531404
[6]
Sarah Adel Bargal, Andrea Zunino, Donghyun Kim, Jianming Zhang, Vittorio Murino, and Stan Sclaroff. 2018. Excitation Backprop for RNNs. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).
[7]
George Chrysostomou and Nikolaos Aletras. 2021. Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification. In Proc. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int. Joint Conf. on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 477--488. https://doi.org/10.18653/v1/2021.acl-long.40
[8]
George Chrysostomou and Nikolaos Aletras. 2022. An Empirical Study on Explanations in Out-of-Domain Settings. In Proc. of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 6920--6938. https://doi.org/10.18653/v1/2022.acl-long.477
[9]
Chrysa Collyda, Konstantinos Apostolidis, Evlampios Apostolidis, Eleni Adamantidou, Alexandros I. Metsai, and Vasileios Mezaris. 2020. A Web Service for Video Summarization. In ACM Int. Conf. on Interactive Media Experiences (Cornella, Barcelona, Spain) (IMX '20). Association for Computing Machinery, New York, NY, USA, 148--153. https://doi.org/10.1145/3391614.3399391
[10]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 248--255. https://doi.org/10.1109/CVPR.2009.5206848
[11]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2--7, 2019, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171--4186. https://doi.org/10.18653/v1/n19--1423
[12]
Jiri Fajtl, Hajar Sadeghi Sokeh, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino. 2019. Summarizing Videos with Attention. In Asian Conf. on Computer Vision (ACCV) 2018 Workshops, Gustavo Carneiro and Shaodi You (Eds.). Springer International Publishing, Cham, 39--54.
[13]
Nikolaos Gkalelis, Dimitrios Daskalakis, and Vasileios Mezaris. 2022. ViGAT: Bottom-Up Event Recognition and Explanation in Video Using Factorized Graph Attention Network. IEEE Access, Vol. 10 (2022), 108797--108816. https://doi.org/10.1109/ACCESS.2022.3213652
[14]
Ioanna Gkartzonika, Nikolaos Gkalelis, and Vasileios Mezaris. 2023. Learning Visual Explanations for DCNN-Based Image Classifiers Using an Attention Mechanism. In Computer Vision -- ECCV 2022 Workshops, Leonid Karlinsky, Tomer Michaeli, and Ko Nishino (Eds.). Springer Nature Switzerland, Cham, 396--411.
[15]
Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. 2014. Creating Summaries from User Videos. In Europ. Conf. on Computer Vision (ECCV) 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 505--520. https://gyglim.github.io/me/
[16]
Yamin Han, Tao Zhuo, Peng Zhang, Wei Huang, Yufei Zha, Yanning Zhang, and Mohan Kankanhalli. 2022. One-shot Video Graph Generation for Explainable Action Reasoning. Neurocomputing, Vol. 488 (2022), 212--225. https://doi.org/10.1016/j.neucom.2022.02.069
[17]
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2021. Self-Attention Attribution: Interpreting Information Interactions Inside Transformer. Proc. of the AAAI Conf. on Artificial Intelligence, Vol. 35, 14 (May 2021), 12963--12971. https://doi.org/10.1609/aaai.v35i14.17533
[18]
Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 3543--3556. https://doi.org/10.18653/v1/N19--1357
[19]
Maurice G Kendall. 1945. The treatment of ties in ranking problems. Biometrika, Vol. 33, 3 (1945), 239--251.
[20]
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is Not Only a Weight: Analyzing Transformers with Vector Norms. In Proc. of the 2020 Conf. on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 7057--7075. https://doi.org/10.18653/v1/2020.emnlp-main.574
[21]
Stephen Kokoska and Daniel Zwillinger. 2000. CRC standard probability and statistics tables and formulae. Crc Press.
[22]
Liangzhi Li, Bowen Wang, Manisha Verma, Yuta Nakashima, Ryo Kawasaki, and Hajime Nagahara. 2021b. SCOUTER: Slot Attention-based Classifier for Explainable Image Recognition. In 2021 IEEE/CVF Int. Conf. on Computer Vision (ICCV). 1026--1035. https://doi.org/10.1109/ICCV48922.2021.00108
[23]
Ping Li, Qinghao Ye, Luming Zhang, Li Yuan, Xianghua Xu, and Ling Shao. 2021c. Exploring global diverse attention via pairwise temporal relation for video summarization. Pattern Recognition, Vol. 111 (2021), 107677. https://doi.org/10.1016/j.patcog.2020.107677
[24]
Zhenqiang Li, Weimin Wang, Zuoyue Li, Yifei Huang, and Yoichi Sato. 2021a. Towards Visually Explaining Video Understanding Networks with Perturbation. 2021 IEEE Winter Conf. on Applications of Computer Vision (WACV) (2021), 1119--1128.
[25]
Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, and Shiqi Wang. 2022. Rethinking Attention-Model Explainability through Faithfulness Violation Test. In Proc. of the 39th Int. Conf. on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 13807--13824. https://proceedings.mlr.press/v162/liu22i.html
[26]
Joonatan M"antt"ari, Sofia Broomé, John Folkesson, and Hedvig Kjellström. 2020. Interpreting Video Features: A Comparison of 3D Convolutional Networks and Convolutional LSTM Networks. In Asian Conference on Computer Vision (ACCV) 2020, Hiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, and Jianbo Shi (Eds.). Springer International Publishing, Cham, 411--426.
[27]
Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell. 2021. CLIP-It! Language-Guided Video Summarization. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 13988--14000. https://proceedings.neurips.cc/paper/2021/file/7503cfacd12053d309b6bed5c89de212-Paper.pdf
[28]
Mariano Ntrougkas, Nikolaos Gkalelis, and Vasileios Mezaris. 2022. TAME: Attention Mechanism Based Feature Fusion for Generating Explanation Maps of Convolutional Neural Networks. In 2022 IEEE Int. Symposium on Multimedia (ISM). 58--65. https://doi.org/10.1109/ISM55400.2022.00014
[29]
Konstantinos E. Papoutsakis and Antonis A. Argyros. 2019. Unsupervised and Explainable Assessment of Video Similarity. In British Machine Vision Conference. https://api.semanticscholar.org/CorpusID:199525379
[30]
Zhao Ren, Kun Qian, Fengquan Dong, Zhenyu Dai, Wolfgang Nejdl, Yoshiharu Yamamoto, and Björn W. Schuller. 2022. Deep attention-based neural networks for explainable heart sound classification. Machine Learning with Applications, Vol. 9 (2022), 100322. https://doi.org/10.1016/j.mlwa.2022.100322
[31]
Chiradeep Roy, Mahesh Shanbhag, Mahsan Nourani, Tahrima Rahman, Samia Kabir, Vibhav Gogate, Nicholas Ruozzi, and Eric D. Ragan. 2019. Explainable Activity Recognition in Videos. In ACM Intelligent User Interfaces (IUI) Workshops.
[32]
Sofia Serrano and Noah A. Smith. 2019. Is Attention Interpretable?. In Proc. of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 2931--2951. https://doi.org/10.18653/v1/P19--1282
[33]
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. 2015. TVSum: Summarizing web videos using titles. In 2015 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). 5179--5187. https://doi.org/10.1109/CVPR.2015.7299154
[34]
Alexandros Stergiou, Georgios Kapidis, Grigorios Kalliatakis, Christos Chrysoulas, Remco Veltkamp, and Ronald Poppe. 2019. Saliency Tubes: Visual Explanations for Spatio-Temporal Convolutions. In 2019 IEEE Int. Conf. on Image Processing (ICIP). 1830--1834. https://doi.org/10.1109/ICIP.2019.8803153
[35]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In 2015 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). 1--9. https://doi.org/10.1109/CVPR.2015.7298594
[36]
Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not Explanation. In Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing and the 9th Int. Joint Conf. on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 11--20. https://doi.org/10.18653/v1/D19--1002
[37]
Chongke Wu, Sicong Shao, Pratik Satam, and Salim Hariri. 2022. An explainable and efficient deep learning framework for video anomaly detection. Cluster Computing, Vol. 25, 4 (Aug. 2022), 2715--2737. https://doi.org/10.1007/s10586-021-03439--5
[38]
Hongyuan Yu, Yan Huang, Lihong Pi, Chengquan Zhang, Xuan Li, and Liang Wang. 2021. End-to-end video text detection with online tracking. Pattern Recognition, Vol. 113 (2021), 107791. https://doi.org/10.1016/j.patcog.2020.107791
[39]
Kunpeng Zhang and Li Li. 2022. Explainable multimodal trajectory prediction using attention models. Transportation Research Part C: Emerging Technologies, Vol. 143 (2022), 103829. https://doi.org/10.1016/j.trc.2022.103829
[40]
Tao Zhuo, Zhiyong Cheng, Peng Zhang, Yongkang Wong, and Mohan Kankanhalli. 2019. Explainable Video Action Reasoning via Prior Knowledge and State Transitions. In Proc. of the 27th ACM Int. Conf. on Multimedia (Nice, France) (MM '19). Association for Computing Machinery, New York, NY, USA, 521--529. https://doi.org/10.1145/3343031.3351040 io
Index Terms
- A Study on the Use of Attention for Explaining Video Summarization
Recommendations
Attention history-based attention for abstractive text summarization
SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied ComputingRecently, encoder-decoder model using attention has shown meaningful results in the abstractive summarization tasks. In the attention mechanism, the attention distribution is generated based only on the current decoder state. However, since there are ...
Hierarchical Variational Network for User-Diversified & Query-Focused Video Summarization
ICMR '19: Proceedings of the 2019 on International Conference on Multimedia RetrievalThis paper focuses on the query-focused video summarization, which is an extended task of video summarization and aims to automatically generate user-oriented summary by highlighting frames/shots relevant to the query. This task is different from ...
Effective Video Summarization by Extracting Parameter-Free Motion Attention
Video summarization remains a challenging task despite increasing research efforts. Traditional methods focus solely on long-range temporal modeling of video frames, overlooking important local motion information that cannot be captured by frame-level ...
Comments
Information & Contributors
Information
Published In
October 2023
82 pages
ISBN:9798400702778
DOI:10.1145/3607540
- General Chairs:
- Mohan S. Kankanhalli,
- Ioannis (Yiannis) Patras,
- Program Chairs:
- Jianquan Liu,
- Yongkang Wong,
- Takahiro Komamizu
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
MM '23
Sponsor:
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 106Total Downloads
- Downloads (Last 12 months)49
- Downloads (Last 6 weeks)5
Reflects downloads up to 14 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in