research-article

Semi-supervised Semantic Segmentation with Mutual Knowledge Distillation

Authors:

Yifan liuAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 5436 - 5444

https://doi.org/10.1145/3581783.3611906

Published: 27 October 2023 Publication History

Abstract

Consistency regularization has been widely studied in recent semi- supervised semantic segmentation methods, and promising per- formance has been achieved. In this work, we propose a new con- sistency regularization framework, termed mutual knowledge dis- tillation (MKD), combined with data and feature augmentation. We introduce two auxiliary mean-teacher models based on consis- tency regularization. More specifically, we use the pseudo-labels generated by a mean teacher to supervise the student network to achieve a mutual knowledge distillation between the two branches. In addition to using image-level strong and weak augmentation, we also discuss feature augmentation. This involves considering various sources of knowledge to distill the student network. Thus, we can significantly increase the diversity of the training samples. Experiments on public benchmarks show that our framework out- performs previous state-of-the-art (SOTA) methods under various semi-supervised settings. Code is available at https://github.com/jianlong-yuan/semi-mmseg.

Supplemental Material

MP4 File

Presentation video

Download
21.47 MB

References

[1]

David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. 2019a. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. Proc. IEEE Conf. Comp. Vis. Patt. Recogn. (2019).

[2]

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. 2019b. Mixmatch: A holistic approach to semi-supervised learning. Proc. Advances in Neural Inf. Process. Syst., Vol. 32 (2019).

[3]

Liang-Chieh Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell D Collins, Ekin D Cubuk, Barret Zoph, Hartwig Adam, and Jonathon Shlens. 2020. Naive-student: Leveraging semi-supervised learning in video sequences for urban scene segmentation. In Proc. Eur. Conf. Comp. Vis. Springer, 695--714.

Digital Library

[4]

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proc. Eur. Conf. Comp. Vis. 801--818.

Digital Library

[5]

Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. 2021. Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 2613--2622.

[6]

François Chollet. 2017. Xception: Deep learning with depthwise separable convolutions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 1251--1258.

[7]

MMSegmentation Contributors. 2020. MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark. https://github.com/open-mmlab/mmsegmentation.

[8]

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016. The cityscapes dataset for semantic urban scene understanding. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 3213--3223.

[9]

Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2015. The pascal visual object classes challenge: A retrospective. Int. J. Comput. Vision, Vol. 111, 1 (2015), 98--136.

Digital Library

[10]

Zhengyang Feng, Qianyu Zhou, Qiqi Gu, Xin Tan, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. 2022. Dmt: Dynamic mutual training for semi-supervised learning. Pattern Recognition (2022), 108777.

[11]

Geoff French, Timo Aila, Samuli Laine, Michal Mackiewicz, and Graham Finlayson. 2019. Semi-supervised semantic segmentation needs strong, high-dimensional perturbations. Proc. British Machine Vis. Conf. (2019).

[12]

Yixiao Ge, Dapeng Chen, and Hongsheng Li. 2020. Mutual mean-teaching: Pseudo label refinery for unsupervised domain adaptation on person re-identification. Proc. Int. Conf. Learn. Representations (2020).

[13]

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020. Bootstrap your own latent-a new approach to self-supervised learning. Proc. Advances in Neural Inf. Process. Syst., Vol. 33 (2020), 21271--21284.

[14]

Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. 2011. Semantic contours from inverse detectors. In Proc. IEEE Int. Conf. Comp. Vis. IEEE, 991--998.

Digital Library

[15]

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 9729--9738.

[16]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 770--778.

[17]

Ruifei He, Jihan Yang, and Xiaojuan Qi. 2021. Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation. In Proc. IEEE Int. Conf. Comp. Vis. 6930--6940.

[18]

Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. 2021. Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning. Proc. Advances in Neural Inf. Process. Syst., Vol. 34 (2021).

[19]

Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Lin, and Ming-Hsuan Yang. 2018. Adversarial learning for semi-supervised semantic segmentation. In Proc. British Machine Vis. Conf.

[20]

Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau. 2020. Guided collaborative training for pixel-wise semi-supervised learning. In Proc. Eur. Conf. Comp. Vis. Springer, 429--445.

Digital Library

[21]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Proc. Advances in Neural Inf. Process. Syst., Vol. 25 (2012).

[22]

Samuli Laine and Timo Aila. 2016. Temporal ensembling for semi-supervised learning. Proc. Int. Conf. Learn. Representations (2016).

[23]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Proc. Eur. Conf. Comp. Vis. Springer, 740--755.

[24]

Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Carneiro. 2021. Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation. arXiv preprint arXiv:2111.12903 (2021).

[25]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 3431--3440.

[26]

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 41, 8 (2018), 1979--1993.

[27]

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. 2014. The Role of Context for Object Detection and Semantic Segmentation in the Wild. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn.

Digital Library

[28]

Yassine Ouali, Céline Hudelot, and Myriam Tami. 2020. Semi-supervised semantic segmentation with cross-consistency training. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 12674--12684.

[29]

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Proc. Advances in Neural Inf. Process. Syst., Vol. 33 (2020), 596--608.

[30]

Nasim Souly, Concetto Spampinato, and Mubarak Shah. 2017. Semi supervised semantic segmentation using generative adversarial network. In Proc. IEEE Int. Conf. Comp. Vis. 5688--5696.

[31]

Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Proc. Advances in Neural Inf. Process. Syst., Vol. 30 (2017).

[32]

Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. 2020. Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 43, 10 (2020), 3349--3364.

[33]

Yulin Wang, Gao Huang, Shiji Song, Xuran Pan, Yitong Xia, and Cheng Wu. 2021. Regularizing deep networks with semantic data augmentation. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https://doi.org/10.1109/TPAMI.2021.3052951

[34]

Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. 2019. Implicit Semantic Data Augmentation for Deep Networks. In Proc. Advances in Neural Inf. Process. Syst. 12635--12644.

[35]

Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. 2022. Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels. arXiv preprint arXiv:2203.03884 (2022).

[36]

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. 2021. SegFormer: Simple and efficient design for semantic segmentation with transformers. Proc. Advances in Neural Inf. Process. Syst., Vol. 34 (2021).

[37]

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020a. Unsupervised data augmentation for consistency training. Proc. Advances in Neural Inf. Process. Syst., Vol. 33 (2020), 6256--6268.

[38]

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020b. Self-training with noisy student improves imagenet classification. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 10687--10698.

[39]

Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. 2021. ST: Make Self-training Work Better for Semi-supervised Semantic Segmentation. arXiv preprint arXiv:2106.05095 (2021).

[40]

Jianlong Yuan, Zelu Deng, Shu Wang, and Zhenbo Luo. 2020. Multi receptive field network for semantic segmentation. In Proc. Winter Conf. on Appl. of Comp. Vis. IEEE, 1883--1892.

[41]

Jianlong Yuan, Yifan Liu, Chunhua Shen, Zhibin Wang, and Hao Li. 2021. A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation. In Proc. IEEE Int. Conf. Comp. Vis. 8229--8238.

[42]

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proc. IEEE Int. Conf. Comp. Vis. 6023--6032.

[43]

Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, and Fang Wen. 2021b. Robust mutual learning for semi-supervised semantic segmentation. arXiv preprint arXiv:2106.00609 (2021).

[44]

Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. 2021a. K-Net: Towards Unified Image Segmentation. In Proc. Advances in Neural Inf. Process. Syst.

[45]

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. 2017. Pyramid scene parsing network. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 2881--2890.

[46]

Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. 2021. Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentation. In Proc. IEEE Int. Conf. Comp. Vis. 7273--7282.

[47]

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ade20k dataset. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 633--641.

[48]

Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. 2021. PseudoSeg: Designing Pseudo Labels for Semantic Segmentation. Proc. Int. Conf. Learn. Representations (2021).

Cited By

Zhao HMeng HYang DXie XWu XLi QNiu JCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)GuidedNet: Semi-Supervised Multi-Organ Segmentation via Labeled Data Guide Unlabeled DataProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681526(886-895)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681526
Wang HZhang QLi YLi X(2024)AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00348(3627-3636)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00348
Yang XGong X(2024)Tuning-Free Universally-Supervised Semantic SegmentationIEEE Access10.1109/ACCESS.2024.351237912(187329-187342)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3512379
Show More Cited By

Index Terms

Semi-supervised Semantic Segmentation with Mutual Knowledge Distillation
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Scene understanding

Recommendations

Improving Semi-Supervised Semantic Segmentation with Dual-Level Siamese Structure Network
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Semi-supervised semantic segmentation (SSS) is an important task that utilizes both labeled and unlabeled data to reduce expenses on labeling training examples. However, the effectiveness of SSS algorithms is limited by the difficulty of fully exploiting ...
Improving Semi-Supervised Text Classification with Dual Meta-Learning
The goal of semi-supervised text classification (SSTC) is to train a model by exploring both a small number of labeled data and a large number of unlabeled data, such that the learned semi-supervised classifier performs better than the supervised ...
Semi- and Weakly- Supervised Semantic Segmentation with Deep Convolutional Neural Networks
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

Successful semantic segmentation methods typically rely on the training datasets containing a large number of pixel-wise labeled images. To alleviate the dependence on such a fully annotated training dataset, in this paper, we propose a semi- and weakly-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
192
Total Downloads

Downloads (Last 12 months)118
Downloads (Last 6 weeks)7

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhao HMeng HYang DXie XWu XLi QNiu JCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)GuidedNet: Semi-Supervised Multi-Organ Segmentation via Labeled Data Guide Unlabeled DataProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681526(886-895)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681526
Wang HZhang QLi YLi X(2024)AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00348(3627-3636)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00348
Yang XGong X(2024)Tuning-Free Universally-Supervised Semantic SegmentationIEEE Access10.1109/ACCESS.2024.351237912(187329-187342)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3512379
Dong JMeng ZLiu DLiu JZhao ZSu F(2024)Boundary-refined prototype generation: A general end-to-end paradigm for semi-supervised semantic segmentationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109021137(109021)Online publication date: Nov-2024
https://doi.org/10.1016/j.engappai.2024.109021
Yin JYan SChen TChen YYao Y(2024)Class Probability Space Regularization for semi-supervised semantic segmentationComputer Vision and Image Understanding10.1016/j.cviu.2024.104146249(104146)Online publication date: Dec-2024
https://doi.org/10.1016/j.cviu.2024.104146
Cao JChen JHuang SZhang D(2024)Leveraging Cross-Augmentation Consensus and Conflict for Semi-supervised Semantic SegmentationPattern Recognition10.1007/978-3-031-78398-2_6(89-104)Online publication date: 2-Dec-2024
https://doi.org/10.1007/978-3-031-78398-2_6
Yuan WLu XZhang RLiu Y(2023)FCKDNet: A Feature Condensation Knowledge Distillation Network for Semantic SegmentationEntropy10.3390/e2501012525:1(125)Online publication date: 7-Jan-2023
https://doi.org/10.3390/e25010125
CHEN H(2023)Contrastive Node Representation Learning in Graph via Mutual Channel Distillation2023 IEEE 6th International Conference on Pattern Recognition and Artificial Intelligence (PRAI)10.1109/PRAI59366.2023.10332051(745-750)Online publication date: 18-Aug-2023
https://doi.org/10.1109/PRAI59366.2023.10332051
Liang CWang WMiao JYang Y(2023)Logic-induced Diagnostic Reasoning for Semi-supervised Semantic Segmentation2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01484(16151-16162)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01484

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents