A Histo-Puzzle Network for Weakly Supervised Semantic Segmentation of Histological Tissue Type
Abstract
Digital pathological images with a large range of Histological Tissue Types (HTTs) contain more sophisticated contours than natural images. In recent years, deep learning algorithms have been widely applied to assist HTT analysis in a weakly-supervised manner by exploiting the class activation maps (CAM). However, the previous methods tend to confusedly activate the most discriminative regions of feature maps, resulting in incomplete segmented contour. This paper proposes a Histo-Puzzle network to improve the HTTs classification and segmentation based on patch-level self-supervised learning. Specifically, our model separates the HTT images into tiled patches by a puzzle module. Then we train a classifier on the supervision of reconstructed CAMs and image-level labels simultaneously. Experiments are conducted on the digital pathology database with 51 hierarchical HTTs. The experimental results show that our proposed method outperforms previous state-of-the-art methods on segmentation tasks of morphological and functional types.
References
[1]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from http://arxiv.org/abs/1409.1556
[2]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, 2016. IEEE, 770-778. https://doi.org/10.1109/CVPR.2016.90
[3]
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R. Manmatha, Mu Li, and Alexander J. Smola. 2022. ResNeSt: Split-Attention Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2022, New Orleans, LA, 2022. IEEE, 2735-2745. https://doi.org/10.1109/CVPRW56347.2022.00309
[4]
Bolei Zhou, Aditya Khosla, Gata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning Deep Features for Discriminative Localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, 2016. IEEE, 2921-2929. https://doi.org/10.1109/CVPR.2016.319
[5]
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, 618-626. https://doi.org/10.1109/ICCV.2017.74
[6]
Jiwoon Ahn and Suha Kwak. 2018. Learning Pixel-Level Semantic Affinity With Image-Level Supervision for Weakly Supervised Semantic Segmentation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. Computer Vision Foundation / IEEE Computer Society, 4981-4990. https://doi.org/10.1109/CVPR.2018.00523
[7]
Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen. 2020. Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, 2020. IEEE, 12272-12281. https://doi.org/10.1109/CVPR42600.2020.01229
[8]
Yonghang Guan, Jun Zhang, Kuan Tian, Sen Yang, Pei Dong, Jinxi Xiang, Wei Yang, Junzhou Huang, Yuyao Zhang, and Xiao Han. 2022. Node-aligned Graph Convolutional Network for Whole-slide Image Representation and Classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, 2022. IEEE, 18791-18801. https://doi.org/10.1109/CVPR52688.2022.01825
[9]
Sanghyun Jo and Injae Yu. 2021. Puzzle-CAM: Improved Localization Via Matching Partial And Full Features. In 2021 IEEE International Conference on Image Processing, Anchorage, AK, 2021. IEEE, 639-643. https://doi.org/10.1109/ICIP42928.2021.9506058
[10]
Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. 2019. FickleNet: Weakly and Semi-Supervised Semantic Image Segmentation Using Stochastic Inference. In IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, 2019. IEEE, 5267-5276. https://doi.org/10.1109/CVPR.2019.00541
[11]
Anna Khoreva, Rodrigo Benenson, Jan Hendrik Hosang, Matthias Hein, and Bernt Schiele. 2017. Simple Does It: Weakly Supervised Instance and Semantic Segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2017. IEEE, 1665-1674. https://doi.org/10.1109/CVPR.2017.181
[12]
Lyndon Chan, Mahdi S. Hosseini, Corwyn Rowsell, Konstantinos N. Plataniotis, and Savvas Damaskinos. 2019. HistoSegNet: Semantic Segmentation of Histological Tissue Type in Whole Slide Images. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 10661-10670. https://doi.org/10.1109/ICCV.2019.01076
[13]
Kangning Liu, Yiqiu Shen, Nan Wu, Jakub Piotr Chledowski, Carlos Fernandezgranda, and Krzysztof J. Geras. 2021. Weakly-supervised High-resolution Segmentation of Mammography Images for Breast Cancer Diagnosis. In Medical Imaging with Deep Learning, 2021, Lubeck, Germany. PMLR, 451-472. https://proceedings.mlr.press/v143/liu21b.html
[14]
Chu Han, Jiatai Lin, Jinhai Mai, Yi Wang, Qingling Zhang, Bingchao Zhao, Xin Chen, Xipeng Pan, Zhenwei Shi, Zeyan Xu, Su Yao, Lixu Yan, Huan Lin, Xiaomei Huang, Changhong Liang, Guoqiang Han, and Zaiyi Liu. 2022. Multi-layer pseudo-supervision for histopathology tissue semantic segmentation using patch-level classification labels. Medical Image Analysis. 80 (2022), 102487. https://doi.org/10.1016/j.media.2022.102487
[15]
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv:1312.6034. Retrieved from http://arxiv.org/abs/1312.6034
[16]
Liangchieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. 2018. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE transactions on pattern analysis and machine intelligence 40, 4 (2048), 834-848. https://doi.org/10.1109/TPAMI.2017.2699184
[17]
Mahdi S. Hosseini, Lyndon Chan, Gabriel Tse, Michael Tang, Jun Deng, Sajad Norouzi, Corwyn Rowsell, Konstantinos N. Plataniotis, and Savvas Damaskinos. 2019. Atlas of Digital Pathology: A Generalized Hierarchical Histological Tissue Type-Annotated Database for Deep Learning. In IEEE Conference on Computer Vision and Pattern Recognition, Long Beach. IEEE, 11747-11756. https://doi.org/10.1109/CVPR.2019.01202
[18]
Ihor Konovalenko, Pavlo Maruschak, Vitaly Brevus, and Olegas Prentkovskis. 2021. Recognition of Scratches and Abrasions on Metal Surfaces Using a Classifier Based on a Convolutional Neural Network. Metals. 11, 4. https://www.mdpi.com/2075-4701/11/4/549
[19]
Ihor Konovalenko, Pavlo Maruschak, Janette Brezinová, Olegas Prentkovskis, and Jakub Brezina. 2022. Research of U-Net-Based CNN Architectures for Metal Surface Defect Detection. Machines. 10, 5. https://www.mdpi.com/2075-1702/10/5/327
[20]
Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N. Balasubramanian. 2018. Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks. In 2018 IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, NV, 2018. IEEE Computer Society, 839-847. https://doi.org/10.1109/WACV.2018.00097
[21]
Alexander Kolesnikov and Christoph H. Lampert. 2016. Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, 2016, Proceedings, Part IV. Springer, 695-711. https://doi.org/10.1007/978-3-319-46493-0_42
[22]
Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang. 2018. Weakly-Supervised Semantic Segmentation Network With Deep Seeded Region Growing. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018. IEEE, 7014-7023. https://doi.org/10.1109/CVPR.2018.00733
Index Terms
- A Histo-Puzzle Network for Weakly Supervised Semantic Segmentation of Histological Tissue Type
Index terms have been assigned to the content through auto-classification.
Recommendations
Boosted MIML method for weakly-supervised image semantic segmentation
Weakly-supervised image semantic segmentation aims to segment images into semantically consistent regions with only image-level labels are available, and is of great significance for fine-grained image analysis, retrieval and other possible ...
White Matter Tract Segmentation with Self-supervised Learning
Medical Image Computing and Computer Assisted Intervention – MICCAI 2020AbstractWhite matter tract segmentation based on diffusion magnetic resonance imaging (dMRI) plays an important role in brain analysis. Deep learning based methods of white matter tract segmentation have been proposed to improve the segmentation accuracy. ...
Semantic-aware superpixel for weakly supervised semantic segmentation
AAAI'23/IAAI'23/EAAI'23: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial IntelligenceWeakly-supervised semantic segmentation aims to train a semantic segmentation network using weak labels. Among weak labels, image-level label has been the most popular choice due to its simplicity. However, since image-level labels lack accurate object ...
Comments
Information & Contributors
Information
Published In
March 2023
598 pages
ISBN:9781450399449
DOI:10.1145/3590003
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 May 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Cooperation Projects between Chongqing Universities in Chongqing and Institutions Affiliated with the Chinese Academy of Sciences
- the National Nature Science Foundation of China under grant No. 61902370
Conference
CACML 2023
CACML 2023: 2023 2nd Asia Conference on Algorithms, Computing and Machine Learning
March 17 - 19, 2023
Shanghai, China
Acceptance Rates
CACML '23 Paper Acceptance Rate 93 of 241 submissions, 39%;
Overall Acceptance Rate 93 of 241 submissions, 39%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 174Total Downloads
- Downloads (Last 12 months)134
- Downloads (Last 6 weeks)23
Reflects downloads up to 14 Sep 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in