research-article

Modality-agnostic Augmented Multi-Collaboration Representation for Semi-supervised Heterogenous Face Recognition

Authors:

Xinbo GaoAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 4647 - 4656

https://doi.org/10.1145/3581783.3612355

Published: 27 October 2023 Publication History

Abstract

Heterogeneous face recognition (HFR) aims to match input face identity across different image modalities. Due to the existing large modality gap and the limited number of training data, HFR is still a challenging problem in biometrics and draws more and more attention. Existing researchers always extract modality invariant features or generate homogeneous images to decrease the modality gap, lacking abundant labeled data to avoid the overfitting problem. In this paper, we proposed a novel Modality-Agnostic Augmented Multi-Collaboration representation for Heterogeneous Face Recognition (MAMCO-HFR) in a semi-supervised manner. The modality-agnostic augmentation strategy is proposed to generate adversarial perturbations to map unlabeled faces into the modality-agnostic domain. The multi-collaboration feature constraint is designed to mine the inherent relationships between diverse layers for discriminative representation. Experiments on several large-scale heterogeneous face datasets (CASIA NIR-VIS 2.0, LAMP-HQ and Tufts Face dataset) prove the proposed algorithm can achieve superior performance compared with state-of-the-art methods. The source code is available at https://github.com/xiyin11/Semi-HFR.

References

[1]

Ann Theja Alex, Vijayan K Asari, and Alex Mathew. 2013. Local difference of gaussian binary pattern: Robust features for face sketch recognition. In 2013 IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 1211--1216.

Digital Library

[2]

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. 2019. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems 32 (2019).

[3]

Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). Ieee, 39--57.

[4]

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. 2020. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 702--703.

[5]

Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5203--5212.

[6]

Terrance DeVries and Graham W Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017).

[7]

Xing Di, Shuowen Hu, and Vishal M Patel. 2021. Heterogeneous face frontalization via domain agnostic learning. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). IEEE, 01-08.

Digital Library

[8]

Xing Di, Benjamin S Riggan, Shuowen Hu, Nathaniel J Short, and Vishal M Patel. 2021. Multi-scale thermal to visible face verification via attribute guided synthesis. IEEE Transactions on Biometrics, Behavior, and Identity Science 3, 2 (2021), 266--280.

[9]

Boyan Duan, Chaoyou Fu, Yi Li, Xingguang Song, and Ran He. 2020. Cross-spectral face hallucination via disentangling independent factors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7930--7938.

[10]

Chaoyou Fu, Xiang Wu, Yibo Hu, Huaibo Huang, and Ran He. 2019. Dual variational generation for low shot heterogeneous face recognition. Advances in neural information processing systems 32 (2019).

[11]

Chaoyou Fu, Xiang Wu, Yibo Hu, Huaibo Huang, and Ran He. 2021. Dvg-face: Dual variational generation for heterogeneous face recognition. IEEE transactions on pattern analysis and machine intelligence 44, 6 (2021), 2938--2952.

[12]

Chaoyou Fu, Xiaoqiang Zhou, Weizan He, and Ran He. 2022. Towards Lightweight Pixel-Wise Hallucination for Heterogeneous Face Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

Digital Library

[13]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139--144.

Digital Library

[14]

Yves Grandvalet and Yoshua Bengio. 2004. Semi-supervised learning by entropy minimization. Advances in neural information processing systems 17 (2004).

[15]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[16]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[17]

Lingxiao He, Wu Liu, Jian Liang, Kecheng Zheng, Xingyu Liao, Peng Cheng, and Tao Mei. 2021. Semi-supervised domain generalizable person re-identification. arXiv preprint arXiv:2108.05045 (2021).

[18]

Ran He, Yi Li, Xiang Wu, Lingxiao Song, Zhenhua Chai, and Xiaolin Wei. 2021. Coupled adversarial learning for semi-supervised heterogeneous face recognition. Pattern Recognition 110 (2021), 107618.

[19]

Ran He, Xiang Wu, Zhenan Sun, and Tieniu Tan. 2018. Wasserstein CNN: Learning invariant features for NIR-VIS face recognition. IEEE transactions on pattern analysis and machine intelligence 41, 7 (2018), 1761--1773.

[20]

Weipeng Hu and Haifeng Hu. 2021. Orthogonal modality disentanglement and representation alignment network for NIR-VIS face recognition. IEEE Transactions on Circuits and Systems for Video Technology 32, 6 (2021), 3630--3643.

[21]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to- image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125--1134.

[22]

Meina Kan, Shiguang Shan, Haihong Zhang, Shihong Lao, and Xilin Chen. 2015. Multi-view discriminant analysis. IEEE transactions on pattern analysis and machine intelligence 38, 1 (2015), 188--194.

[23]

Dong-Hyun Lee et al. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, Vol. 3. 896.

[24]

Dong-Hyun Lee et al. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, Vol. 3. 896.

[25]

Zhen Lei, Dong Yi, and Stan Z Li. 2012. Discriminant image filter learning for face recognition with local binary pattern like representation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2512--2517.

[26]

Hangyu Li, Nannan Wang, Xinpeng Ding, Xi Yang, and Xinbo Gao. 2021. Adaptively learning facial expression representation via cf labels and distillation. IEEE Transactions on Image Processing 30 (2021), 2016--2028.

[27]

Hangyu Li, Nannan Wang, Xi Yang, Xiaoyu Wang, and Xinbo Gao. 2022. Towards semi-supervised deep facial expression recognition with an adaptive confidence margin. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4166--4175.

[28]

Stan Li, Dong Yi, Zhen Lei, and Shengcai Liao. 2013. The casia nir-vis 2.0 face database. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 348--353.

Digital Library

[29]

Zhifeng Li, Dihong Gong, Qiang Li, Dacheng Tao, and Xuelong Li. 2016. Mutual component analysis for heterogeneous face recognition. ACM Transactions on Intelligent Systems and Technology (TIST) 7, 3 (2016), 1--23.

Digital Library

[30]

Decheng Liu, Xinbo Gao, Nannan Wang, Jie Li, and Chunlei Peng. 2020. Coupled attribute learning for heterogeneous face recognition. IEEE Transactions on Neural Networks and Learning Systems 31, 11 (2020), 4699--4712.

[31]

Decheng Liu, Xinbo Gao, Nannan Wang, Chunlei Peng, and Jie Li. 2021. Iterative local re-ranking with attribute guided synthesis for face sketch recognition. Pattern Recognition 109 (2021), 107579.

Digital Library

[32]

Decheng Liu, Jie Li, Nannan Wang, Chunlei Peng, and Xinbo Gao. 2018. Composite components-based face sketch recognition. Neurocomputing 302 (2018), 46--54.

[33]

Mandi Luo, Haoxue Wu, Huaibo Huang, Weizan He, and Ran He. 2022. Memory-modulated transformer network for heterogeneous face recognition. IEEE Trans- actions on Information Forensics and Security 17 (2022), 2095--2109.

[34]

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017).

[35]

Khawla Mallat, Naser Damer, Fadi Boutros, Arjan Kuijper, and Jean-Luc Dugelay. 2019. Cross-spectrum thermal to visible face recognition based on cascaded image synthesis. In 2019 International Conference on Biometrics (ICB). IEEE, 1--8.

[36]

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 5191--5198.

[37]

Karen Panetta, Qianwen Wan, Sos Agaian, Srijith Rajeev, Shreyas Kamath, Rahul Rajendran, Shishir Paramathma Rao, Aleksandra Kaszowska, Holly A Taylor, Arash Samani, et al. 2018. A comprehensive database for benchmarking imaging systems. IEEE transactions on pattern analysis and machine intelligence 42, 3 (2018), 509--520.

[38]

Neehar Peri, Joshua Gleason, Carlos D Castillo, Thirimachos Bourlai, Vishal M Patel, and Rama Chellappa. 2021. A synthesis-based approach for thermal-to- visible face verification. In 2021 16th IEEE international conference on automatic face and gesture recognition (FG 2021). IEEE, 01--08.

Digital Library

[39]

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. 2016. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advances in neural information processing systems 29 (2016).

[40]

Shreyas Saxena and Jakob Verbeek. 2016. Heterogeneous face recognition with CNNs. In Computer Vision--ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15--16, 2016, Proceedings, Part III 14. Springer, 483--491.

[41]

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE interna- tional conference on computer vision. 618--626.

[42]

Ming Shao, Dmitry Kit, and Yun Fu. 2014. Generalized transfer subspace learning through low-rank constraint. International Journal of Computer Vision 109, 1--2 (2014), 74--93.

Digital Library

[43]

Abhishek Sharma and David W Jacobs. 2011. Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch. In CVPR 2011. IEEE, 593--600.

Digital Library

[44]

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems 33 (2020), 596--608.

[45]

Lingxiao Song, Man Zhang, Xiang Wu, and Ran He. 2018. Adversarial discriminative heterogeneous face recognition. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.

[46]

Zongcai Sun, Chaoyou Fu, Mandi Luo, and Ran He. 2021. Self-Augmented Heterogeneous Face Recognition. In 2021 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 1--8.

[47]

Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).

[48]

Rui Wang, Jimei Yang, Dong Yi, and Stan Z Li. 2009. An analysis-by-synthesis method for heterogeneous face biometrics. In Advances in Biometrics: Third International Conference, ICB 2009, Alghero, Italy, June 2-5, 2009. Proceedings 3. Springer, 319--326.

Digital Library

[49]

Xiang Wu, Ran He, Zhenan Sun, and Tieniu Tan. 2018. A light CNN for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security 13, 11 (2018), 2884--2896.

[50]

Xiang Wu, Huaibo Huang, Vishal M Patel, Ran He, and Zhenan Sun. 2019. Disentangled variational representation for heterogeneous face recognition. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 9005--9012.

Digital Library

[51]

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. Advances in neural information processing systems 33 (2020), 6256--6268.

[52]

Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. 2020. Knowledge distillation meets self-supervision. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX. Springer, 588--604.

[53]

Fanglei Xue, Qiangchang Wang, and Guodong Guo. 2021. Transfer: Learning relation-aware facial expression representations with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3601--3610.

[54]

Ziming Yang, Jian Liang, Chaoyou Fu, Mandi Luo, and Xiao-Yu Zhang. 2022. Heterogeneous Face Recognition via Face Synthesis With Identity-Attribute Disentanglement. IEEE Transactions on Information Forensics and Security 17 (2022), 1344--1358.

[55]

Bangjie Yin, Luan Tran, Haoxiang Li, Xiaohui Shen, and Xiaoming Liu. 2019. To- wards interpretable face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9348--9357.

[56]

Yu Yin, Songyao Jiang, Joseph P Robinson, and Yun Fu. 2020. Dual-attention GAN for large-pose face frontalization. In 2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020). IEEE, 249--256.

Digital Library

[57]

Aijing Yu, Haoxue Wu, Huaibo Huang, Zhen Lei, and Ran He. 2021. LAMP-HQ: A large-scale multi-pose high-quality database and benchmark for NIR-VIS face recognition. International Journal of Computer Vision 129, 5 (2021), 1467--1483.

[58]

He Zhang, Vishal M Patel, Benjamin S Riggan, and Shuowen Hu. 2017. Generative adversarial network-based synthesis of visible faces from polarimetrie thermal faces. In 2017 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 100--107.

Digital Library

[59]

Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. 2020. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning. PMLR, 11278--11287.

[60]

Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. 2019. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3713--3722.

[61]

Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4320--4328.

[62]

Jian Zhao, Yu Cheng, Yan Xu, Lin Xiong, Jianshu Li, Fang Zhao, Karlekar Jayashree, Sugiri Pranata, Shengmei Shen, Junliang Xing, et al. 2018. Towards pose invariant face recognition in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2207--2216.

Cited By

George AMarcel S(2024)From Modalities to Styles: Rethinking the Domain Gap in Heterogeneous Face RecognitionIEEE Transactions on Biometrics, Behavior, and Identity Science10.1109/TBIOM.2024.33653506:4(475-485)Online publication date: Oct-2024
https://doi.org/10.1109/TBIOM.2024.3365350
George AMarcel S(2024)Modality Agnostic Heterogeneous Face Recognition with Switch Style Modulators2024 IEEE International Joint Conference on Biometrics (IJCB)10.1109/IJCB62174.2024.10744437(1-10)Online publication date: 15-Sep-2024
https://doi.org/10.1109/IJCB62174.2024.10744437
Ermaimaiti YWang PAi Shanjiang Y(2024)Face photo-line drawings synthesis based on local extraction preserving generative adversarial networksThe Imaging Science Journal10.1080/13682199.2024.2315848(1-16)Online publication date: 11-Feb-2024
https://doi.org/10.1080/13682199.2024.2315848
Show More Cited By

Index Terms

Modality-agnostic Augmented Multi-Collaboration Representation for Semi-supervised Heterogenous Face Recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object identification
        Object recognition
      2. Computer vision tasks
        Biometrics

Recommendations

Scalable Semi-Supervised Clustering for Face Recognition with Insufficient Labelled Samples
Abstract
Face recognition is an effortless job for humans; however, it is computationally challenging as it is difficult to develop a computational model for recognizing faces. It becomes more challenging especially when the number of labeled examples of ...
A Semi-supervised 2DPCA Face Recognition Method Based on Self-Training
ICCIS '12: Proceedings of the 2012 Fourth International Conference on Computational and Information Sciences

By combining self-training method of the semi-supervised learning with two-dimensional principal component analysis (2DPCA), a semi-supervised learning based face recognition method is proposed. On the basis of two-dimensional principal component ...
Age-Invariant Face Recognition

One of the challenges in automatic face recognition is to achieve temporal invariance. In other words, the goal is to come up with a representation and matching scheme that is robust to changes due to facial aging. Facial aging is a complex process that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
144
Total Downloads

Downloads (Last 12 months)99
Downloads (Last 6 weeks)7

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

George AMarcel S(2024)From Modalities to Styles: Rethinking the Domain Gap in Heterogeneous Face RecognitionIEEE Transactions on Biometrics, Behavior, and Identity Science10.1109/TBIOM.2024.33653506:4(475-485)Online publication date: Oct-2024
https://doi.org/10.1109/TBIOM.2024.3365350
George AMarcel S(2024)Modality Agnostic Heterogeneous Face Recognition with Switch Style Modulators2024 IEEE International Joint Conference on Biometrics (IJCB)10.1109/IJCB62174.2024.10744437(1-10)Online publication date: 15-Sep-2024
https://doi.org/10.1109/IJCB62174.2024.10744437
Ermaimaiti YWang PAi Shanjiang Y(2024)Face photo-line drawings synthesis based on local extraction preserving generative adversarial networksThe Imaging Science Journal10.1080/13682199.2024.2315848(1-16)Online publication date: 11-Feb-2024
https://doi.org/10.1080/13682199.2024.2315848
Elboushaki AHannane RAfdel K(2024)Similarity-based face image retrieval using sparsely embedded deep features and binary code learningInternational Journal of Multimedia Information Retrieval10.1007/s13735-024-00337-513:3Online publication date: 8-Jul-2024
https://doi.org/10.1007/s13735-024-00337-5
Zhao CQin YZhang BZhao YWu B(2024)An end-to-end occluded person re-identification network with smoothing corrupted feature predictionArtificial Intelligence Review10.1007/s10462-024-11047-z58:2Online publication date: 20-Dec-2024
https://doi.org/10.1007/s10462-024-11047-z
Liu YLi XLiu YZhong W(2024)SimpliFusion: a simplified infrared and visible image fusion networkThe Visual Computer10.1007/s00371-024-03423-1Online publication date: 29-May-2024
https://doi.org/10.1007/s00371-024-03423-1

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents