research-article

Siamese-rPPG network: remote photoplethysmography signal estimation from face videos

Authors:

Chiou-Ting Hsu,

Shang-Hung ChangAuthors Info & Claims

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

Pages 2066 - 2073

https://doi.org/10.1145/3341105.3373905

Published: 30 March 2020 Publication History

Abstract

Remote photoplethysmography (rPPG) is a contactless method for heart rate (HR) estimation from face videos. In this paper, we propose to estimate rPPG signals directly from input video sequences in an end-to-end manner. We propose a novel Siamese-rPPG network to simultaneously learn the heterogeneous and homogeneous features from two facial regions. Furthermore, to analyze the temporal periodicity of rPPG signals, we construct the network with 3D CNNs and jointly train the two-branch model under the negative Pearson loss function. Experimental results on three benchmark datasets: COHFACE, UBFC, and PURE, show that our method significantly outperforms existing methods with a large margin.

References

[1]

John Allen. 2007. Photoplethysmography and Its Application in Clinical Physiological Measurement. Physiological Measurement 28, 3 (Feb 2007), R1--R39.

[2]

Yannick Benezeth, Serge Bobbia, Keisuke Nakamura, Randy Gomez, and Julien Dubois. 2019. Probabilistic Signal Quality Metric for Reduced Complexity Unsupervised Remote Photoplethysinography. 1--5.

[3]

Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. 2017. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognition Letters (10 2017).

Digital Library

[4]

Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature Verification Using a "Siamese" Time Delay Neural Network. In Proceedings of the 6th International Conference on Neural Information Processing Systems (NIPS'93). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 737--744. http://dl.acm.org/citation.cfm?id=2987189.2987282

Digital Library

[5]

Weixuan Chen and Daniel McDuff. 2018. DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks. In Computer Vision - ECCV 2018, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 356--373.

Digital Library

[6]

Weixuan Chen and Daniel J. McDuff. 2018. DeepMag: Source Specific Motion Magnification Using Gradient Ascent. CoRR abs/1808.03338 (2018). arXiv:1808.03338 http://arxiv.org/abs/1808.03338

[7]

Gerard de Haan and Vincent Jeanne. 2013. Robust Pulse Rate From Chrominance-Based rPPG. IEEE Transactions on Biomedical Engineering 60, 10 (Oct 2013). 2878--2886.

[8]

Javier Hernandez-Ortega, Julian Fierrez, Aythami Morales, and Pedro Tome. 2018. Time Analysis of Pulse-Based Face Anti-Spoofing in Visible and NIR. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[9]

A. B. Hertzmann. 1937. Observations on the Finger Volume Pulse Recorded Photo-electrically. American Journal of Physiology 119 (1937), 334--335.

[10]

Guillaume Heusch, André Anjos, and Sébastien Marcel. 2017. A Reproducible Study on Remote Heart Rate Measurement. CoRR abs/1709.00962 (2017). arXiv:1709.00962 http://arxiv.org/abs/1709.00962

[11]

Branislav HollÃd'nder. 2018. Siamese Networks: Algorithm, Applications And PyTorch Implementation. https://becominghuman.ai/Siamese-networks-algorithm-applications-and-pytorch-implementation-4ffa3304c18 Last accessed on Aug 29, 2019.

[12]

Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009), 1755--1758.

Digital Library

[13]

Georg Lempe, Sebastian Zaunseder, Tom Wirthgen, Stephan Zipser, and Hagen Malberg. 2013. ROI Selection for Remote Photo plethysmography. In Bildverarbeitung für die Medizin 2013. Springer Berlin Heidelberg, Berlin, Heidelberg, 99--103.

[14]

Peixi Li, Keisuke Nakamura Yannick Benezeth, Randy Gomez, and Fan Yang. 2019. Model-based Region of Interest Segmentation for Remote Photoplethysinography. In 14th International Conference on Computer Vision Theory and Applications. 383--388.

[15]

Xiaobai Li, Iman Alikhani, Jingang Shi, Tapio Seppanen, Juhani Junttila, Kirsi Majamaa-Voltti, Mikko Tulppo, and Guoying Zhao. 2018. The OBF Database: A Large Face Video Database for Remote Physiological Signal Measurement and Atrial Fibrillation Detection. In 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018). 242--249.

[16]

Xiaobai Li, Jie Chen, Guoying Zhao, and Matti PietikÃd'inen. 2014. Remote Heart Rate Measurement from Face Videos under Realistic Situations. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. 4264--4271.

Digital Library

[17]

Yaojie Liu, Amin Jourabloo, and Xiaoming Liu. 2018. Learning Deep Models for Face Anti-Spoofing: Binary or Auxiliary Supervision. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 389--398.

[18]

Richard Macwan, Yannick Benezeth, and Alamin Mansouri. 2019. Heart rate estimation using remote photoplethysmography with multi-objective optimization. Biomedical Signal Processing and Control 49 (03 2019), 24--33.

[19]

Richard Macwan, Serge Bobbia, Yannick Benezeth, Julien Dubois, and Alamin Mansouri. 2018. Periodic Variance Maximization Using Generalized Eigenvalue Decomposition Applied to Remote Photoplethysmography Estimation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1413--14138.

[20]

Xuesong Niu, Hu Han, Shiguang Shan, and Xilin Chen. 2018. SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific. In 2018 24th International Conference on Pattern Recognition (ICPR). 3580--3585.

[21]

Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard. 2010. Non-contact Automated Cardiac Pulse Measurements Using Video Imaging and Blind Source Separation. Optics Express 18, 10 (2010), 10762--10774.

[22]

Ying Qiu, Yang Liu, Juan Arteaga-Falconi, Haiwei Dong, and Abdulmotaleb E. Saddik. 2019. EVM-CNN: Real-Time Contactless Heart Rate Estimation From Facial Video. IEEE Transactions on Multimedia 21, 7 (July 2019), 1778--1787.

[23]

Ronny Stricker, Steffen MÃijller, and Horst-Michael Gross. 2014. Non-contact video-based pulse rate measurement on a mobile service robot. Proceedings - IEEE International Workshop on Robot and Human Interactive Communication 2014, 1056--1062.

[24]

Sergey Tulyakov, Xavier Alameda-Pineda, Elisa Ricci, Lijun Yin, Jeffrey F. Cohn, and Nicu Sebe. 2016. Self-Adaptive Matrix Completion for Heart Rate Estimation from Face Videos under Realistic Conditions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2396--2404.

[25]

Radim Špetlík, Vojtěch Franc, Jan Čech, and Jiří Matas. 2018. Visual Heart Rate Estimation with Convolutional Neural Network. In Proceedings of British Machine Vision Conference.

[26]

Wim Verkruysse, Lars O. Svaasand, and J S. Nelson. 2008. Remote Plethysmo-graphic Imaging Using Ambient Light. Optics Express 16, 26 (2008), 21434--21445.

[27]

Wenjin Wang, Albertus C. den Brinker, Sander Stuijk, and Gerard de Haan. 2017. Algorithmic Principles of Remote PPG. IEEE Transactions on Biomedical Engineering 64, 7 (July 2017), 1479--1491.

[28]

Wenjin Wang, Sander Stuijk, and Gerard de Haan. 2016. A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation. IEEE Transactions on Biomedical Engineering 63, 9 (Sep. 2016), 1974--1984.

[29]

Zhi-Kuan Wang, Ying Kao, and Chiou-Ting Hsu. 2019. Vision-based Heart Rate Estimation via a Two-stream CNN. In 2019 IEEE International Conference on Image Processing (ICIP). 3327--3331.

[30]

Sun Yu, Sijung Hu, Vicente Azorin-Peris, Jonathon A. Chambers, Yisheng Zhu, and Stephen E. Greenwald. 2011. Motion-compensated noncontact imaging photoplethysmography to monitor cardiorespiratory status during exercise. Journal of biomedical optics 16 (07 2011), 077010.

[31]

Zitong Yu, Xiaobai Li, and Guoying Zhao. 2019. Recovering remote Photoplethysmograph Signal from Facial videos Using Spatio-Temporal Convolutional Networks. CoRR abs/1905.02419 (2019). arXiv:1905.02419 http://arxiv.org/abs/1905.02419

Cited By

Zhu QWong CLazri ZChen MFu CWu M(2025)A Comparative Study of Principled rPPG-Based Pulse Rate Tracking Algorithms for Fitness ActivitiesIEEE Transactions on Biomedical Engineering10.1109/TBME.2024.344278572:1(152-165)Online publication date: Jan-2025
https://doi.org/10.1109/TBME.2024.3442785
Boccignone GConte DCuculo VD’Amelio AGrossi GLanzarotti R(2025)Enhancing rPPG pulse-signal recovery by facial sampling and PSD ClusteringBiomedical Signal Processing and Control10.1016/j.bspc.2024.107158101(107158)Online publication date: Mar-2025
https://doi.org/10.1016/j.bspc.2024.107158
Liu TXiao HSun YZuo KLi ZYang ZLiu S(2025)PhysKANNet: A KAN-based model for multiscale feature extraction and contextual fusion in remote physiological measurementBiomedical Signal Processing and Control10.1016/j.bspc.2024.107111100(107111)Online publication date: Feb-2025
https://doi.org/10.1016/j.bspc.2024.107111
Show More Cited By

Index Terms

Siamese-rPPG network: remote photoplethysmography signal estimation from face videos
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Activity recognition and understanding

Recommendations

Multi-scale Multi-structure Siamese Network (MMSNet) for Primary Open-Angle Glaucoma Prediction
Machine Learning in Medical Imaging
Abstract
Primary open-angle glaucoma (POAG) is one of the leading causes of irreversible blindness in the United States and worldwide. POAG prediction before onset plays an important role in early treatment. Although deep learning methods have been ...
Prediction of cardiovascular risk by measuring carotid intima media thickness from an ultrasound image for type II diabetic mellitus subjects using machine learning and transfer learning techniques
Abstract
Cardiovascular disease (CVD) is a fatal disease that causes increased death in developing and developed nations. Among the various reasons, the increase in carotid intima media thickness (CIMT) is also a significant reason for CVD. It is expected ...
Remote photoplethysmography application to the analysis of time-frequency changes of human heart rate variability
FRUCT '18: Proceedings of the 18th Conference of Open Innovations Association FRUCT

In this article we present the possibilities of using remote photoplethysmography (rPPG or imaging PPG) technology to estimate time-frequency changes of human heart rate variability. We propose improvements for algorithm presented in our recent study. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

March 2020

2348 pages

ISBN:9781450368667

DOI:10.1145/3341105

Conference Chairs:
Chih-Cheng Hung
Kennesaw State University
,
Tomas Cerny
Baylor University
,
Program Chairs:
Dongwan Shin
New Mexico Tech
,
Alessio Bechini
University of Pisa, Italy

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Qualcomm
Ministry of Science and Technology (MOST)

Conference

SAC '20

Sponsor:

SIGAPP

SAC '20: The 35th ACM/SIGAPP Symposium on Applied Computing

March 30 - April 3, 2020

Brno, Czech Republic

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25

Sponsor:
sigapp

The 40th ACM/SIGAPP Symposium on Applied Computing

March 31 - April 4, 2025

Catania , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

58
Total Citations
View Citations
1,407
Total Downloads

Downloads (Last 12 months)157
Downloads (Last 6 weeks)28

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhu QWong CLazri ZChen MFu CWu M(2025)A Comparative Study of Principled rPPG-Based Pulse Rate Tracking Algorithms for Fitness ActivitiesIEEE Transactions on Biomedical Engineering10.1109/TBME.2024.344278572:1(152-165)Online publication date: Jan-2025
https://doi.org/10.1109/TBME.2024.3442785
Boccignone GConte DCuculo VD’Amelio AGrossi GLanzarotti R(2025)Enhancing rPPG pulse-signal recovery by facial sampling and PSD ClusteringBiomedical Signal Processing and Control10.1016/j.bspc.2024.107158101(107158)Online publication date: Mar-2025
https://doi.org/10.1016/j.bspc.2024.107158
Liu TXiao HSun YZuo KLi ZYang ZLiu S(2025)PhysKANNet: A KAN-based model for multiscale feature extraction and contextual fusion in remote physiological measurementBiomedical Signal Processing and Control10.1016/j.bspc.2024.107111100(107111)Online publication date: Feb-2025
https://doi.org/10.1016/j.bspc.2024.107111
Zou BZhao YHu XHe CYang T(2024)Remote physiological signal recovery with efficient spatio-temporal modelingFrontiers in Physiology10.3389/fphys.2024.142835115Online publication date: 14-Oct-2024
https://doi.org/10.3389/fphys.2024.1428351
Juan CChenchu YRencheng SJing FYu L(2024)Facial video-based heart rate measurement against irregular motion artifactsJournal of Image and Graphics10.11834/jig.23042829:7(2024-2034)Online publication date: 2024
https://doi.org/10.11834/jig.230428
Qia nLi KGuo DHu BWang MCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Cluster-Phys: Facial Clues Clustering Towards Efficient Remote Physiological MeasurementProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680670(330-339)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680670
El Boussaki HLatif RSaddik A(2024)Deep Learning Methods for Non-Contact Heart Rate Monitoring: A Review2024 World Conference on Complex Systems (WCCS)10.1109/WCCS62745.2024.10765543(1-6)Online publication date: 11-Nov-2024
https://doi.org/10.1109/WCCS62745.2024.10765543
Zhong RZhou YGou C(2024)EST-TSANet: Video-Based Remote Heart Rate Measurement Using Temporal Shift Attention Network and ESTmapIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2023.333141473(1-14)Online publication date: 2024
https://doi.org/10.1109/TIM.2023.3331414
Li QGuo DQian WTian XSun XZhao HWang M(2024)Channel-Wise Interactive Learning for Remote Heart Rate Estimation From Facial VideoIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.333240834:6(4542-4555)Online publication date: Jun-2024
https://doi.org/10.1109/TCSVT.2023.3332408
Qian WGuo DLi KZhang XTian XYang XWang M(2024)Dual-Path TokenLearner for Remote Photoplethysmography-Based Physiological Measurement With Facial VideosIEEE Transactions on Computational Social Systems10.1109/TCSS.2024.335671311:3(4465-4477)Online publication date: Jun-2024
https://doi.org/10.1109/TCSS.2024.3356713
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten