research-article

Open access

DeepMag: Source-Specific Change Magnification Using Gradient Ascent

Authors:

Daniel McDuffAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 40, Issue 1

Article No.: 2, Pages 1 - 14

https://doi.org/10.1145/3408865

Published: 09 September 2020 Publication History

All formats PDF

Abstract

Many important physical phenomena involve subtle signals that are difficult to observe with the unaided eye, yet visualizing them can be very informative. Current motion magnification techniques can reveal these small temporal variations in video, but require precise prior knowledge about the target signal, and cannot deal with interference motions at a similar frequency. We present DeepMag, an end-to-end deep neural video-processing framework based on gradient ascent that enables automated magnification of subtle color and motion signals from a specific source, even in the presence of large motions of various velocities. The advantages of DeepMag are highlighted via the task of video-based physiological visualization. Through systematic quantitative and qualitative evaluation of the approach on videos with different levels of head motion, we compare the magnification of pulse and respiration to existing state-of-the-art methods. Our method produces magnified videos with substantially fewer artifacts and blurring whilst magnifying the physiological changes by a similar degree.

Supplementary Material

chen (chen.zip)

Supplemental movie, appendix, image and software files for, DeepMag: Source-Specific Change Magnification Using Gradient Ascent

Download
56.62 MB

References

[1]

Guha Balakrishnan, Fredo Durand, and John Guttag. 2013. Detecting pulse from head motions in video. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2013), 3430--3437.

Digital Library

[2]

Pak-Hei Chan, Chun-Ka Wong, Yukkee C. Poh, Louise Pun, Wangie Wan-Chiu Leung, Yu-Fai Wong, Michelle Man-Ying Wong, Ming-Zher Poh, Daniel Wai-Sing Chu, and Chung-Wah Siu. 2016. Diagnostic performance of a smartphone-based photoplethysmographic application for atrial fibrillation screening in a primary care setting. Journal of the American Heart Association 5, 7 (2016), e003428.

[3]

Weixuan Chen and Daniel McDuff. 2018. DeepPhys: Video-based physiological measurement using convolutional attention networks. arXiv preprint arXiv:1805.07888 (2018).

[4]

Gerard de Haan and Vincent Jeanne. 2013. Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering 60, 10 (2013), 2878--2886.

[5]

Gerard de Haan and Arno van Leest. 2014. Improved motion robustness of remote-PPG by using the blood volume pulse signature. Physiological Measurement 35, 9 (2014), 1913.

[6]

Mohamed Elgharib, Mohamed Hefeeda, Fredo Durand, and William T. Freeman. 2015. Video magnification in presence of large motions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4119--4127.

[7]

Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing higher-layer features of a deep network. University of Montreal 1341, 3 (2009), 1.

[8]

Justin R. Estepp, Ethan B. Blackford, and Christopher M. Meier. 2014. Recovering pulse rate during motion artifact with a multi-imager array for non-contact imaging photoplethysmography. In Proceedings of the 2014 IEEE International Conference on Systems, Man and Cybernetics (SMC). IEEE, 1462--1469.

[9]

Thomas B. Fitzpatrick. 1988. The validity and practicality of sun-reactive skin types I through VI. Archives of Dermatology 124, 6 (1988), 869--871.

[10]

Martin Fuchs, Tongbo Chen, Oliver Wang, Ramesh Raskar, Hans-Peter Seidel, and Hendrik P. A. Lensch. 2010. Real-time temporal shaping of high-speed video streams. Computers 8 Graphics 34, 5 (2010), 575--584.

[11]

Temujin Gautama and M. A. Van Hulle. 2002. A phase-based approach to the estimation of the optical flow field using spatial filtering. IEEE Transactions on Neural Networks 13, 5 (2002), 1127--1136.

Digital Library

[12]

Rik Janssen, Wenjin Wang, Andreia Moço, and Gerard de Haan. 2016. Video-based respiration monitoring with automatic region of interest detection. Physiological Measurement 37, 1 (2016), 100--114. http://stacks.iop.org/0967-3334/37/i=1/a=100?key=crossref.be9e80b618c48e376025e318d84dff96.

[13]

Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. In Advances in Neural Information Processing Systems. 972--981.

[14]

Julian F. P. Kooij and Jan C. van Gemert. 2016. Depth-aware motion magnification. In Proceedings of the European Conference on Computer Vision. Springer, 467--482.

[15]

Antony Lam and Yoshinori Kuno. 2015. Robust heart rate measurement from video using select random patches. In Proceedings of the IEEE International Conference on Computer Vision. 3640--3648.

Digital Library

[16]

Ce Liu, Antonio Torralba, William T. Freeman, Frédo Durand, and Edward H. Adelson. 2005. Motion magnification. ACM Transactions on Graphics (TOG) 24, 3 (2005), 519--526.

Digital Library

[17]

Daniel McDuff. 2018. Deep super resolution for recovering physiological information from videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[18]

Daniel McDuff, Justin R. Estepp, Alyssa M. Piasecki, and Ethan B. Blackford. 2015. A survey of remote optical photoplethysmographic imaging methods. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 6398--6404.

[19]

Daniel McDuff, Sarah Gontarek, and Rosalind Picard. 2014. Improvements in remote cardio-pulmonary measurement using a five band digital camera. IEEE Transactions on Biomedical Engineering 61, 10 (2014), 2593--2601.

[20]

Hamed Monkaresi, Rafael A. Calvo, and Hong Yan. 2014. A machine learning approach to improve contactless heart rate monitoring using a webcam. IEEE Journal of Biomedical and Health Informatics 18, 4 (2014), 1153--1160.

[21]

Alexander Mordvintsev, Christopher Olah, and Mike Tyka. 2015. Deepdream-a code example for visualizing neural networks. Google Res 2 (2015).

[22]

Masahiro Mori. 1970. The uncanny valley. Energy 7, 4 (1970), 33--35.

[23]

Tae-Hyun Oh, Ronnachai Jaroensri, Changil Kim, Mohamed Elgharib, Frédo Durand, William T. Freeman, and Wojciech Matusik. 2018. Learning-based video motion magnification. arXiv preprint arXiv:1804.02684 (2018).

[24]

Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. 2017. Feature visualization. Distill 2, 11 (2017).

[25]

Ahmed Osman, Jay Turcot, and Rana El Kaliouby. 2015. Supervised learning approach to remote heart rate estimation from facial videos. In Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 1. IEEE, 1--6.

[26]

Silvia L. Pintea and Jan C. van Gemert. 2016. Making a case for learning motion representations with phase. In Proceedings of the European Conference on Computer Vision. Springer, 55--64.

[27]

Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard. 2010. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express 18, 10 (2010), 10762--10774.

[28]

Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard. 2011. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering 58, 1 (2011), 7--11.

[29]

Javier Portilla and Eero P. Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision 40, 1 (2000), 49--71.

Digital Library

[30]

E. P. Simoncelli, W. T. Freeman, E. H. Adelson, and D. J. Heeger. 1992. Shiftable multiscale transforms. IEEE Transactions on Information Theory 38, 2 (March 1992), 587--607. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=119725.

Digital Library

[31]

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013).

[32]

Supasorn Suwajanakorn, Steven M. Seitz, and Ira Kemelmacher-Shlizerman. 2017. Synthesizing Obama: Learning lip sync from audio. ACM Transactions on Graphics (TOG) 36, 4 (2017), 95.

Digital Library

[33]

Chihiro Takano and Yuji Ohta. 2007. Heart rate measurement based on a time-lapse image. Medical Engineering 8 Physics 29, 8 (2007), 853--857.

[34]

K. S. Tan, R. Saatchi, H. Elphick, and D. Burke. 2010. Real-time vision based respiration monitoring system. In Proceedings of the 7th International Symposium on Communication Systems Networks and Digital Signal Processing (CSNDSP) (2010), 770--774.

[35]

L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. A. Clifton, and C. Pugh. 2014. Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiological Measurement 35, 5 (2014), 807--831.

[36]

Sergey Tulyakov, Xavier Alameda-Pineda, and Elisa Ricci. 2016. Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In Proceedings of the Computer Vision Pattern Recognition (2016), 2396--2404.

[37]

Wim Verkruysse, Lars O. Svaasand, and J. Stuart Nelson. 2008. Remote plethysmographic imaging using ambient light. Optics Express 16, 26 (2008), 21434--21445.

[38]

Neal Wadhwa, Michael Rubinstein, Frédo Durand, and William T. Freeman. 2013. Phase-based video motion processing. ACM Transactions on Graphics (TOG) 32, 4 (2013), 80.

Digital Library

[39]

Neal Wadhwa, Michael Rubinstein, Frédo Durand, and William T. Freeman. 2014. Riesz pyramids for fast phase-based video magnification. In Proceedings of the 2014 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--10.

[40]

Jue Wang, Steven M. Drucker, Maneesh Agrawala, and Michael F. Cohen. 2006. The cartoon animation filter. In ACM Transactions on Graphics (TOG), Vol. 25. ACM, 1169--1173.

[41]

Wenjin Wang, Albertus Den Brinker, Sander Stuijk, and Gerard De Haan. 2016. Algorithmic principles of remote-PPG. IEEE Transactions on Biomedical Engineering PP, 99 (2016), 1--12.

[42]

Wenjin Wang, Sander Stuijk, and Gerard de Haan. 2015. Exploiting spatial redundancy of image sensor for motion robust rPPG. IEEE Transactions on Biomedical Engineering 62, 2 (2015), 415--425.

[43]

Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John V Guttag, Frédo Durand, and William T Freeman. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics 31, 4 (2012), 65.

Digital Library

[44]

Shuchang Xu, Lingyun Sun, and Gustavo Kunde Rohde. 2014. Robust efficient estimation of heart rate pulse from video. Biomedical Optics Express 5, 4 (2014), 1124.

[45]

Yichao Zhang, Silvia L. Pintea, and Jan C. Van Gemert. 2017. Video acceleration magnification. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (2017), 502--510.

Cited By

Chen WYi ZLim LLim RZhang AQian ZHuang JHe JLiu B(2024)Deep learning and remote photoplethysmography powered advancements in contactless physiological measurementFrontiers in Bioengineering and Biotechnology10.3389/fbioe.2024.142010012Online publication date: 17-Jul-2024
https://doi.org/10.3389/fbioe.2024.1420100
Narayanswamy GLiu YYang YMa CLiu XMcDuff DPatel S(2024)BigSmall: Efficient Multi-Task Learning for Disparate Spatial and Temporal Physiological Measurements2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00773(7899-7909)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00773
McDuff D(2023)Camera Measurement of Physiological Vital SignsACM Computing Surveys10.1145/355851855:9(1-40)Online publication date: 16-Jan-2023
https://doi.org/10.1145/3558518
Show More Cited By

Index Terms

DeepMag: Source-Specific Change Magnification Using Gradient Ascent
1. Computer systems organization
  1. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics

Recommendations

Phase-based video motion processing

We introduce a technique to manipulate small movements in videos based on an analysis of motion in complex-valued image pyramids. Phase variations of the coefficients of a complex-valued steerable pyramid over time correspond to motion, and can be ...
Preprocessing Realistic Video for Contactless Heart Rate Monitoring Using Video Magnification
CRV '15: Proceedings of the 2015 12th Conference on Computer and Robot Vision

This research seeks to improve the outcomes of Eulerian Video Magnification in real life scenarios. We address the core requirement in Eulerian Magnification that the person in the video be completely still. The proposed system pre-processes the video ...
Optimizing User Interface Layouts via Gradient Descent
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Automating parts of the user interface (UI) design process has been a longstanding challenge. We present an automated technique for optimizing the layouts of mobile UIs. Our method uses gradient descent on a neural network model of task performance with ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 40, Issue 1

February 2021

139 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3420236

Editor:
Marc Alexa
TU Berlin, Germany

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2020

Accepted: 01 May 2020

Revised: 01 March 2020

Received: 01 July 2018

Published in TOG Volume 40, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
1,063
Total Downloads

Downloads (Last 12 months)252
Downloads (Last 6 weeks)37

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen WYi ZLim LLim RZhang AQian ZHuang JHe JLiu B(2024)Deep learning and remote photoplethysmography powered advancements in contactless physiological measurementFrontiers in Bioengineering and Biotechnology10.3389/fbioe.2024.142010012Online publication date: 17-Jul-2024
https://doi.org/10.3389/fbioe.2024.1420100
Narayanswamy GLiu YYang YMa CLiu XMcDuff DPatel S(2024)BigSmall: Efficient Multi-Task Learning for Disparate Spatial and Temporal Physiological Measurements2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00773(7899-7909)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00773
McDuff D(2023)Camera Measurement of Physiological Vital SignsACM Computing Surveys10.1145/355851855:9(1-40)Online publication date: 16-Jan-2023
https://doi.org/10.1145/3558518
Singh JMurala SKosuru G(2023)Lightweight Network For Video Motion Magnification2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00208(2040-2049)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00208
Singh JMurala SKosuru G(2023)Multi Domain Learning for Motion Magnification2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01337(13914-13923)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01337
GANAHA WKAMITANI TNISHIYAMA MIWAI Y(2022)Video Generation Using Motion Magnification for Visualization of High-Frequency Components in Body Sway身体動揺に含まれる高周波成分の可視化のための運動拡大を用いた映像生成Journal of the Japan Society for Precision Engineering10.2493/jjspe.88.92688:12(926-936)Online publication date: 5-Dec-2022
https://doi.org/10.2493/jjspe.88.926
Sun ZLi X(2022)Privacy-Phys: Facial Video-Based Physiological Modification for Privacy ProtectionIEEE Signal Processing Letters10.1109/LSP.2022.318596429(1507-1511)Online publication date: 2022
https://doi.org/10.1109/LSP.2022.3185964
Liu XJiang ZFromm JXu XPatel SMcDuff DGhassemi MNaumann TPierson E(2021)MetaPhysProceedings of the Conference on Health, Inference, and Learning10.1145/3450439.3451870(154-163)Online publication date: 8-Apr-2021
https://dl.acm.org/doi/10.1145/3450439.3451870
Liu HChen PZhao Z(2021)Towards A Robust Meta-Reinforcement Learning-Based Scheduling Framework for Time Critical Tasks in Cloud Environments2021 IEEE 14th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD53861.2021.00082(637-647)Online publication date: Sep-2021
https://doi.org/10.1109/CLOUD53861.2021.00082

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents