skip to main content
10.1145/3636534.3649366acmconferencesArticle/Chapter ViewAbstractPublication PagesmobicomConference Proceedingsconference-collections
research-article
Open access

Exploring the Feasibility of Remote Cardiac Auscultation Using Earphones

Published: 29 May 2024 Publication History

Abstract

The elderly over 65 accounts for 80% of COVID deaths in the United States. In response to the pandemic, the federal, state governments, and commercial insurers are promoting video visits, through which the elderly can access specialists at home over the Internet, without the risk of COVID exposure. However, the current video visit practice barely relies on video observation and talking. The specialist could not assess the patient's health conditions by performing auscultations.
This paper tries to address this key missing component in video visits by proposing Asclepius, a hardware-software solution that turns the patient's earphones into a stethoscope, allowing the specialist to hear the patient's fine-grained heart sound (i.e., PCG signals) in video visits. To achieve this goal, we contribute a low-cost plug-in peripheral that repurposes the earphone's speaker into a microphone and uses it to capture the patient's minute PCG signals from her ear canal. As the PCG signals suffer from strong attenuation and multi-path effects when propagating from the heart to ear canals, we then propose efficient signal processing algorithms coupled with a data-driven approach to de-reverberate and further correct the amplitude and frequency distortion in raw PCG receptions. We implement Asclepius on a 2-layer PCB board and follow the IRB protocol to evaluate its performance with 30 volunteers. Our extensive experiments show that Asclepius can effectively recover Phonocardiogram (PCG) signals with different types of earphones. The objective blind testing and subjective interview with five cardiologists further confirm the clinical efficacy and efficiency of our system. PCG signal samples, benchmark results, and cardiologist interviews can be found at: https://asclepius-system.github.io/

References

[1]
C. F. Anderson. Clinical auscultation of the cardiovascular system. Mayo Clinic Proceedings, 1990.
[2]
E. K. Antonsson, R. W. Mann. The frequency content of gait. Journal of biomechanics, 1985.
[3]
A. N. Balaji, A. Ferlini, F. Kawsar, A. Montanari. Stereo-bp: Non-invasive blood pressure sensing with earables. Proceedings of the 24th International Workshop on Mobile Computing Systems and Applications, 96--102, 2023.
[4]
Bradycardia. https://en.wikipedia.org/wiki/Bradycardia.
[5]
N. Bui, N. Pham, J. J. Barnitz, Z. Zou, P. Nguyen, H. Truong, T. Kim, N. Farrow, A. Nguyen, J. Xiao, et al. ebp: A wearable system for frequent and comfortable blood pressure monitoring from user's ear. The 25th annual international conference on mobile computing and networking, 2019.
[6]
K.-J. Butkow, T. Dang, A. Ferlini, D. Ma, C. Mascolo. Motion-resilient heart rate monitoring with in-ear microphones. arXiv preprint arXiv:2108.09393, 2021.
[7]
N. E. Bylund, M. Ressner, H. Knutsson. 3d wiener filtering to reduce reverberations in ultrasound image sequences. Scandinavian Conference on Image Analysis. Springer, 2003.
[8]
Y. Cao, C. Cai, A. Yu, F. Li, J. Luo. Earace: Empowering versatile acoustic sensing via earable active noise cancellation platform. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2023.
[9]
Cardiac cycle. https://en.wikipedia.org/wiki/Cardiac_cycle.
[10]
J. Chan, N. Ali, A. Najafi, A. Meehan, L. R. Mancl, E. Gallagher, R. Bly, S. Gollakota. An off-the-shelf otoacoustic-emission probe for hearing screening via a smartphone. Nature Biomedical Engineering, 2022.
[11]
J. Chan, A. Glenn, M. Itani, L. R. Mancl, E. Gallagher, R. Bly, S. Patel, S. Gollakota. Wireless earbuds for low-cost hearing screening. arXiv preprint arXiv:2212.05435, 2022.
[12]
I. Chatterjee, M. Kim, V. Jayaram, S. Gollakota, I. Kemelmacher, S. Patel, S. M. Seitz. Clearbuds: wireless binaural earbuds for learning-based speech enhancement. Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, 2022.
[13]
T. Chen, X. Fan, Y. Yang, L. Shangguan. Towards remote auscultation with commodity earphones. Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, 2022.
[14]
T. Chen, L. Shangguan, Z. Li, K. Jamieson. Metamorph: Injecting inaudible commands into over-the-air voice controlled systems. Network and Distributed Systems Security (NDSS) Symposium, 2020.
[15]
T. Chen. The design and implementation of a steganographic communication system over in-band acoustical channels. ACM Transactions on Sensor Networks, 2023.
[16]
Z. Chen, T. Zheng, C. Cai, J. Luo. Movi-fi: Motion-robust vital signs waveform recovery via deep interpreted rf sensing. Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021.
[17]
F. A. Choudhry, J. T. Grantham, A. T. Rai, J. P. Hogg. Vascular geometry of the extracranial carotid arteries: an analysis of length, diameter, and tortuosity. Journal of neurointerventional surgery, 2016.
[18]
R. R. Choudhury. Earable computing: A new area to think about. Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications, 147--153, 2021.
[19]
J. S. Dhillon, C. Ramos, B. C. Wünsche, C. Lutteroth. Designing a web-based telehealth system for elderly people: An interview study in new zealand. 2011 24th International Symposium on Computer-Based Medical Systems (CBMS), 1--6. IEEE, 2011.
[20]
A doctor's touch. https://www.ted.com/talks/abraham_verghese_a_doctor_s_touch.
[21]
M. Doerbecker, S. Ernst. Combination of two-channel spectral subtraction and adaptive wiener post-filtering for noise reduction and dereverberation. 1996 8th European Signal Processing Conference (EUSIPCO 1996). IEEE, 1996.
[22]
J. Eargle. The Microphone Book: From mono to stereo to surround-a guide to microphone design and application. Routledge, 2012.
[23]
Headphone impedance demystified. https://www.headphonesty.com/2019/04/headphone-impedance-demystified/.
[24]
Equivalent series resistance. https://en.wikipedia.org/wiki/Equivalent_series_resistance.
[25]
FAITH AND THE STETHOSCOPE. Website.
[26]
X. Fan, D. Pearl, R. Howard, L. Shangguan, T. Thormundsson. Apg: Audioplethysmography for cardiac monitoring in hearables. Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, 2023.
[27]
X. Fan, L. Shangguan, S. Rupavatharam, Y. Zhang, J. Xiong, Y. Ma, R. Howard. Headfi: bringing intelligence to all headphones. Proceedings of MobiCom, 2021.
[28]
A. Ferlini, D. Ma, R. Harle, C. Mascolo. Eargate: gait-based user identification with in-ear microphones. Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021.
[29]
A. Ferlini, A. Montanari, C. Min, H. Li, U. Sassi, F. Kawsar. In-ear ppg for vital signs. IEEE Pervasive Computing, 2021.
[30]
S. N. Gajarawala, J. N. Pelkowski. Telehealth benefits and barriers. The Journal for Nurse Practitioners, 17(2), 218--221, 2021.
[31]
S. K. Ghosh, R. K. Tripathy, R. Ponnalagu. Evaluation of performance metrics and denoising of pcg signal using wavelet based decomposition. IEEE 17th India Council International Conference, 2020.
[32]
D. Gill, N. Gavrieli, N. Intrator. Detection and identification of heart sounds using homomorphic envelogram and self-organizing probabilistic model. Computers in Cardiology, 2005. IEEE, 2005.
[33]
A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, H. E. Stanley. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 2000.
[34]
G. Grimmett, D. Stirzaker. Probability and random processes. Oxford university press, 2020.
[35]
Group delay and phase delay. Website.
[36]
U. Ha, S. Assana, F. Adib. Contactless seismocardiography via deep learning radars. Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, 2020.
[37]
Y. Jin, Y. Gao, X. Guo, J. Wen, Z. Li, Z. Jin. Earhealth: an earphone-based acoustic otoscope for detection of multiple ear diseases in daily life. Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, 2022.
[38]
S.-H. Kang, B. Joe, Y. Yoon, G.-Y. Cho, I. Shin, J.-W. Suh, et al. Cardiac auscultation using smartphones: pilot study. JMIR mHealth and uHealth.
[39]
E. Kaniusas. Acoustical signals of biomechanical systems. World Scientific, 2007.
[40]
R. Khusainov, D. Azzi, I. E. Achumba, S. D. Bersch. Real-time human ambulation, activity, and physiological monitoring: Taxonomy of issues, techniques, applications, challenges and limitations. Sensors, 2013.
[41]
K. Kondo, Y. Takahashi, T. Komatsu, T. Nishino, K. Takeda. Computationally efficient single channel dereverberation based on complementary wiener filter. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013.
[42]
N. Koonjoo, B. Zhu, G. C. Bagnall, D. Bhutto, M. Rosen. Boosting the signal-to-noise of low-field mri with deep learning image reconstruction. Scientific reports, 11(1), 1--16, 2021.
[43]
F. Kreuk, Y. Adi, B. Raj, R. Singh, J. Keshet. Hide and speak: Towards deep neural networks for speech steganography. arXiv preprint arXiv:1902.03083, 2019.
[44]
F.-J. Kung, M. R. Bai. Estimation of the noise and reverberation covariance matrices with application in speech enhancement using the multichannel wiener filter. INTER-NOISE and NOISE-CON Congress and Conference Proceedings. Institute of Noise Control Engineering, 2020.
[45]
A. Leatham. Auscultation of the Heart and Phonocardiography. Churchill London, 1970.
[46]
S. Leng, R. S. Tan, K. T. C. Chai, C. Wang, D. Ghista, L. Zhong. The electronic stethoscope. Biomedical engineering online, 2015.
[47]
M. Lewkowicz, M. Gitterman. Theory of heart sounds. Journal of sound and vibration, 117(2), 263--275, 1987.
[48]
A. A. Luisada, D. M. MacCanon. The phases of the cardiac cycle. American heart journal, 83(5), 705--711, 1972.
[49]
D. Ma, A. Ferlini, C. Mascolo. Oesense: employing occlusion effect for in-ear human sensing. Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, 2021.
[50]
N. Mamorita, N. Arisaka, R. Isonaka, T. Kawakami, A. Takeuchi. Development of a smartphone app for visualizing heart sounds and murmurs. Cardiology, 2017.
[51]
M. K. Mandal, S. Sanyal. Compact wideband bandpass filter. IEEE microwave and wireless components letters, 2005.
[52]
Max5402 datasheet. https://pdfserv.maximintegrated.com/en/ds/MAX5402.pdf.
[53]
S. McGee. Evidence-based physical diagnosis. Elsevier Health Sciences, 2021.
[54]
H. Møller, D. Hammershøi, C. B. Jensen, M. F. Sørensen. Transfer characteristics of headphones measured on human ears. Journal of the Audio Engineering Society, 1995.
[55]
A. Mousavi, A. B. Patel, R. G. Baraniuk. A deep learning approach to structured signal recovery. 2015 53rd annual allerton conference on communication, control, and computing (Allerton), 1336--1343. IEEE, 2015.
[56]
P. A. Ongley. Heart sounds and murmurs: A clinical and phonocardiographic study. Grune & Stratton, 1960.
[57]
C. J. Owen, J. P. Wyllie. Determination of heart rate in the baby at birth. Resuscitation, 2004.
[58]
S. Pascual, A. Bonafonte, J. Serra. Segan: Speech enhancement generative adversarial network. arXiv preprint arXiv:1703.09452, 2017.
[59]
Patient experience: The clinician connection with patients, matters. https://www.littmann.com/3M/en_US/littmann-stethoscopes/advantages/promotions/clinician-patient-connection/.
[60]
A. N. Pelech. The physiology of cardiac auscultation. Pediatric Clinics, 51(6), 1515--1535, 2004.
[61]
N. Pham, T. Dinh, Z. Raghebi, T. Kim, N. Bui, P. Nguyen, H. Truong, F. Banaei-Kashani, A. Halbower, T. Dinh, et al. Wake: a behind-the-ear wearable system for microsleep detection. Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, 2020.
[62]
Report shows overwhelming patient interest in post-pandemic virtual care. Website.
[63]
J. Prakash, Z. Yang, Y.-L. Wei, H. Hassanieh, R. R. Choudhury. Earsense: earphones as a teeth activity sensor. Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, 2020.
[64]
C. M. van Ravenswaaij-Arts, L. A. Kollee, J. C. Hopman, G. B. Stoelinga, H. P. van Geijn. Heart rate variability. Annals of internal medicine, 1993.
[65]
Reflection coefficient. https://en.wikipedia.org/wiki/Reflection_coefficient.
[66]
T. Röddiger, C. Clarke, P. Breitling, T. Schneegans, H. Zhao, H. Gellersen, M. Beigl. Sensing with earables: A systematic literature review and taxonomy of phenomena. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022.
[67]
O. Ronneberger, P. Fischer, T. Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. Springer, 2015.
[68]
Safety sound level recommended by epa and who. https://www.cdc.gov/nceh/hearing_loss/what_noises_cause_hearing_loss.html.
[69]
S. E. Schmidt, C. Holst-Hansen, C. Graff, E. Toft, J. J. Struijk. Segmentation of heart sound recordings by a duration-dependent hidden markov model. Physiological measurement, 2010.
[70]
V. Schoebel, C. Wayment, M. Gaiser, C. Page, J. Buche, A. J. Beck. Telebehavioral health during the covid-19 pandemic: a qualitative analysis of provider experiences and perspectives. Telemedicine and e-Health, 27(8), 947--954, 2021.
[71]
Z. Shi, T. Gu, Y. Zhang, X. Zhang. mmbp: Contact-free millimetre-wave radar based approach to blood pressure measurement. Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, 2022.
[72]
N. Silverman, N. Schiller. Apex echocardiography. a two-dimensional technique for evaluating congenital heart disease. Circulation, 1978.
[73]
X. Song, K. Huang, W. Gao. Facelistener: Recognizing human facial expressions via acoustic sensing on commodity headphones. Proceedings of IEEE IPSN, 2022.
[74]
Sound cards impedance. https://audioxpress.com/article/practical-test-measurement-sound-cards-for-data-acquisition-in-audio-measurements-part-4.
[75]
Can a Speaker be Converted Into an Audio Microphone? Website.
[76]
D. B. Springer, L. Tarassenko, G. D. Clifford. Logistic regression-hsmm-based heart sound segmentation. IEEE transactions on biomedical engineering, 2015.
[77]
Stethoscope. Website.
[78]
T. O. D. Stethoscope. https://store.thinklabs.com/products/thinklabs-one-digital-stethoscope.
[79]
A. Strazza, A. Sbrollini, M. Olivastrelli, A. Piersanti, S. Tomassini, I. Marcantoni, M. Morettini, S. Fioretti, L. Burattini. Pcg-decompositor: A new method for fetal phonocardiogram filtering based on wavelet transform multi-level decomposition. Mediterranean Conference on Medical and Biological Engineering and Computing, 2020.
[80]
K. Sun, X. Zhang. Ultrase: single-channel speech enhancement using ultrasound. Proceedings of the 27th annual international conference on mobile computing and networking, 160--173, 2021.
[81]
X. Sun, J. Xiong, C. Feng, W. Deng, X. Wei, D. Fang, X. Chen. Earmonitor: In-ear motion-resilient acoustic sensing using commodity earphones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2023.
[82]
S. Swarup, A. N. Makaryus. Digital stethoscope: Technology update. Medical Devices: Evidence and Research, 2018.
[83]
Tachycardia. https://en.wikipedia.org/wiki/Tachycardia.
[84]
D. Ulyanov, A. Vedaldi, V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
[85]
Umhs michigan heart sound and murmur library. https://www.med.umich.edu/lrc/psb_open/html/repo/primer_heartsound/primer_heartsound.html.
[86]
Understanding impedance. https://www.soundonsound.com/techniques/understanding-impedance.
[87]
H. K. Walker, W. D. Hall, J. W. Hurst. Clinical methods: the history, physical, and laboratory examinations, 1990.
[88]
Y. Wang, D. Wang. A deep neural network for time-domain signal reconstruction. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4390--4394. IEEE, 2015.
[89]
H. Wu, K. H. K. Patel, X. Li, B. Zhang, C. Galazis, N. Bajaj, A. Sau, X. Shi, L. Sun, Y. Tao, et al. A fully-automated paper ecg digitisation algorithm using deep learning. Nature Scientific Reports, 2022.
[90]
L. Xiang, Y. Chen, W. Chang, Y. Zhan, W. Lin, Q. Wang, D. Shen. Deep-learning-based multi-modal fusion for fast mr reconstruction. IEEE Transactions on Biomedical Engineering, 2018.
[91]
Z. Xiao, T. Chen, Y. Liu, Z. Li. Mobile phones know your keystrokes through the sounds from finger's tapping on the screen. 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), 2020.
[92]
Z. Yang, R. R. Choudhury. Personalizing head related transfer functions for earables. Proceedings of the 2021 ACM SIGCOMM 2021 Conference, 2021.
[93]
Z. Yang, Y.-L. Wei, S. Shen, R. R. Choudhury. Ear-ar: indoor acoustic augmented reality on earphones. Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, 2020.

Cited By

View all
  • (2024)EchoGuide: Active Acoustic Guidance for LLM-Based Eating Event Analysis from Egocentric VideosProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676611(40-47)Online publication date: 5-Oct-2024
  • (2024)On the Production and Measurement of Cardiac Sounds in the Ear CanalCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680526(685-690)Online publication date: 5-Oct-2024
  • (2024)Enabling Hands-Free Voice Assistant Activation on EarphonesProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661890(155-168)Online publication date: 3-Jun-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ACM MobiCom '24: Proceedings of the 30th Annual International Conference on Mobile Computing and Networking
December 2024
2476 pages
ISBN:9798400704895
DOI:10.1145/3636534
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 May 2024

Check for updates

Author Tags

  1. remote health
  2. cardiac monitoring
  3. earable computing

Qualifiers

  • Research-article

Funding Sources

Conference

ACM MobiCom '24
Sponsor:

Acceptance Rates

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1,349
  • Downloads (Last 6 weeks)445
Reflects downloads up to 13 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)EchoGuide: Active Acoustic Guidance for LLM-Based Eating Event Analysis from Egocentric VideosProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676611(40-47)Online publication date: 5-Oct-2024
  • (2024)On the Production and Measurement of Cardiac Sounds in the Ear CanalCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680526(685-690)Online publication date: 5-Oct-2024
  • (2024)Enabling Hands-Free Voice Assistant Activation on EarphonesProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661890(155-168)Online publication date: 3-Jun-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media