research-article

MusicalHeart: a hearty way of listening to music

Authors:

Shahriar Nirjon,

Robert F. Dickerson,

John A. Stankovic,

Feng ZhaoAuthors Info & Claims

SenSys '12: Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems

Pages 43 - 56

https://doi.org/10.1145/2426656.2426662

Published: 06 November 2012 Publication History

Abstract

MusicalHeart is a biofeedback-based, context-aware, automated music recommendation system for smartphones. We introduce a new wearable sensing platform, Septimu, which consists of a pair of sensor-equipped earphones that communicate to the smartphone via the audio jack. The Septimu platform enables the MusicalHeart application to continuously monitor the heart rate and activity level of the user while listening to music. The physiological information and contextual information are then sent to a remote server, which provides dynamic music suggestions to help the user maintain a target heart rate. We provide empirical evidence that the measured heart rate is 75% -- 85% correlated to the ground truth with an average error of 7.5 BPM. The accuracy of the person-specific, 3-class activity level detector is on average 96.8%, where these activity levels are separated based on their differing impacts on heart rate. We demonstrate the practicality of MusicalHeart by deploying it in two real world scenarios and show that MusicalHeart helps the user achieve a desired heart rate intensity with an average error of less than 12.2%, and its quality of recommendation improves over time.

References

[1]

Fingertip pulse oximeter. http://www.naturespiritproduct.com.

[2]

Handheld ecg monitor (md100b). http://www.choicemmed.com/.

[3]

Power meter. http://msoon.com/LabEquipment/PowerMonitor/.

[4]

F. J. Aherne, N. A. Thacker, and P. Rockett. The bhattacharyya metric as an absolute similarity measure for frequency coded data. Kybernetika, 34(4):363--368, 1998.

[5]

A. Andric and G. Haus. Automatic playlist generation based on tracking user's listening habits. Multimedia Tools and Applications, 29(2):127--151, June 2006.

Digital Library

[6]

R. Aylward and J. A. Paradiso. A compact, high-speed, wearable sensor network for biomotion capture and interactive media. In 6th International Conference on Information Processing in Sensor Networks 2007 (IPSN '07), pages 380--389, Cambridge, MA.

Digital Library

[7]

L. Bao and S. S. Intille. Activity recognition from user-annotated acceleration data. IEEE Pervasive Computing, 3001:1--17, 2004.

[8]

G. Bernatzky et. al. Stimulating music increases motor coordination in patients afflicted with morbus parkinson. Neuroscience Letters, 361(1--3):4--8, May 2004.

[9]

J. T. Biehl, P. D. Adamczyk, and B. P. Bailey. Djogger: a mobile dynamic music device. In ACM SIGCHI Conference on Human Factors in computing systems 2006 (CHI '06), Montreal, Canada.

Digital Library

[10]

P. Celka, C. Verjus, R. Vetter, P. Renevey, and V. Neuman. Motion resistant earphone located infrared based heart rate measurement device. Biomedical Engineering, Feb 2004.

[11]

T. Choudhury et. al. The mobile sensing platform: An embedded activity recognition system. IEEE Pervasive Computing, 7(2):32--41, April 2008.

Digital Library

[12]

R. de Oliveira and N. Oliver. Triplebeat: enhancing exercise performance with persuasion. In 10th International Conference on Human Computer Interaction with Mobile Devices and Services 2008 (MobileHCI '08), pages 255--264.

Digital Library

[13]

T. Denning, A. Andrew, R. Chaudhri, C. Hartung, J. Lester, G. Borriello, and G. Duncan. Balance: towards a usable pervasive wellness application with accurate activity inference. In 10th Workshop on Mobile Computing Systems and Applications 2009 (HotMobile '09).

Digital Library

[14]

J. Edworthy and H. Waring. The effects of music tempo and loudness level on treadmill exercise. Ergonomics, 49(15):1597--610, Dec 2006.

[15]

J. Escher and D. Evequoz. Music and heart rate variability. study of the effect of music on heart rate variability in healthy adolescents. Praxis, 88(21):951--2, May 1999.

[16]

A. Flexer, D. Schnitzer, M. Gasser, and G. Widmer. Playlist generation using start and end songs. In 9th International Conference on Music Information Retrieval 2008 (ISMIR '08), pages 173--178.

[17]

G. Friesen, T. Jannett, M. Jadallah, S. Yates, S. Quint, and H. Nagle. A comparison of the noise sensitivity of nine qrs detection algorithms. Biomedical Engineering, 37(1):85--98, Jan 1990.

[18]

A. Goldberger, L. Amaral, L. Glass, J. Hausdorff, P. Ivanov, R. Mark, J. Mietus, G. Moody, C.-K. Peng, and H. Stanley. Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic signals. Circulation, 101(23):215--220, Jun 2000.

[19]

L. Harmat, J. Takacs, and R. Bodizs. Music improves sleep quality in students. Advanced Nursing, 62:327--335, May 2008.

[20]

J. L. Hellerstein, Y. Diao, S. Parekh, and D. M. Tilbury. Feedback Control of Computing Systems. Wiley-IEEE Press, Aug 2004.

Digital Library

[21]

N. Jausovec, K. Jausovec, and I. Gerlic. The influence of mozart's music on brain activity in the process of learning. Clinical Neuro-physiology, 117(12):2703--14, Dec 2006.

[22]

C. I. Karageorghis, L. Jones, and D. C. Low. Relationship between exercise heart rate and music tempo preference. Research Quarterly for Exercise and Sport, 26:240--250, 2006.

[23]

Y.-S. Kuo, S. Verma, T. Schmid, and P. Dutta. Hijacking power and bandwidth from the mobile phone's audio interface. In 1st ACM Symposium on Computing for Development 2010 (DEV '10).

Digital Library

[24]

E. Labbe, N. Schmidt, J. Babin, and M. Pharr. Coping with stress: the effectiveness of different types of music. Applied Psychophysiology Biofeedback, 32(3--4):163--8, Dec 2007.

[25]

H. Lai and M. Good. Music improves sleep quality in older adults. Advanced Nursing, 49(3):234--44, Feb 2005.

[26]

O. Lartillot. Mirtoolbox user's manual. Finnish Centre of Excelence in Interdisciplinary Music Research, Dec 2011.

[27]

N. H. Liu, S. J. Hsieh, and C. F. Tsai. An intelligent music playlist generator based on the time parameter with artificial neural networks. Expert Systems with Applications, 37(4):2815--2825, April 2010.

Digital Library

[28]

K. Lorincz, B.-r. Chen, G. W. Challen, A. R. Chowdhury, S. Patel, P. Bonato, and M. Welsh. Mercury: a wearable sensor network platform for high-fidelity motion analysis. In 7th ACM Conference on Embedded Networked Sensor Systems 2009 (SenSys '09), pages 183--196, Berkeley, CA.

Digital Library

[29]

N. Oliver and F. Flores-Mangas. Mptrain: a mobile, music and physiology-based personal trainer. In 8th International Conference on Human Computer Interaction with Mobile Devices and Services 2006 (MobileHCI '06), pages 21--28, Helsinki, Finland.

Digital Library

[30]

N. Oliver and L. Kreger-Stickles. Papa: Physiology and purpose-aware automatic playlist generation. In 7th International Conference on Music Information Retrieval 2006 (ISMIR '06), pages 250--253.

[31]

J. Pan and W. Tompkins. A real-time qrs detection algorithm. Biomedical Engineering, 32(3):230--236, March 1985.

[32]

J. C. Platt, C. J. Burges, S. Swenson, C. Weare, and A. Zheng. Learning a gaussian process prior for automatically generating music playlists. In Advances in Neural Information Processing Systems 2001 (NIPS '01), pages 1425--1432.

[33]

M.-Z. Poh, K. Kim, A. Goessling, N. Swenson, and R. Picard. Heart-phones: Sensor earphones and mobile application for non-obtrusive health monitoring. In International Symposium on Wearable Computers 2009 (ISWC '09), pages 153--154.

Digital Library

[34]

N. Ravi, N. Dandekar, P. Mysore, and M. L. Littman. Activity recognition from accelerometer data. In 17th Conference on Innovative Applications of Artificial Intelligence 2005 (IAAI '05), pages 1541--1546, Pittsburgh, PA.

Digital Library

[35]

S. Simpson and C. Karageorghis. The effects of synchronous music on 400-m sprint performance. Sports Sciences, 24(10):1095--102, Oct. 2006.

[36]

A. Thiagarajan, J. Biagioni, T. Gerlich, and J. Eriksson. Cooperative transit tracking using smartphones. In 8th ACM Conference on Embedded Networked Sensor Systems 2010 (SenSys '10), pages 85--98, Zurich, Switzerland.

Digital Library

[37]

X. Zhu, Y.-Y. Shi, H.-G. Kim, and K.-W. Eom. An integrated music recommendation system. Consumer Electronics, 52(3):917--925, Aug. 2006.

Digital Library

Cited By

Spence ABangay S(2024)Domain-Agnostic Representation of Side-ChannelsEntropy10.3390/e2608068426:8(684)Online publication date: 13-Aug-2024
https://doi.org/10.3390/e26080684
Christofferson KBhalla SCafazzo JMariakakis AKostakos VKay JHoang T(2024)On the Production and Measurement of Cardiac Sounds in the Ear CanalCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680526(685-690)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3680526
Butkow KFerlini AKawsar FMascolo CMontanari AKostakos VKay JHoang T(2024)EarTune: Exploring the Physiology of Music ListeningCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680519(644-649)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3680519
Show More Cited By

Index Terms

MusicalHeart: a hearty way of listening to music
1. Computer systems organization
  1. Embedded and cyber-physical systems
  2. Real-time systems

Recommendations

Tempo and autonomic control of the heart
EVA '15: Proceedings of the Conference on Electronic Visualisation and the Arts

Music is a powerful stimulus and has been found to influence cardiovascular activity, including heart rate, blood pressure, respiration and heart rate variability. Previous research has used complex pieces of music which render it difficult to determine ...
The impossibility of control: real-time negotiations with the heart
EVA '15: Proceedings of the Conference on Electronic Visualisation and the Arts

We present a practice-based research project, which examines temporal relationships between physiological processes with human movement and mediated environments. Ongoing research will provide a foundation for the creation of performances in which real-...
Effect of CPAP therapy on daytime cardiovascular regulations in patients with obstructive sleep apnea

Obstructive sleep apnea (OSA) is a sleep disorder with a high prevalence that causes pathological changes in cardiovascular regulation during the night and also during daytime. We investigated whether the treatment of OSA at night by means of continuous ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SenSys '12: Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems

November 2012

404 pages

ISBN:9781450311694

DOI:10.1145/2426656

General Chair:
Rasit Eskicioglu
University of Manitoba
,
Program Chairs:
Andrew Campbell
Dartmouth College
,
Koen Langendoen
Delft University of Technology

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 November 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Division of Electrical, Communications and Cyber Systems

Conference

SenSys '12

Sponsor:

SenSys '12: The 10th ACM Conference on Embedded Network Sensor Systems

November 6 - 9, 2012

Ontario, Toronto, Canada

Acceptance Rates

Overall Acceptance Rate 174 of 867 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

82
Total Citations
View Citations
1,235
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)8

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Spence ABangay S(2024)Domain-Agnostic Representation of Side-ChannelsEntropy10.3390/e2608068426:8(684)Online publication date: 13-Aug-2024
https://doi.org/10.3390/e26080684
Christofferson KBhalla SCafazzo JMariakakis AKostakos VKay JHoang T(2024)On the Production and Measurement of Cardiac Sounds in the Ear CanalCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680526(685-690)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3680526
Butkow KFerlini AKawsar FMascolo CMontanari AKostakos VKay JHoang T(2024)EarTune: Exploring the Physiology of Music ListeningCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680519(644-649)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3680519
Gao MTong XChen JChen YXiao FHan JShu YLiu JTan RHe YChen J(2024)Eternity in a Second: Quick-pass Continuous Authentication Using Out-ear MicrophonesProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699366(675-688)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699366
Wu YChen YZhang JGong XBi H(2024)Ubi-AD: Towards Ubiquitous, Passive Alzheimer Detection using the SmartwatchACM Transactions on Sensor Networks10.1145/365617420:5(1-22)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3656174
Cao YLi FChen HLiu XWang Y(2024)Tongue-Jaw Movement Recognition Through Acoustic Sensing on SmartphonesIEEE Transactions on Mobile Computing10.1109/TMC.2022.322259423:1(879-894)Online publication date: Jan-2024
https://doi.org/10.1109/TMC.2022.3222594
Zou YWeng JLei HWang DLeung VWu K(2024)EarPrint: Earphone-Based Implicit User Authentication With Behavioral and Physiological AcousticsIEEE Internet of Things Journal10.1109/JIOT.2024.341762211:19(31128-31143)Online publication date: 1-Oct-2024
https://doi.org/10.1109/JIOT.2024.3417622
Zhao ZLi FXie YXie HZhang KZhang LWang Y(2024)HearBP: Hear Your Blood Pressure via In-ear Acoustic Sensing Based on Heart SoundsIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621249(991-1000)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621249
Butkow KDang TFerlini AMa DLiu YMascolo C(2024)An evaluation of heart rate monitoring with in-ear microphones under motionPervasive and Mobile Computing10.1016/j.pmcj.2024.101913(101913)Online publication date: Mar-2024
https://doi.org/10.1016/j.pmcj.2024.101913
Lee EMin CLee JYu JKang SEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)GrooveMeter: Enabling Music Engagement-aware Apps by Detecting Reactions to Daily Music Listening via Earable SensingProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611968(7728-7736)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611968
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents