research-article

A Natural User Interface for Gestural Expression and Emotional Elicitation to Access the Musical Intangible Cultural Heritage

Authors:

Christina Volioti,

Sotiris Manitsaris,

Stelios Hadjidimitriou,

Vasileios Charisis,

Leontios Hadjileontiadis,

Eleni Katsouli,

Fabien Moutarde,

Athanasios ManitsarisAuthors Info & Claims

Journal on Computing and Cultural Heritage (JOCCH), Volume 11, Issue 2

Article No.: 10, Pages 1 - 20

https://doi.org/10.1145/3127324

Published: 12 April 2018 Publication History

Abstract

This article describes a prototype natural user interface, named the Intangible Musical Instrument, which aims to facilitate access to knowledge of performers that constitutes musical Intangible Cultural Heritage using off-the-shelf motion capturing that is easily accessed by the public at large. This prototype is able to capture, model, and recognize musical gestures (upper body including fingers) as well as to sonify them. The emotional status of the performer affects the sound parameters at the synthesis level. Intangible Musical Instrument is able to support both learning and performing/composing by providing to the user not only intuitive gesture control but also a unique user experience. In addition, the first evaluation of the Intangible Musical Instrument is presented, in which all the functionalities of the system are assessed. Overall, the results with respect to this evaluation were very promising.

References

[1]

Reuben M. Baron and David A. Kenny. 1986. The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology 51, 6, 1173--1182.

[2]

Peter M. Bentler. 1990. Comparative fit indexes in structural models. Psychological Bulletin 107, 238--246.

[3]

Peter M. Bentler and Chih-Ping Chou. 1987. Practical issues in structural modeling. Sociological Methods 8 Research 16, 1, 78--117.

[4]

Frédéric Bevilacqua, Fabrice Guédy, Nobert Schnell, Emmanuel Fléty, and Nicolas Leroy. 2007. Wireless sensor interface and gesture-follower for music pedagogy. In Proceedings of the International Conference of New Interfaces for Musical Expression. 124--129.

Digital Library

[5]

Frédéric Bevilacqua, Bruno Zamborlin, Anthony Sypniewski, Norbert Schnell, Fabrice Guédy, and Nicolas Rasamimanana. 2009. Continuous realtime gesture following and recognition. In Proceedings of the 8th International Conference on Gesture in Embodied Communication and Human-Computer Interaction (GW'09), Stefan Kopp and Ipke Wachsmuth (Eds.). Springer-Verlag, Berlin, Heidelberg, 73--84.

Digital Library

[6]

Frédéric Bevilacqua, Norbert Schnell, Nicolas Rasamimanana, Bruno Zamborlin, and Fabrice Guédy. 2011. Online gesture analysis and control of audio processing. In Musical Robots and Interactive Multimodal Systems (Springer Tracts in Advanced Robotics, 74), J. Solis and K. C. Ng (Eds.). Berlin: Springer, 127--142.

[7]

Jan L. Broeckx. 1981. Muziek, ratio en affect over de wisselwerking van rationeel denken en affectief beleven bij voortbrengst en ontvangst van muziek. Antwerpen: Metropolis.

[8]

Bernd Buxbaum. 2002. Optische laufzeitmessung und CDMA auf Basis der PMD-technologie mittels phasenvariabler PN-modulation. Schaker Verlag, Aachen.

[9]

Claude Cadoz and Marcelo M. Wanderley. 2000. Gesture-music. Trends in gestural control of music. In Trends in Gestural Control of Music, M. M. Wanderley and M. Battier (Eds.). Paris: IRCAM/Centre Pompidou, 71--94.

[10]

Antonio Camurri, Gualtiero Volpe, Giovanni de Poli, and Marc Leman. 2005. Communicating expressiveness and affect in multimodal interactive systems. IEEE MultiMedia 12, 1, 43--53.

Digital Library

[11]

Martin R. L. Clayton. 2000. Time in Indian Music: Rhythm Metre and Form in Indian Rag Performance. Oxford: Oxford University Press.

[12]

Wilson Coker. 1972. Music 8 Meaning: A Theoretical Introduction to Musical Aesthetics. New York: Free Press.

[13]

Naomi Cumming. 2000. The Sonic Self: Musical Subjectivity and Signification. Bloomington: Indiana University Press.

[14]

Francois Delalande. 1988. La gestique de Gould: éléments pour une sémiologie du geste musical. In Glenn Gould pluriel, G. Guertin (Ed.). Montreal: Louise Courteau éditrice.

[15]

John Delery and Harold D. Doty. 1996. Modes of theorizing in strategic human resource management: Test of universalistic, contingency and configurational performance predictions. Academy of Management Journal 39, 802--835.

[16]

Andy P. Field. 2005. Discovering Statistics Using SPSS (2nd ed.). London: Sage.

Digital Library

[17]

Anders Friberg and Johan Sundberg. 1999. Does music performance allude to locomotion? A model of final ritardandi derived from measurements of stopping runners. Journal of the Acoustical Society of America 105, 3, 146--148.

[18]

Aalf Gabrielsson and Erik Lindström. 2010. The role of structure in the musical expression of emotions. In Handbook of Music and Emotions Theory, Research, Applications, P. N. Juslin and J. A. Sloboda (Eds.). New York: Oxford University Press, 367--400.

[19]

Leontios J. Hadjileontiadis. 2014. Conceptual blending in biomusic composition space: The “brainswarm” paradigm. In Proceedings of the ICM/SMC Conference.

[20]

Robert S. Hatten. 1994. Musical Meaning in Beethoven: Markedness, Correlation, and Interpretation. Bloomington: Indiana University Press.

[21]

Rolf Inge Godøy and Mark Leman. 2009. Musical Gestures: Sound, Movement, and Meaning, Rolf Inge Godoy and Marc Leman (Eds.). New York: Routledge.

[22]

Alexander R. Jensenius. 2007. ACTION -- SOUND Developing Methods and Tools to Study. Ph.D. Dissertation, University of Oslo.

[23]

Patrik N. Juslin. 2003. Five facets of musical expression: A psychologist's perspective on music performance. Psychology of Music 31, 1, 273--302.

[24]

Patrik N. Juslin and John Sloboda. 2010. Handbook of Music and Emotions Theory, Research, Applications. New York: Oxford University Press.

[25]

Patrik N. Juslin and Renee Timmers. 2010. Expression and communication of emotion in music performance. In Handbook of Music and Emotions Theory, Research, Applications, P. N. Juslin and J. A. Sloboda (Eds.). New York: Oxford University Press, 453--489.

[26]

Henry F. Kaiser. 1974. An index of factorial simplicity. Psychometrika 39, 31--36.

[27]

Anastasia A. Katou, Pawan S. Budhwar, and Charmi Patel. 2014. Content vs. process in the HRM-performance relationship: An empirical examination. Human Resource Management 53, 4, 527--544.

[28]

Peter Keller. 2008. Joint action in music performance. In Enacting Intersubjectivity: A Cognitive and Social Perspective on the Study of Interaction, F. Morganti, A. Carassa, and G. Riva (Eds.). Amsterdam, 205--221.

[29]

Rex B. Kline. 1998. Principles and Practice of Structural Equation Modeling. New York: Guilford Press.

[30]

Stefan Koelsch and Walter A. Siebel. 2005. Towards a neural basis of music perception. Trends in Cognitive Sciences 9, 12, 578--584.

[31]

Marc Leman. 2010. Music, gesture, and the formation of embodied meaning. In Musical Gestures: Sound, Movement, and Meaning, Rolf Inge Godoy and Marc Leman (Eds.). New York: Routledge, 126--153.

[32]

Pieter-Jan Maes, Marc Leman, Micheline Lesaffre, Michiel Demey, and Dirk Moelants. 2010. From expressive gesture to sound. The development of an embodied mapping trajectory inside a musical interface. Journal on Multimodal User Interfaces 3, 1, 67--78.

[33]

David Mcneill. 1992. Hand and Mind: What Gestures Reveal About Thought. Chicago: University of Chicago Press.

[34]

Alva Noë. 2004. Action in Perception. Cambridge, MA: MIT Press.

[35]

Elazar J. Pedhazur and Liora Pedhazur-Schmelkin. 1991. Measurement, Design, and Analysis: An Integrated Approach. Hillsdale, NJ: Lawrence Erlbaum.

[36]

Carroll C. Pratt. 1931/1968. The Meaning of Music: A Study In Psychological Aesthetics. New York: Johnson.

[37]

James A. Russell. 1980. A circumflex model of affect. Journal of Personality and Social Psychology 39, 1161--1178.

[38]

Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. 2013. Real-time human pose recognition in parts from a single depth image. Machine Learning for Computer Vision, SCI, 411, 119--135.

[39]

Alexander Truslit. 1938. Gestaltung und Bewegung in der Musik. Berlin-Lichterfelde: C.F. Vieweg.

[40]

N. Smirnov. 1948. Table for estimating the goodness of fit of empirical distributions. Annals of Mathematical Statistics 19, 279--281.

[41]

UNESCO. 2003. Convention of the Safeguarding of Intangible Cultural Heritage of UNESCO. Retrieved from https://ich.unesco.org/en/convention.

[42]

Frank Weichert, Daniel Bachmann, Bartholomäus Rudak, and Denis Fisseler. 2013. Analysis of the accuracy and robustness of the leap motion controller. Sensors 13, 6380--6393.

[43]

Matthias Wiedemann, Markus Sauer, Frauke Driewer, and Klaus Schilling. 2008. Analysis and characterization of the PMD camera for application in mobile robotics. In Proceedings of the 17th IFAC World Congress, 6--11.

[44]

Liwei Zhao. 2001. Synthesis and acquisition of laban movement analysis qualitative parameters for communicative gestures. Ph.D. Dissertation. University of Pennsylvania, Philadelphia. AAI3015399.

Digital Library

Cited By

Li J(2023)Research on digital display of intangible cultural heritageJournal of Education, Humanities and Social Sciences10.54097/ehss.v8i.42248(54-61)Online publication date: 7-Feb-2023
https://doi.org/10.54097/ehss.v8i.4224
Volioti CTsagaris ATrigkas DIliou TKatsantonis MMavridis I(2023)HapticSOUND: An Interactive Learning Experience with a Digital Musical InstrumentApplied Sciences10.3390/app1312714913:12(7149)Online publication date: 14-Jun-2023
https://doi.org/10.3390/app13127149
Hirsch LLi JMayer SButz A(2022)A Survey of Natural Design for InteractionProceedings of Mensch und Computer 202210.1145/3543758.3543773(240-254)Online publication date: 4-Sep-2022
https://dl.acm.org/doi/10.1145/3543758.3543773
Show More Cited By

Index Terms

A Natural User Interface for Gestural Expression and Emotional Elicitation to Access the Musical Intangible Cultural Heritage
1. Computing methodologies
  1. Machine learning
2. Information systems

Recommendations

Digital Methods in Intangible Cultural Heritage Research: A Case Study in Tango Argentino

With this article, we present the ongoing research project “Tango Danceability of Music in European Perspective” and the transdisciplinary research design it is built upon. Three main aspects of tango argentino are in focus—the music, the dance, and the ...
On mapping emotional states and implicit gestures to sonification output from the 'Intangible Musical Instrument'
MOCO '16: Proceedings of the 3rd International Symposium on Movement and Computing

Sonification is an interdisciplinary field of research, aiming at generating sound from data based on systematic, objective and reproducible transformations. Towards this direction, expressive gestures play an important role in music performances ...
Intangible cultural heritage preservation: An exploratory study of digitization of the historical literature of Chinese Kunqu opera librettos

This article introduces the study contents and some research findings regarding digital preservation methods for Chinese Kunqu opera libretto historical literature, including historical literature electronic libretto transformation, libretto musical ...

Comments

Information & Contributors

Information

Published In

cover image Journal on Computing and Cultural Heritage

Journal on Computing and Cultural Heritage Volume 11, Issue 2

June 2018

124 pages

ISSN:1556-4673

EISSN:1556-4711

DOI:10.1145/3199679

Editor:
Roberto Scopigno
CNR-ISTI, Italy

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 April 2018

Accepted: 01 June 2017

Revised: 01 June 2017

Received: 01 January 2017

Published in JOCCH Volume 11, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Widget Corporation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
386
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li J(2023)Research on digital display of intangible cultural heritageJournal of Education, Humanities and Social Sciences10.54097/ehss.v8i.42248(54-61)Online publication date: 7-Feb-2023
https://doi.org/10.54097/ehss.v8i.4224
Volioti CTsagaris ATrigkas DIliou TKatsantonis MMavridis I(2023)HapticSOUND: An Interactive Learning Experience with a Digital Musical InstrumentApplied Sciences10.3390/app1312714913:12(7149)Online publication date: 14-Jun-2023
https://doi.org/10.3390/app13127149
Hirsch LLi JMayer SButz A(2022)A Survey of Natural Design for InteractionProceedings of Mensch und Computer 202210.1145/3543758.3543773(240-254)Online publication date: 4-Sep-2022
https://dl.acm.org/doi/10.1145/3543758.3543773
Sluÿters AOusmer MRoselli PVanderdonckt J(2022)QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion ControllerProceedings of the ACM on Human-Computer Interaction10.1145/35322116:EICS(1-47)Online publication date: 17-Jun-2022
https://dl.acm.org/doi/10.1145/3532211
Zhang JBryan-Kinns N(2022)QiaoLe: Accessing Traditional Chinese Musical Instruments in VR2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW55335.2022.00080(357-362)Online publication date: Mar-2022
https://doi.org/10.1109/VRW55335.2022.00080
Isiaka FAdamu Z(2022)Custom emoji based emotion recognition system for dynamic business webpagesInternational Journal of Intelligent Computing and Cybernetics10.1108/IJICC-11-2021-025415:4(497-509)Online publication date: 10-Jan-2022
https://doi.org/10.1108/IJICC-11-2021-0254
Yu MZhang MYu CMa XYang XZhang JKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)We Can Do More to Save Guqin: Design and Evaluate Interactive Systems to Make Guqin More Accessible to the General PublicProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445175(1-12)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445175

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents