Piano Learning and Improvisation through Adaptive Visualisation and Digital Augmentation
Pages 41 - 45
Abstract
The task of learning the piano has been a centuries-old challenge for novices, experts and technologists. Several innovations have been introduced to support proper posture, movement, and motivation, while sight-reading and improvisation remain the least-explored areas. In this PhD, we address this gap by redesigning the piano augmentation as an interactive and adaptive space. Specifically, we will explore how to support learners with adaptive visualisations through a two-pronged approach: (1) by designing adaptive visualisations based on the proficiency of the learner to support regular piano playing and (2) by assisting them with expert annotations projected on the piano to encourage improvisation. To this end, we will build a model to understand the complexities of learners’ spatiotemporal data and use these to support learning. We will then evaluate our approach through user studies enabling practice and improvisation. Our work contributes to how adaptive visualisations can push music instrument learning and support multi-target selection tasks in immersive spaces.
References
[1]
István Barakonyi and Dieter Schmalstieg. 2005. Augmented Reality Agents in the Development Pipeline of Computer Entertainment. In International Conference on Entertainment Computing, Fumio Kishino, Yoshifumi Kitamura, Hirokazu Kato, and Noriko Nagata (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 345–356. https://doi.org/10.1007/11558651_34
[2]
Jonathan Chow, Haoyang Feng, Robert Amor, and Burkhard C. Wünsche. 2013. Music Education Using Augmented Reality with a Head Mounted Display. In Proceedings of the Fourteenth Australasian User Interface Conference - Volume 139 (Melbourne, Australia) (AUIC ’13). Australian Computer Society, Inc., AUS, 73–79.
[3]
Yawen E. Chyu. 2004. Teaching improvisation to piano students of elementary to intermediate levels. Ph.D. Dissertation. Ohio State University.
[4]
Peter Cope and Hugh Smith. 1997. Cultural context in musical instrument learning. British Journal of Music Education 14, 3 (1997), 283–289.
[5]
Andrea Creech. 2010. Learning a musical instrument: The case for parental support. Music Education Research 12, 1 (2010), 13–32.
[6]
Ryan Daniel. 2006. Exploring music instrument teaching and learning environments: Video analysis as a means of elucidating process and learning outcomes. Music Education Research 8, 2 (2006), 191–215.
[7]
Jordan Aiko Deja. 2021. Adaptive Visualisations Using Spatiotemporal and Heuristic Models to Support Piano Learning. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (Utrecht, Netherlands) (UMAP ’21). Association for Computing Machinery, New York, NY, USA, 286–290. https://doi.org/10.1145/3450613.3459656
[8]
Jordan Aiko Deja. 2021. Encouraging Improvisation in Piano Learning Using Adaptive Visualisations and Spatiotemporal Models. In Adjunct Publication of the 23rd International Conference on Mobile Human-Computer Interaction. ACM Digital Library, Toulousse, France, 1–4. https://doi.org/10.1145/3447527.3474865
[9]
Jordan Aiko Deja, Kevin Gray Chan, Migo Andres Dancel, Allen Vincent Gonzales, and John Patrick Tobias. 2018. Flow: A Musical Composition Tool Using Gesture Interactions and Musical Metacreation. In HCI International 2018 – Posters’ Extended Abstracts, Constantine Stephanidis (Ed.). Springer International Publishing, Cham, 169–176.
[10]
Jordan Aiko Deja, Sven Mayer, Klen Čopič Pucihar, and Matjaž Kljun. 2022. A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences?Proceedings of the ACM Human Computer Interaction Volume ISS (2022), 1–28. https://doi.org/10.1145/3567719
[11]
Jordan Aiko Deja, Sven Mayer, Klen Čopič Pucihar, and Matjaž Kljun. 2022. The Vision of a Human-Centered Piano. In Intelligent Music Interfaces 2022 Workshop. arXiv, New Orleans, USA, 1–4. https://doi.org/10.48550/ARXIV.2204.06945
[12]
Dominique Fober, Stéphane Letz, and Yann Orlarey. 2007. Vemus-feedback and groupware technologies for music instrument learning. In 4th Sound and Music Computing Conference. HAL Open Science, Lefkada, Greece, 117–123.
[13]
Alix Goguey, Carl Gutwin, Zhe Chen, Pang Suwanaposee, and Andy Cockburn. 2021. Interaction Pace and User Preferences. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 195, 14 pages. https://doi.org/10.1145/3411764.3445772
[14]
Adam Goodwin and Richard Green. 2013. Key detection for a virtual piano teacher. In 28th International Conference on Image and Vision Computing New Zealand(IVCNZ ’13). IEEE, New York, NY, USA, 282–287. https://doi.org/10.1109/IVCNZ.2013.6727030
[15]
Marc Hassenzahl, Michael Burmester, and Franz Koller. 2003. AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. Vieweg+Teubner Verlag, Wiesbaden, 187–196. https://doi.org/10.1007/978-3-322-80058-9_19
[16]
Feng Huang, Yu Zhou, Yao Yu, Ziqiang Wang, and Sidan Du. 2011. Piano AR: A Markerless Augmented Reality Based Piano Teaching System. In Proceedings of the 2011 Third International Conference on Intelligent Human-Machine Systems and Cybernetics - Volume 02(IHMSC ’11). IEEE Computer Society, USA, 47–52. https://doi.org/10.1109/IHMSC.2011.82
[17]
Jakob Karolus, Annika Kilian, Thomas Kosch, Albrecht Schmidt, and Paweł W. Wozniak. 2020. Hit the Thumb Jack! Using Electromyography to Augment the Piano Keyboard. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. Association for Computing Machinery, New York, NY, USA, 429–440. https://doi.org/10.1145/3357236.3395500
[18]
Mohammed K Khalil, Fred Paas, Tristan E Johnson, and Andrew F Payer. 2005. Design of interactive and dynamic anatomical visualizations: the implication of cognitive load theory. The Anatomical Record Part B: The New Anatomist: An Official Publication of the American Association of Anatomists 286, 1(2005), 15–20.
[19]
Sunjun Kim, Byungjoo Lee, and Antti Oulasvirta. 2018. Impact activation improves rapid button pressing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM Digital Library, Montreal, Canada, 1–8.
[20]
Melina Klepsch, Florian Schmitz, and Tina Seufert. 2017. Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load. Frontiers in psychology 8 (2017), 1997.
[21]
David A Kolb. 2014. Experiential learning: Experience as the source of learning and development. FT press, New Jersey.
[22]
Sébastien Lallé and Cristina Conati. 2019. The role of user differences in customization: a case study in personalization for infovis-based content. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM Digital Library, Los Angeles, USA, 329–339.
[23]
Kevin C. Lam, Carl Gutwin, Madison Klarkowski, and Andy Cockburn. 2021. The Effects of System Interpretation Errors on Learning New Input Mechanisms. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 713, 13 pages. https://doi.org/10.1145/3411764.3445366
[24]
Edward W Large and Caroline Palmer. 2002. Perceiving temporal regularity in music. Cognitive science 26, 1 (2002), 1–37.
[25]
Byungjoo Lee. 2016. Temporal Pointing. Yonsei University. http://www.leebyungjoo.com/Temporal-Pointing-1
[26]
Byungjoo Lee, Qiao Deng, Eve Hoggan, and Antti Oulasvirta. 2017. Boxer: a multimodal collision technique for virtual objects. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. ACM Digital Library, Quebec, Canada, 252–260.
[27]
Byungjoo Lee and Antti Oulasvirta. 2016. Modelling Error Rates in Temporal Pointing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1857–1868. https://doi.org/10.1145/2858036.2858143
[28]
Edward A Lippman. 1984. Progressive temporality in music. Journal of Musicology 3, 2 (1984), 121–141.
[29]
Andrew P. McPherson and Youngmoo E. Kim. 2011. Multidimensional Gesture Sensing at the Piano Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 2789–2798. https://doi.org/10.1145/1978942.1979355
[30]
Eunji Park and Byungjoo Lee. 2020. An Intermittent Click Planning Model. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376725
[31]
Jeff Pressing. 1998. Error correction processes in temporal pattern production. Journal of Mathematical Psychology 42, 1 (1998), 63–101.
[32]
Remy MJP Rikers, Pascal WM Van Gerven, and Henk G Schmidt. 2004. Cognitive load theory as a tool for expertise development. Instructional Science 32, 1-2 (2004), 173–182.
[33]
Katja Rogers, Amrei Röhlig, Matthias Weing, Jan Gugenheimer, Bastian Könings, Melina Klepsch, Florian Schaub, Enrico Rukzio, Tina Seufert, and Michael Weber. 2014. P.I.A.N.O.: Faster Piano Learning with Interactive Projection. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (Dresden, Germany) (ITS ’14). Association for Computing Machinery, New York, NY, USA, 149–158. https://doi.org/10.1145/2669485.2669514
[34]
Marc Ericson C Santos, Angie Chen, Takafumi Taketomi, Goshiro Yamamoto, Jun Miyazaki, and Hirokazu Kato. 2014. Augmented Reality Learning Experiences: Survey of Prototype Design and Evaluation. IEEE Transactions on Learning Technologies 7, 1 (2014), 38–56. https://doi.org/10.1109/TLT.2013.37
[35]
Dirk Vorberg and Hans-Henning Schulze. 2002. Linear phase-correction in synchronization: Predictions, parameter estimation, and simulations. Journal of Mathematical Psychology 46, 1 (2002), 56–87.
[36]
Matthias Weing, Amrei Röhlig, Katja Rogers, Jan Gugenheimer, Florian Schaub, Bastian Könings, Enrico Rukzio, and Michael Weber. 2013. P.I.A.N.O.: Enhancing Instrument Learning via Interactive Projected Augmentation. In Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication (Zurich, Switzerland) (UbiComp ’13 Adjunct). Association for Computing Machinery, New York, NY, USA, 75–78. https://doi.org/10.1145/2494091.2494113
[37]
Alan M Wing and AB Kristofferson. 1973. The timing of interresponse intervals. Perception & Psychophysics 13, 3 (1973), 455–460.
[38]
Alan M Wing and Alfred B Kristofferson. 1973. Response delays and the timing of discrete motor responses. Perception & Psychophysics 14, 1 (1973), 5–12.
[39]
Beste F. Yuksel, Kurt B. Oleson, Lane Harrison, Evan M. Peck, Daniel Afergan, Remco Chang, and Robert JK Jacob. 2016. Learn Piano with BACh: An Adaptive Learning Interface That Adjusts Task Difficulty Based on Brain State. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 5372–5384. https://doi.org/10.1145/2858036.2858388
Index Terms
- Piano Learning and Improvisation through Adaptive Visualisation and Digital Augmentation
Recommendations
A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences?
Humans have been developing and playing musical instruments for millennia. With technological advancements, instruments were becoming ever more sophisticated. In recent decades computer-supported innovations have also been introduced in hardware design, ...
P.I.A.N.O.: Faster Piano Learning with Interactive Projection
ITS '14: Proceedings of the Ninth ACM International Conference on Interactive Tabletops and SurfacesLearning to play the piano is a prolonged challenge for novices. It requires them to learn sheet music notation and its mapping to respective piano keys, together with articulation details. Smooth playing further requires correct finger postures. The ...
MirrorFugue for the Composer, Performer and Improviser
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing SystemsThis video illustrates the usage of an integrated workbench for learning and composing music on the piano. Built on a Yamaha Disklavier augmented with projection, the system suports music represented either as symbolic notation or as MirrorFugue, where ...
Comments
Information & Contributors
Information
Published In
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 November 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ISS '22
Sponsor:
Acceptance Rates
Overall Acceptance Rate 147 of 533 submissions, 28%
Upcoming Conference
ISS '24
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 118Total Downloads
- Downloads (Last 12 months)52
- Downloads (Last 6 weeks)3
Reflects downloads up to 14 Sep 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format