research-article

A multisensory approach for the design of food and drink enhancing sonic systems

Authors:

Carlos Velasco,

Felipe Reinoso Carvalho,

Anton NijholtAuthors Info & Claims

MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food Interaction

Article No.: 7, Pages 1 - 7

https://doi.org/10.1145/3007577.3007578

Published: 16 November 2016 Publication History

Abstract

Everyday eating and drinking experiences involve multiple, interrelated, sensory inputs. However, when it comes to human-food interaction design (HFI) research, certain senses have received more attention than others have. Here, we focus on audition, a sense that has received limited attention in such context. In particular, we highlight the role of food/drink-related eating sounds, as a potential input for human-food interaction design. We review some of the few systems that have built on such sounds within food and drink contexts. We also present a multisensory design framework and discuss how the systematic connections that exist between the senses may provide some guidelines for the integration of eating sounds in HFI design. Finally, we present some key prospects that we foresee for research in technology design in HFI.

References

[1]

Auvray, M. and Spence, C. 2008. The multisensory perception of flavor. Conscious. Cogn. 17(3), 1016--1031. http://dx.doi.org/10.1016/j.concog.2007.06.005

[2]

Spence, C. 2015. Multisensory flavor perception. Cell 161(1), 24--35. http://dx.doi.org/10.1016/j.cell.2015.03.007

[3]

Prescott, J. 1999. Flavour as a psychological construct: implications for perceiving and measuring the sensory qualities of foods. Food Qual. Prefer. 10(4), 349--356. http://dx.doi.org/10.1016/S0950-3293(98)00048-2

[4]

Prescott, J. 2015. Multisensory processes in flavour perception and their influence on food choice. Curr. Opin. Food Sci. 3, 47--52. http://dx.doi.org/10.1016/j.cofs.2015.02.007

[5]

Haverkamp, M. 2009. Synesthetic design-building multi-sensory arrangements. In Kai Bronner & Rainer Hirt (eds.), Audio Branding: Brands, Sound and Communication (pp. 164--181). Nomos, Berlin, Germany.

[6]

Haverkamp, M. 2010. Synesthetic approach for evaluation of the cross-sensory quality of multi-media applications. In Quality of Multimedia Experience (QoMEX) (Trondheim, Norway, June 21--23, 2010). QoMEX. IEEE, 136--141. DOI= http://dx.doi.org/10.1109/QOMEX.2010.5516107

[7]

Haverkamp, M. 2013. Synesthetic design: Handbook for a multi-sensory approach. Birkhäuser Verlag, Basel.

[8]

Spence, C. 2015. Eating with our ears: assessing the importance of the sounds of consumption on our perception and enjoyment of multisensory flavour experiences. Flavour 4:3. http://dx.doi.org/10.1186/2044-7248-4-3

[9]

Zampini, M. and Spence, C. 2010. Assessing the role of sound in the perception of food and drink. Chemosen. Perception, 3(1), 57--67. http://dx.doi.org/10.1007/s12078-010-9064-2

[10]

Reinoso Carvalho, F., Velasco, C., van Ee, R., Leboeuf, Y. and Spence, C. 2016. Music influences hedonic and taste ratings in beer. Front. Psychol. 7: 636. http://dx.doi.org/10.3389/fpsyg.2016.00636

[11]

Vickers, Z. and Bourne, M. C. 1976. Crispness in foods: A review. J. Food Sci. 41(5), 1153--1157. http://dx.doi.org/10.1111/j.1365-2621.1976.tb14406.x

[12]

Zampini, M. and Spence, C. 2004. The role of auditory cues in modulating the perceived crispness and staleness of potato chips. J. Sense. Stud. 19(5), 347--363. h http://dx.doi.org/10.1111/j.1745-459x.2004.080403.x

[13]

Velasco, C., Jones, R., King, S. and Spence, C. 2013. The sound of temperature: What information do pouring sounds convey concerning the temperature of a beverage. J. Sense. Stud. 28(5), 335--345. http://dx.doi.org/10.1111/joss.12052

[14]

Yan, K. S. and Dando, R. 2015. A crossmodal role for audition in taste perception. J. Exp. Psychol. Hum. Percept. 41(3), 590--596. http://dx.doi.org/10.1037/xhp0000044

[15]

Spence, C. and Piqueras-Fiszman, B. 2013. Technology at the dining table. Flavour 2:16. http://dx.doi.org/10.1186/2044-7248-2-16

[16]

Spence, C. and Piqueras-Fiszman, B. 2014. The perfect meal: the multisensory science of food and dining. Wiley-Blackwell, Oxford, UK.

[17]

Spence, C. 2012. Auditory contributions to flavour perception and feeding behaviour. Physiol. Behav. 107(4), 505--515. http://dx.doi.org/10.1016/j.physbeh.2012.04.022

[18]

Obrist, M., Velasco, C., Vi, C. T., Ranasinghe, N., Israr, A., Cheok, A. D., Spence, C. and Gopalakrishnakone, P. 2016. Touch, taste, & smell user interfaces: The future of multisensory HCI. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 3285--3292. DOI= http://dx.doi.org/10.1145/2851581.2856462

Digital Library

[19]

Hashimoto, Y., Nagaya, N., Kojima, M., Miyajima, S., Ohtaki, J., Yamamoto, A., Mitani, T. and Inami, M. 2006. Straw-like user interface: virtual experience of the sensation of drinking using a straw. In Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology (ACE '06). ACM, New York, NY, USA, Article 42. DOI= http://dx.doi.org/10.1145/1178823.1178873

Digital Library

[20]

Hashimoto, Y., Inami, M. and Kajimoto. H. 2008. Straw-Like User Interface (II): A New Method of Presenting Auditory Sensations for a More Natural Experience. In Proceedings of the 6th international conference on Haptics: Perception, Devices and Scenarios (EuroHaptics '08), Manuel Ferre (Ed.). Springer-Verlag, Berlin, Heidelberg, 484--493. DOI= http://dx.doi.org/10.1007/978-3-540-69057-3_62

Digital Library

[21]

Petit, O., Cheok, A. D., Spence, C., Velasco, C. and Karunanayaka, K. T. 2015. Sensory marketing in light of new technologies. In Proceedings of the 12th International Conference on Advances in Computer Entertainment Technology (ACE '15). ACM, New York, NY, USA, Article 53, 4 pages. DOI= http://dx.doi.org/10.1145/2832932.2837006

Digital Library

[22]

Koizumi, N., Tanaka, H., Uema, Y. and Inami, M. 2011. Chewing jockey: augmented food texture by using sound based on the cross-modal effect. In Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology (ACE '11). ACM, New York, NY, USA, Article 2, 4 pages. DOI= http://dx.doi.org/10.1145/2071423.2071449

Digital Library

[23]

Kadomura, A., Tsukada, K. and Siio. I. 2013. EducaTableware: computer-augmented tableware to enhance the eating experiences. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 3071--3074. http://dx.doi.org/10.1145/2468356.2479613

Digital Library

[24]

Haverkamp, M. 2015. Can synesthetic perception help to define attractive product design? In 5^th International Congress of Synaesthesia, Science & Arts. Alcalá la Real, Jaén, Spain.

[25]

Calvert, G., Spence, C. and Stein, B. E. 2004. The handbook of multisensory processes. MIT press, Cambridge, MA.

[26]

Velasco, C., Obrist, M., Petit, O., Karunanayaka, K., Cheok, A. D. and Spence, C. (2016). Cross-modal correspondences in the context of digital taste and flavor interfaces. Presented at CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.

[27]

Spence, C. 2011. Crossmodal correspondences: A tutorial review. Atten. Percept. Psychophys. 73(4), 971--995. http://dx.doi.org/10.3758/s13414-010-0073-7

[28]

Chen, L. and Vroomen, J. (2013). Intersensory binding across space and time: a tutorial review. Atten. Percept. Psychophys., 75(5), 790--811. http://dx.doi.org/10.3758/s13414-013-0475-4.

[29]

Iordanescu, L., Grabowecky, M. and Suzuki, S. 2011. Object-based auditory facilitation of visual search for pictures and words with frequent and rare targets. Acta Psychol. 137(2), 252--259. http://dx.doi.org/10.1016/j.actpsy.2010.07.017

[30]

Milliman, R. E. 1986. The influence of background music on the behavior of restaurant patrons. J. Cons. Res. 13(2), 286--289. http://dx.doi.org/10.1086/209068

[31]

Marks, L. E. 1978. The unity of the senses: Interrelations among the modalities. Academic Press, New York, NY.

[32]

Martino, G. and Marks, L. E. 2001. Synesthesia: Strong and weak. Curr. Dir. Psychol. Sci. 10(2), 61--65. http://dx.doi.org/10.1111/1467-8721.00116

[33]

Reinoso Carvalho, F., Van Ee, R., Rychtarikova, M., Touhafi, A., Steenhaut, K., Persoone, D., Spence, C. and Leman, M. 2015. Does music influence the multisensory tasting experience?. J. Sense. Stud. 30(5), 404--412. http://dx.doi.org/10.1111/joss.12168

[34]

Velasco, C., Woods, A. T., Petit, O., Cheok, A. D. and Spence, C. 2016. Crossmodal correspondences between taste and shape, and their implications for product packaging: A review. Food Qual. Prefer., 52, 17--26. http://dx.doi.org/10.1016/j.foodqual.2016.03.005

[35]

Bahrick, L. E., Lickliter, R. and Flom, R. 2004. Intersensory redundancy guides infants' selective attention, perceptual and cognitive development. Curr. Dir. Psychol. Sci. 13(99), 102. http://dx.doi.org/10.1111/j.1467-7687.2006.00539.x

[36]

Parise, C. V., Knorre, K. and Ernst, M. O. 2014. Natural auditory scene statistics shapes human spatial hearing. Proc. Natl. Acad. Sci. U.S.A., 111(16), 6104--6108. http://dx.doi.org/10.1073/pnas.1322705111

[37]

Deroy, O., Crisinel, A. S. and Spence, C. 2013. Crossmodal correspondences between odors and contingent features: odors, musical notes, and geometrical shapes. Psychon. Bull. Rev., 20(5), 878--896. http://dx.doi.org/10.3758/s13423-013-0397-0

[38]

Palmer, S. E., Langlois, T. A. and Schloss, K. B. 2016. Music-to-color associations of single-line piano melodies in non-synesthetes. Multis. Res. 29(1-3), 157--193. http://dx.doi.org/10.1163/22134808-00002486

[39]

Parise, C. and Spence, C. 2013. Audiovisual cross-modal correspondences in the general population. In J. Simner & E. Hubbard (Eds.), The Oxford handbook of synaesthesia (pp. 790--815). Oxford University Press, Oxford, UK.

[40]

Ludden, G. D., Kudrowitz, B. M., Schifferstein, H. N. and Hekkert, P. 2012. Surprise and humor in product design. Humor 25(3), 285--309. http://dx.doi.org/10.1515/humor-2012-0015

[41]

Ludden, G. D. and Schifferstein, H. N. 2007. Effects of visual-auditory incongruity on product expression and surprise. Int. J. Des. 1(3), 29--39.

[42]

Ludden, G. D., Schifferstein, H. N. and Hekkert, P. 2008. Surprise as a design strategy. Des. Issues 24(2), 28--38. http://dx.doi.org/10.1162/desi.2008.24.2.28

[43]

Mandler, G. P. 1982. The structure of value: Accounting for taste. In M. S. Clark & S. T. Fiske (Eds.), Affect and cognition. The 17th Annual Carnegie Symposium on Cognition (pp. 3--36). Lawrence Erlbaum, Hillsdale, NJ.

[44]

Marinetti, F. T. 1909. The founding and manifesto of futurism. Retrieved from http://web.itu.edu.tr/~inceogl4/thresholds/futuristmanifesto.doc

[45]

Howells, T. 2014. Experimental eating. Black Dog Publishing, London, UK.

[46]

Fox, J., Bailenson, J. and Binney, J. 2009. Virtual experiences, physical behaviors: The effect of presence on imitation of an eating avatar. Presence-Teleop. Virt., 18(4), 294--303. http://dx.doi.org/10.1162/pres.18.4.294

Digital Library

[47]

Project Nourishing, retrieved from http://www.projectnourished.com/

[48]

Spence, C. and Wang, Q. J. 2015. Sensory expectations elicited by the sounds of opening the packaging and pouring a beverage. Flavour 4:35. http://dx.doi.org/10.1186/s13411-015-0044-y

[49]

Stroebele, N., & de Castro, J. M. 2006. Listening to music while eating is related to increases in people's food intake and meal duration. Appetite 47(3), 285--289. http://dx.doi.org/10.1016/j.appet.2006.04.001

[50]

Guéguen, N., Jacob, C., Le Guellec, H., Morineau, T., & Lourel, M. 2008. Sound level of environmental music and drinking behavior: A field experiment with beer drinkers. Alcohol Clin. Exp. Res. 32(10), 1795--1798. http://dx.doi.org/10.1111/j.1530-0277.2008.00764.x

[51]

Elder, R. S., & Mohr, G. S. 2016. The crunch effect: Food sound salience as a consumption monitoring cue. Food Qual. Prefer. 51, 39--46. http://dx.doi.org/10.1016/j.foodqual.2016.02.015

Cited By

Wilson DMeester FTakahashi TSingh RButtar H(2024)Food Color, Taste, Smell, Culinary Plate, Flavor, Locale, and their Impact on Nutrition: Present and Future Multisensory Food Augmentation and Noncommunicable Disease Prevention: An OverviewBotanicals and Natural Bioactives: Prevention and Treatment of Diseases10.2174/9789815238273124020004(1-24)Online publication date: 24-Mar-2024
https://doi.org/10.2174/9789815238273124020004
Choi EKang Y(2024)A Multi-sensory Kiosk Interface to Familiarize Users with New FoodsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435458:1(1-26)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643545
Chen WYang YLiu KPai YYamaoka JMinamizawa K(2024)Cymatics Cup: Shape-Changing Drinks by Leveraging CymaticsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642920(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642920
Show More Cited By

Index Terms

A multisensory approach for the design of food and drink enhancing sonic systems
1. Applied computing
  1. Law, social and behavioral sciences
    1. Psychology

Recommendations

Exploring the Design Space for Human-Food-Technology Interaction: An Approach from the Lens of Eating Experiences
Embedded in everyday practices, food can be a rich resource for interaction design. This article focuses on eating experiences to uncover how bodily, sensory, and socio-cultural aspects of eating can be better leveraged for the design of user experience. ...
First Multimodal Banquet: Exploring Innovative Technology for Commensality and Human-Food Interaction (CoFI2024)
ICMI '24: Proceedings of the 26th International Conference on Multimodal Interaction

Commensality, the act of eating together, offers a rich multisensory and social experience that technology can enhance. Dining involves interactions with food, where smells, colors, sounds, and textures contribute to a multisensory experience. The table ...
You Better Eat to Survive! Exploring Edible Interactions in a Virtual Reality Game
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

"You Better Eat to Survive!" is a two-player virtual reality game that involves eating real food to survive and ultimately escape from a virtual island. Eating is sensed through capturing chewing sounds via a low-cost microphone solution. Unlike most VR ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MHFI '16: Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food Interaction

November 2016

48 pages

ISBN:9781450345613

DOI:10.1145/3007577

Editors:
Anton Nijholt
HMI, University of Twente, Netherlands, and Imagineering Institute, Malaysia
,
Carlos Velasco
BI Norwegian Business School, Norway, and University of Oxford, UK
,
Gijs Huisman
HMI, University of Twente, Netherlands
,
Kasun Karunanayaka
Imagineering Institute, Malaysia

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

CAPES Foundation, Brazil

Conference

ICMI '16

ICMI '16: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 16, 2016

Tokyo, Japan

Acceptance Rates

MHFI '16 Paper Acceptance Rate 6 of 8 submissions, 75%;

Overall Acceptance Rate 6 of 8 submissions, 75%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
704
Total Downloads

Downloads (Last 12 months)44
Downloads (Last 6 weeks)4

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wilson DMeester FTakahashi TSingh RButtar H(2024)Food Color, Taste, Smell, Culinary Plate, Flavor, Locale, and their Impact on Nutrition: Present and Future Multisensory Food Augmentation and Noncommunicable Disease Prevention: An OverviewBotanicals and Natural Bioactives: Prevention and Treatment of Diseases10.2174/9789815238273124020004(1-24)Online publication date: 24-Mar-2024
https://doi.org/10.2174/9789815238273124020004
Choi EKang Y(2024)A Multi-sensory Kiosk Interface to Familiarize Users with New FoodsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435458:1(1-26)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643545
Chen WYang YLiu KPai YYamaoka JMinamizawa K(2024)Cymatics Cup: Shape-Changing Drinks by Leveraging CymaticsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642920(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642920
Wang YObie HLi ZSalim FGrundy JMueller F(2024)GustosonicSense: Towards understanding the design of playful gustosonic eating experiencesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642182(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642182
Spielmann NRossi P(2024)Let’s drink to that! How wider glass rim leads to more pleasurable purchasesJournal of Business Research10.1016/j.jbusres.2024.114579175(114579)Online publication date: Mar-2024
https://doi.org/10.1016/j.jbusres.2024.114579
Karhu NRantala JFarooq ASand APennanen KLappi JNayak MSozer NRaisamo R(2024)The effects of haptic, visual and olfactory augmentations on food consumed while wearing an extended reality headsetJournal on Multimodal User Interfaces10.1007/s12193-024-00447-8Online publication date: 11-Dec-2024
https://doi.org/10.1007/s12193-024-00447-8
Vi CCornelio PObrist MYeomans M(2023)“Sweet: I did it”! Measuring the sense of agency in gustatory interfacesFrontiers in Computer Science10.3389/fcomp.2023.11282295Online publication date: 12-May-2023
https://doi.org/10.3389/fcomp.2023.1128229
James MRanasinghe NTang AOehlberg L(2022)Flavor-Videos: Enhancing the Flavor Perception of Food while Eating with VideosProceedings of the 2022 ACM International Conference on Interactive Media Experiences10.1145/3505284.3529967(33-46)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3505284.3529967
Deng JOlivier PAndres JEllis KWee RFloyd Mueller F(2022)Logic Bonbon: Exploring Food as Computational ArtifactProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501926(1-21)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501926
Gayler TSas CKalnikaitė V(2022)Exploring the Design Space for Human-Food-Technology Interaction: An Approach from the Lens of Eating ExperiencesACM Transactions on Computer-Human Interaction10.1145/348443929:2(1-52)Online publication date: 16-Jan-2022
https://dl.acm.org/doi/10.1145/3484439
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten