Computing / Digital Studios

We present a study that explores the affordance evoked by sound and sound-gesture mappings. In order to do this, we make use of a sensor system with minimal form factor in a user study that minimizes cultural association. The present study focuses on understanding how participants describe sounds and gestures produced while playing designed sonic interaction mappings. This approach seeks to move from object-centric affordance towards investigating embodied gestural sonic affordances.

Sonic interaction is the continuous relationship between user actions and sound, mediated by some technology. Because interaction with sound may be task oriented or experience-based it is important to understand the nature of action-sound relationships in order to design rich sonic interactions. We propose a participatory approach to sonic interaction design that first considers the affordances of sounds in order to imagine embodied interaction, and based on this, generates interaction models for interaction designers wishing to work with sound. We describe a series of workshops, called Form Follows Sound, where participants ideate imagined sonic interactions, and then realize working interactive sound prototypes. We introduce the Sonic Incident technique, as a way to recall memorable sound experiences. We identified three interaction models for sonic interaction design: conducting; manipulating; substituting. These three interaction models offer interaction designers and developers a framework on which they can build richer sonic interactions.

Listening comprises a tension between different signifiers and affordances, shifting temporalities, multiple perspectives, the evocation of past experience and imagination of bodily gestures and actions yet to happen. Discussing the many facets that listening unveils about our everyday sonic experience is significant within the field of Sonic Interaction Design (SID) and sound studies. The current panorama of embodied sensory technology and interactive computational sonic objects pose new ways to understand how people listen, not only using their ears but also through their bodies. By analysing the complexity and variety of listening, we can enrich our knowledge of sonic experience - beyond simple sound feedback - and envision novel scenarios of bodily sonic interactions. An attention to the topic of the temporality of listening experience can potentially help us to understand unconsidered aspects of sonic experience, envisioning novel and improving existing design of interactions that fit in already rich multi-sensory dimensions of our everyday.

This paper presents *retro-active listening*, a concept which may be helpful to extend an understanding of sonic experience, focusing on its extra-auditory contextual aspects and embodied, kinaesthetic qualities based on the process of remembering and imagining to be listening to a sound from our memory. Retro-active listening exploits evocation of listening as an action rather than anamnesis of a sound. It is intended here as a mode of listening which can be investigated with the scope of generating participatory sonic interaction design. It feeds a qualitative method which can help research in sound studies and Sonic Interaction Design to explore the specificity of personal rencounters of listening situations as happened in the past, as evoked by listeners. It focuses on considering the unconscious, embodied, kinaesthetic aspects of sonic experience. It aims to help researchers to gather further insight into the problematic and yet unexplored aspects of participants’ everyday sonic experience, which they may have forgotten or failed to notice. The idea and challenge of using this concept is that by investigating diverse everyday sonic experience specifically from the past, with a strong focus on embodiment and affect, we can “resonate” multiple meanings, contexts, references and unknown forms of felt knowledge in an embodied resonance that is specific to thinking sonically.

This paper further reports the usage of the concept of retro-active listening to develop methods for our participatory sonic interaction design workshops. They can help researchers to retrieve the contexts in which sound was heard by participants and their specific, particular reactions and feelings. It may help to evoke a felt, corporeal awareness and bodily sensitivity that can be retained during the design phases. From this concept I developed the technique of the *sonic incident*, in which my collaborators and I explore participants’ recounted stories of problematic episodes from their everyday life in which sounds were memorable. These episodes are then used as case-studies and starting point for imagining possible interaction with sound and interactive technologies. Retro-active listening helped us to access forgotten, unnoticed and un-gestured embodied action-sound relationships and contextual information from participants’ everyday, helping us to explore not only a more “sonic” but also a more “embodied” approach to Interaction Design.

This paper presents work in progress on applying a
Multimodal interaction (MMI) approach to studying
interactive music performance. We report on a study
where an existing musical work was used to provide a
gesture vocabulary. The biophysical sensing already used
in the work was used as input modality, and augmented
with several other input sensing modalities not in the
original piece. The bioacoustics-based sensor,
accelerometer sensors, and full-body motion capture
system generated data recorded into a multimodal
database. We plotted the data from the different
modalities and offer observations based on visual analysis
of the collected data. Our preliminary results show that
there is complementarity of different forms in the
information. We noted three types of complementarity:
synchronicity, coupling, and correlation.