The Notion of Participative and Enacting Sonic Interaction (PESI)

The project deals with the social aspect of participation and musicianship dynamics in collective music performance. The PESI research proposes a novel collaborative environment using mobile phones as tangible and expressive musical instruments, in parallel with a spatial system. This extended system incorporates the mobile instruments and motion tracking technology to create an environment in which performers are not only free to move and interact with each other but where their social interactions contribute to the sonic outcome. PESI system is designed for co-located collaboration and provides embodied and spatial opportunities for musical exploration. PESI accounts for forms of interaction with digital technologies that are embedded in physical and social environments [1].

The user-test jam session with musicians; Ilkka Niemeläinen, Antti Ikonen and James Andean

We implemented the PESI system in two parts – on the body and in space. On-body component consists of mobile phones and portable speakers. Performers use the mobile devices to control their individual instruments. Each one wears a pair speakers on his/her body that are directly connected to the mobile device. In-space, the multi-user tracking and central management unit provides a robust tracking and analysis system that adapts quickly and reliably to any changes in a dynamic, group improvisation performance. It receives a continuous stream of data from the rest of the components. It uses this data to extract actions and to modulate the individual sound outputs. This allows a new layer of sound to be presented based on the social relations among the performers.

Motion tracking with 2 Kinect sensors provides a solution to avoid losing track of the occluded participant

Our previous study showed that novel strategies in the development of new interfaces with extended systems enable players to become more aware of their social interaction [2]. The main emphasis of this study was to understand more about bodily interaction among participants, specially in relation to their location, distance and coordination as a group, and using these parameters as sound producing events [3]. The results of the analysis show the suitability of our approach but also the need for iterating on our design on the basis of the perspectives brought forth by the users.

We have previously described and presented the technical architecture of the system and the project itself in TEI 2013 [4] and -upcoming – in NIME 2013 [5].

We have been also developing a repertoire to be performed with the PESI system.

In-Hands: free-improv set by Ilkka Niemeläinen, Antti Ikonen and James Andean

 

 

The PESI Repertoire

 

PESI Extended System version 0.02 (2012-2013) coming soon!!
PESI software system with OnBody and InSpace components

PESI ToolKit version 0.01 (2011-2012)
PESI_integrationStation.zip
MobileMusicSourceDocumentation.pdf
AudioSynthesisTools.zip

[1] Dourish, P. (2004). Where the Action Is: The foundations of Embodied Interaction. The MIT Press, Cambridge, MA.

[2] Pugliese, R., Tahiroğlu, K., Goddard, C., Nesfield, J. (2012). A Qualitative Evaluation of Augmented Human-Human Interaction in Mobile Group Improvisation. in the Proceedings of New Interfaces for Music Expression (NIME). Ann Arbor, Michigan, USA.

[3] Tahiroğlu, K. and Nesfield J. (2012). Embodied Interactions in Collaborative Music Experience. Chapter in Senses of Embodiment Art, Technics, Media (Mika Elo and Miika Luoto eds.), Peter Lang. Spring 2013. In Press.

[4] Correia, N.N., Tahiroğlu, K. and Espada, M. (2013). Pesi: Extending Mobile Music Instruments with Social Interaction, in Seventh International Conference on Tangible, Embedded and Embodied Interaction (TEI), Work in Progress, Barcelona, Spain.

[5] Tahiroğlu, K., Correia, N.N., and Espada, M. (2013). PESI Extended System: In Space, On Body, with 3 Musicians, -upcoming – in the Proceedings of New Interfaces for Music Expression (NIME), Daejeon + Seoul, Korea Republic.