The research Networks of Intelligent Sonic Agents – NOISA studies a new way to inquire the engaging relationship we have with digital artefacts and new media practices. The main argument here is that NOISA provides an intelligent system, that acts to maintain and deepen the users engagement by learning from the user behaviour while interacting with digital environments.
Network of Intelligent Sonic Agents (NOISA) aims to create a reactive system that uses ambient intelligence to adapt the current system state. The ambient intelligence includes techniques to measure in real-time the internal states of a user. These states include motivation, affective states and reactiveness. The task is to maintain these states – if the user loses his motivation or interest, the system will react by changing its behaviour in order to make the system interesting again. Or if the user is very motivated and highly concentrated the system can provide more subtle controls. The system controls the level of control based on the states of the user. The concept is used in a musical context, but it can be extended to any other.
At the core of NOISA, an intelligent layer learns relationships between the sonic agents, the user and the environment. The intelligent layer adapts the behavior of the agents from data gathered from the users manipulations of the agent parameters, as well as additional data about the environment and the psychological state of the user. Each sonic agent has a machine-learning model for predicting the agent’s state from other agents’ data as well as environmental and user state data. On a system wide scale the data can be divided into input- output data (agent states) and input-only data (user and the environment). The learning weights of the concurrent data samples are determined by an engagement variable derived from the user model.
the first version of the NOISA interface
Regression algorithms are considered for the machine-learning problem, because the sonic agent parameters are represented by continuous values. Currently, the most promising results have been obtained with decision tree regression. The ability of the tree model to derive the predicted parameter value from decisions and branches leads to predictable and representative behaviour based on the learning data. The ability to predict complete system states could be expected to have a more musical result. Applying a regression algorithm that attempts to fit a function to the learning data, such as support vector regression, seems to result in the system seeking some internal balance leading to a less dynamic behaviour. Additionally, auto-regressive components in the learning model can provide the system with a better sense of temporal structure. Exploration of the sonic behaviour of different regression algorithms is expected to guide the future development.
— Compositions for NOISA —
3 Agents + 1
The composition 3 agents + 1 exploits recently developed affordances in NOISA musical instruments. These instruments act as three networked-agents in live performance to maintain and deepen the performer’s engagement with their interfaces in an unusual way. The instruments are extended to incorporate the performance as they become part of the distribution of decision-making, transforming their physical control inputs consistently and communicate with the performer within own acoustic contexts.
NOISA Etude #1
“NOISA Etude #1” is a first set of performance instructions created to showcase compelling, evolving and complex soundscapes only possible when operating the NOISA instruments, integrating the system’s response as part of a musical piece. The multi-layered sound interaction design is based on radical transformations of acoustic instruments performing works from the classical music repertoire. This first “Etude” is based on frequency-tuned granular synthesis systems.
NOISA Etude #2
“NOISA Etude #2” is a second set of performance instructions. The system is fed with variations of a fixed musical motif, encouraging the system to recognise elements of the motive and create its own set of different versions emulating a human musical compositional process. Also, the Myo Armband is used in a creative way as an independent element for dynamic control, using raw data extracted from the muscles’ tension. This piece is accompanied by a performance score with detailed instructions regarding the performer actions, pauses for system reaction, and different course routes depending on the system’s response.
