UNIVERSIDADE ESTADUAL DE CAMPINAS SISTEMA DE BIBLIOTECAS DA UNICAMP
REPOSITÓRIO DA PRODUÇÃO CIENTIFICA E INTELECTUAL DA UNICAMP
Versão do arquivo anexado / Version of attached file:
Versão do Editor / Published Version
Mais informações no site da editora / Further information on publisher's website:
https://library.iated.org/view/PARTESOTTI2017INT
DOI: 10.21125/inted.2017.2302
Direitos autorais / Publisher's copyright statement:
©2017
by The International Academy of Technology, Education and Development
(IATED). All rights reserved.
DIRETORIA DE TRATAMENTO DA INFORMAÇÃO Cidade Universitária Zeferino Vaz Barão Geraldo
CEP 13083-970 – Campinas SP Fone: (19) 3521-6493 http://www.repositorio.unicamp.br
INTERACTIVE MUSICAL TECHNOLOGY ENHANCES CREATIVITY: A
CASE STUDY WITH E-MOCOMU TECHNOLOGY
Elena Partesotti1, Jônatas Manzolli2, Alicia Peñalba Acitores1
1University of Valladolid (Spain) 2University of Campinas (Brazil)
Abstract
This paper presents a case study on interactive musical technology and discusses how it enhances Creative Empowerment gained through the proprioceptive awareness of participants within an interactive environment. We propose the concepts of Creative Empowerment and Sensorimotor Maps [1] in order to explain this thesis. Our viewpoint is that this kind of technology could be applied both in therapeutic and educative environments, bringing benefits in the learning process of people of any age and gender. Our study also connects DMI technology to Mixed Interactive Reality, which is based on the proprioceptive interaction of the user and integrates diverse perceptual modalities [1]. The research verifies how e-mocomu’s prototype technology might integrate the following sensorimotor contingencies: visual, auditory and proprioceptive, showing the potentialities that this kind of interactive instrument could offer.
Keywords: Creative Empowerment, Sensorimotor Maps, proprioception, learning,
MIR technology.
1 INTRODUCTION
The introduction of musical technology in therapeutic and pedagogical settings has raised major interests, although this is not yet a common practice. In recent years, the massive development of music technologies such as Digital Musical Instruments has attracted interest from diverse disciplines, opening a dialogue regarding the effectiveness of such implementation in various cognate fields. Digital Musical Interfaces (DMIs) contain a control surface and a sound generation unit, are designed for various contexts, such as musical and artistic applications, and can be used by non-experts and non-experts [2]. In music, implementation of these gestural controllers in everyday life is increasingly common, enabling the user to become both the creative and the performer, regardless of his/her musical skills. One example is Biophilia by the the composer Bjork, which implement the concept of the ReacTable by Jordà [3], or the Music Gloves by Imogen Heap, designed to make music with the performer's hands.
In recent literature, studies have shown the effectiveness of the technological environment applied to pedagogic and therapeutic setting [4, 5, 6, 7]. As Repetto outlines [8], metaphors are always present in the expressive modalities of children, therefore allowing an immediate approach to technology. Parmigiani [9] outlines the diverse phases that occur during the experimentation and exploration processes between the user and the technology. They consist of:
1. Planning phase 2. Creative phase 3. Research phase
4. Proposal phase 5. Organizational phase 6. Achievement phase 7. Check the above phases
These phases could also be expanded to the therapeutic setting, since they encompass different stages in the formation and expression of a communicative bond with therapist. Furthermore, the implementation of interactive systems in educational programs could represent an innovative and alternative tool for teachers, offering personalised learning and at the same time proposing an expressive instrument thorough kinetic, musical and artistic training [4]. As Camurri et al., [4] suggest, the interactive technology could enhance a multiple intelligences' based-training paradigm. In their study, the authors [4] apply an audio-visual technology for pedagogical training, observing an incremental improvement in their skills – in particular, in the treatment of dyslexia – paving also its implementation with ASD clients. In this context, technology permits a ludic approach to education, in particular a game-based learning [10]. This ludic approach has been successfully implemented in the recovery of post-stroke patients, through Rehabilitation Gaming Systems [11]. In yet another study with patients affected by Parkinson’s’ disease, the authors observed consistent improvement of motivation and physical benefits, after a period (between 2 and 6 months) of music therapy with an aesthetically resonant environment with audio-visual feedback [12]. The aesthetically resonant environment refers to the CARE-HERE framework, a project that aimed to deploy interactive, virtually-spaced audio-visual feedback for a therapeutic purpose. All of the abovementioned examples of interactive systems promote the learning process through subject performance, encouraging self-expression and communication, and above all promoting better proprioceptive awareness. Thus, a proprioception-based interactive system allows the user to develop cognitive skills [12], and to achieve self-awareness and self-control independently from their limited mobility. The music therapist Berger [13], in fact, outlines the importance of a eurhythmic-based sensorimotor approach, which promotes proprioception in clients with neurophysiological, medical and sensory problems, observing a sense of wellness and betterment in them. Furthermore, Magee [14] talking about music technology in therapeutic and health settings, stressed the importance of a somatosensory experience to enhance the perception of cause-and-effect. This process permits a person to "move towards [his/her] goals”, once the client has understood how to properly manipulate the musical technology. A similar concept has been outlined by Benford & Giannachi [15], regarding creativity induced by control over the technology: “[…] mixed reality performances and other similar experiences designed in such a way that they do involve a certain degree of steering, they also facilitate emergent creativity and play and stimulate thought about the consequences of performing ‘command-and-control’ technologies”. Another concept which emphasizes the process of gaining control, is the abovementioned aesthetic resonation. This notion was outlined both by Ellis and Brooks [16, 17], describing the user’s experience with diverse DMIs and referring to “a situation where the response to an intent is so immediate and aesthetically pleasing as to make one forget the physical movement (and often effort) involved in the conveying of the intention” [16].
Brooks & Husselblad highlight the concept of aesthetic resonance that emerges in their audio-visual environment, referring to “a situation where the response to an intent is so immediate and aesthetically pleasing as to make one forget the physical movement
(and often effort) involved in the conveying of the intention” [16]. This concept shares commonalities with the Creative Empowerment hereinafter proposed. An interactive system, in fact, could foster the development of language, social skills and emotional expression in children with DSA. As the music therapist Berger outlined [18], the sensorial integration within the session may help in the early stage of DSA treatment, enhancing the development of emotional and cognitive responses. This kind of integration may be possible through an audio-visual interactive system, an integration that may strengthen creative and emotional expression [19], speeding up the recovery process of clients with Parkinson's [12] and offering an innovative tool to grab the attention of students.
Despite the limitations that may occur within the music therapy sessions [14], which are mainly due to a lack of training, a DMI would be easier to manipulate in classrooms by the teacher.
1.1 A performative space
Multiple Affordances (MA) and Creative Empowerment (CE) take place in a Mixed Interactive Reality (MIR) technology based on the user’s movements. These concepts are grounded on the Sensorimotor Contingency Theory and Gibson’s theory of visual perception, in particular the concept of Affordance [20, 21]. While, in the former, a subject that is aware of his/her movement in the space “is able to make a use of information not only about that to which he or she is perceptually sensitive, but also about the character of his or her perceptual tracking of the environment, we say is aware of what he or she perceives”, in the latter, the Affordance refers to the characteristics of the environment offered to the subject, “a point of contact between the creature and its environment, an environment in which the creature moves around and within which it acts” [22]. As outlined by Partesotti [1], both MA and Sensorimotor Maps (SM) are determined by the subject’s exploration of the environment and the mapping done a priori. While MA are the possible paths made available to the subject depending on the interaction with the MIR, the user can change the type of interaction or the ways are utilised, in accordance with a specific goal within the environment. Enclosed by the MA, we find the SM that involve the creation of sensorimotor paths linked to changes produced by the perceptual information arriving from the environment with the performer’s movements. Thus, they are tied to proprioceptive awareness. The concept of CE therefore relies on the SM and MA, “as happens when the subject gains full control over the technology, and is therefore capable of creative expression, self-control and awareness” [1]. The conceptual diagram in Table 1 describes the co-determination characteristic of the entire process [1], which involves: a) Exploratory movement made possible by the MIR’s mapping, determined a
priori.
b) Determination of SM, the user’s exploratory movement and proprioceptive awareness with the environment, MA and CE.
Table 1. Conceptual diagram of the co-determination processes between MA, SM and CE
2 AIMS
The case study uses e-mocomu (e-motion, colours and music), a prototype technology developed to enable users to control sounds and colours by means of their movements, and which integrates diverse perceptual modalities that permit their creative expression. Throughout the proprioceptive interaction, the subject becomes both performer and composer. The movement, in fact, allows the interaction and the perception of the interactive system which surround the user. As previously mentioned [12,13], learning through bodily movement strengthens the development of cognitive processes and enactive knowledge. Moreover, as “cognitive control can be understood as emotional process” [23], we predict an improvement in the emotional processing of the participants. In fact, the subject faces several conflict-based tasks and then must regulate a goal-oriented behaviour in response. We hypothesize that this process is made possible through self-control and creative self-expression triggered by Creative Empowerment. Thus, this study aims to offer a perspective on the possibilities that music interactive technology could offer if properly implemented in the field of education and music therapy.
3 METHODOLOGY
E-mocomu technology stands for e-motion, colour and music, and is an alternate gestural controller in which sounds and colours are regulated by the spatial movements of users. The case study examined 17 participants between 20 and 30 years of age, all of whom were recruited from the University of Brazil. Because of the audio-visual integration, we chose to divide the participants into 3 groups: Neutral Group (NG) with
User Multiple Affordances Mapping Proprioception Sensorimotor Maps Exploratory Movement Creative Empowerment
no previous musical knowledge, Music Group (MG) with previous musical knowledge, and Art Group with previous artistic knowledge. Each participant followed a set of tasks for gestural and audio-visual analysis, standing in a dark room with a screen in front of him/her and eventually was free to improvise with e-mocomu's technology. A Microsoft Kinect was set in front of the users, in order to track their arm movement.
The experiment was structured as follows:
1) Phase 0. At the beginning, each participant was free to explore the structure of e-mocomu for a few seconds, and interact with only with their right hand. 2) Phase 1 and 3. Participants were asked to follow with their right hand the
audio-visual circle displayed on the screen in front of them.
3) Phase 2 and 4. Participants were asked to repeat the task observed in phase 1 and 3.
4) Phase 5. Free audio-visual improvisation.
After phase 0, we ran the experiment which involved participants watching two short training-videos, each one minute in length (phases 1 and 3). After each training phase, participants performed an assessment task (phase 2 and 4) in order to evaluate which sensorimotor contingency was strongest (visual-proprioceptive, auditory-proprioceptive or auditory-proprioceptive). In each of these parts, we presented two of the three stimuli, and assessed whether participants were able to recognise the third. Participants were instructed to listen to and watch the colour-sound pair on the screen and put their right hand in the exact corresponding position in space as per their previous training. When presented with a colour corresponding to a particular pitch-note, participants were asked to sing the appropriate pitch, or name the. The experimental tasks lasted approximately 10 minutes, while the improvisation phase was not time-limited.
During the study, it was possible to collect data regarding self-perceived outcomes for each of the participants. In fact, every performance was videotaped; furthermore, we recorded the length and the acceleration of users’ movements in time, during their improvisation session, in order to track their SM and their CE. Before and after the test, we collected additional information with the SAM test (Self-Assessment Manikin) technique, developed by Lang (1980) [24]. This test provided a formal emotional evaluation regarding participants' levels of valence and arousal. We then compared participant’s Arousal and Valence to data from their performances, including: time of the improvisation sessions, acceleration curves and analysis of their movements based on the video annotations.
4 RESULTS
The results outline a change both in arousal and in valence before and after the experiment. These two parameters are responsible for an augmentation in positive expectations related to the user's performance. Figure 2 shows the change before and after the test of the Neutral Group, in particular, illustrating a stabilisation of arousal and an augmentation in valence. Additionally, a correlation between the change in valence and arousal levels and the duration in time of the users’ performance (phase 5) was found: users that showed a significant change in both arousal and valence, spent more time in the last phase. In other words, subjects that performed for a longer session had a more regular acceleration curve, thus a higher CE, and their
self-perceived outcome from the experiment was positive. Figure 3 shows the acceleration curve of 2 participants from Music Group during the improvisation session. For the first participant (75 seconds over a total of 2 minutes of improvisation’s length) there was a clear and linear pattern, in which the subject had already created the SM applying MA offered by the MIR, thus experiencing a process of CE interacting within it. On the contrary, the second (only 35 seconds of length) participant showed a discontinuous acceleration curve, which is corroborated by the video analysis of the performance, which showed that the user did not explore nor interact with the technology to the same degree, instead indicating frustration.
Figure 2. Acceleration curves of 2 participants from Music Group. 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576 0 0.1 0.2 0.3 0.4 0.5 0.6 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0 5 10 15 20 25 30 35 40 Ac ce le ra ti on Time
Figure 3. The levels of Valence and Arousal of the Neutral Group before and after the experiment.
5 CONCLUSIONS
Users employing the technology under scrutiny demonstrated increased levels of both arousal and valence. This can be attributed to a positive self-perceived outcome of the performance and a positive increase in the user’s self-confidence. The SM and MA that emerge from the user’s exploration of the environment were properly adapted during the improvisation session (phase 5), as evidence from a correspondence between the acceleration graphs and positive self-esteem outcomes from SAM. Therefore, as the users reaches the CE, the technology responded by offering the user multiple, diverse ways to afford it, thus generating a co-determination between MA and SM in a cyclic, seamless process. Consequently, as the results outlined, the duration of the creative performance is connected to the CE, gained through a proprioceptive awareness. Hence, the case study highlights how a DMI, that arouses the CE through multi-sensory integration, could benefit the learning process of the participant, both in educational and therapeutic contexts.
Technology is growing faster and there is a clear need for appropriate training in the field of music technology, allowing teachers and therapists to offer an innovative tool to the pupils. Moreover, with the introduction of a DMI within the music therapy sessions or in the classroom, the therapeutic bond could be strengthened in the early stage of the user’s process. This occurs when the subject reach the CE easily and through the help of the professional (therapist or teacher). As the experiment
0 0,2 0,4 0,6 0,8 1 1,2 1 2 3 4 5 6 NG Valence
Prima dell'esperimentoBefore Dopo l'esperimentoAfter
0 0,2 0,4 0,6 0,8 1 1,2 1 2 3 4 5 6 NG Arousal
demonstrates, these types of musical technologies can facilitate a shift from traditional learning towards a mode of learning in which the learner's errors have an explorative implication. These errors can trigger changes in one's motivational and emotional state (as these cannot be considered separately [22]), consequently driving changes in the subject’s behaviour. In particular, emotion “would play an important role in the engagement of cognitive and behavioural resources to resolve conflict … emotional stimuli help promote adaptive behavioural responses” [22]. Therefore, MIR technologies may be applied to study cognitive control, observing the emotional states of the user in order to foster a system of learning based on the development of bodily and cognitive processes.
The increase in valence, moreover, may encourage the user to experience the technological environment and to share it within the group. More experiments with MIR technologies will be necessary to verify the limitations and benefits of multi-sensory based interactive tools in educative and therapeutic disciplines, with a focus on emotional states, conflict resolution and Rehabilitation Gaming Systems.
6 FURTHER STEPS
Additional steps in the research conducted in this field concern the design of a technology that could fulfil the following aims:
a) Providing an opportunity to create a self-profile for each client by recording the individual’s data in order to track the appropriate parameters for evaluation. This process could in turn help the music therapist and the teacher in the analysis of the material related to the user in question, in order to design future sessions and monitor progress.
b) Developing easy-to-use evaluation software in collaboration with music therapists and teachers.
c) Implementing more multi-sensory feedback, e.g. haptic and audio-visual, according to the client’s disability, and corroborating its efficacy with appropriate experiments.
d) Exploring the potential of gestural controllers within the music therapy practice and the pedagogic field.
e) Providing music therapists with proper training regarding technological designs so that the professional can set up and manipulate the technology according to the client's needs.
Finally, it is to be expected that future research in this area will focus on the creative processes involved and emotional process - enhanced by the use of DMIs - in music therapy sessions.
ACKNOWLEDGEMENTS
We thank NICS’ staff at Unicamp, for their help during the prototype’s design and for additional help during the experiment.
REFERENCES
[1] E. Partesotti, “L'interazione nelle tecnologie musicali di realtà mista: il prototipo e-mocomu come esempio multimodale com propositi terapeutici,” Ph.D. Thesis, University of Valladolid, 2016.
[2] E. R. Miranda, M. M. Wanderley, “New digital musical instruments: control and interaction beyond the keyboard,” vol. 21. AR Editions, Inc., 2006.
[3] S. Jordà, G. Geiger, M. Alonso, M. Kaltenbrunner, "The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces", Proceedings of the 1st
international conference on Tangible and embedded interaction, pp. 139-146, ACM, 2007.
[4] A. Camurri, G. Volpe, S. Canazza, C. Canepa, A. Roda, S. Zanolla, and G. Foresti, "The ‘Stanza Logo–Motoria”: an interactive environment for learning and communication," In Proc. of
Sound and Music Computing Conference, pp. 353-360, 2010.
[5] A. Kontogeorgakopoulos, R. Wechsler, W. Keay-Bright, "Camera-Based Motion Tracking and Performing Arts for Persons with Motor Disabilities and Autism," Disability Informatics and Web
Accessibility for Motor Limitations, pp. 294-322, 2013.
[6] A. Peñalba, M. Valles, E. Partesotti, R. Castanon, and M. Sevillano, "Types of interaction in the use of MotionComposer, a device that turns movement into sound," In Proceedings of ICMEM–
The International Conference on the Multimodal Experience of Music, 2015.
[7] Varese, Un’ipotesi percorribile per le pirme classi elementari, in D. Parmigiani (Ed.), “Tecnologie per la didattica” (pp. 308-316), Milano, Italia: Apogeo Editore, 2011.
[8] Repetto, L’interattività del video: percorsi di riflessione, in D. Parmigiani (Ed.), “Tecnologie per la didattica” (pp. 88-104), Milano, Italia: Apogeo Editore, 2011.
[9] Parmigiani, Il disegno si anima. Intervista a Emanuele Luzzati, in D. Parmigiani, “Tecnologie per la didattica” (pp. 122-146), Milano, Italia: Apogeo Editore, 2011.
[10] M. Maier, B. Rubio Ballester, E. Duarte, A. Duff, P. FMJ Verschure. "Social integration of stroke patients through the multiplayer rehabilitation gaming system," In International Conference on
Serious Games, pp. 100-114. Springer International Publishing, 2014.
[11] M. da Silva Cameirão, S. Bermúdez i Badia, E. Duarte, P. FMJ Verschure. "Virtual reality based rehabilitation speeds up functional recovery of the upper extremities after stroke: a randomized controlled pilot study in the acute phase of stroke using the rehabilitation gaming system,"
Restorative neurology and neuroscience 29, no. 5, pp. 287-298, 2011.
[12] A. Camurri, B. Mazzarino, G. Volpe, P. Morasso, F. Priano, C. Re, "Application of multimedia techniques in the physical rehabilitation of Parkinson's patients," Computer Animation and
Virtual Worlds 14, no. 5, pp. 269-278, 2003.
[13] D.S. Berger, “Eurhythmics for Autism and Other Neurophysiologic Diagnoses: A Sensorimotor Music-Based Treatment Approach,” Jessica Kingsley Publishers, 2015.
[14] W.L. Magee., Indicators and Contraindications for Using Music Technology with Clinical Populations: When to Use and When not to use, in W.L. Magee (Ed.) “Music Technology in Therapeutic and Health Settings” (pp. 83-107), London, England: Jessica Kingsley Publishers, 2014.
[15] S. Benford, G. Giannachi, “Performing mixed reality,” London, England: The MIT Press, 2011. [16] A.L. Brooks, S. Hasselblad, “Creating aesthetically resonant environments for the handicapped,
elderly and rehabilitation: Sweden,” Proc. 5th Int. conference on Disability, Virtual Reality and
Associated Technologies. Oxford, England, pp. 191-198, 2004.
[17] P. Ellis, "Improving quality of life and well-being for children and the elderly through vibroacoustic sound therapy," In International Conference on Computers for Handicapped
Persons, pp. 416-422. Springer Berlin Heidelberg, 2004.
[18] D. Berger, “Music therapy, sensory integration and the autistic child,” Jessica Kingsley
Publishers, 2002.
[19] E. Partesotti, T. F. Tavares, "Color and emotion caused by auditory stimuli," In ICMC, 2014. [20] A. Noë, J. K. O’Regan, "On the brain-basis of visual consciousness: A sensorimotor account,"
Vision and mind: Selected readings in the philosophy of perception, pp. 567-598, 2002.
[21] J.J. Gibson, “The ecological approach to visual perception: classic edition,” Psychology Press, 2014.
[22] P. Dourish, “Where the action is: the foundations of embodied interaction,” MIT press, 2004. [23] M. Inzlicht, B. D. Bartholow, J. B. Hirsh, "Emotional foundations of cognitive control," Trends in
Cognitive Sciences 19, no. 3, pp. 126-132, 2015.
[24] P.J. Lang, "Behavioral treatment and bio-behavioral assessment: Computer applications," pp. 119-137, 1980.
View publication stats View publication stats