Residency Overview

A brief overview of the residency is presented, primarily based on the field notes. In the organisation of the field notes and of the analysis of the interview of the choreographer, a clear distinction emerged: between the first two sessions, as a more explorative and design focus moment; and the following two, focused on rehearsing. For this reason, the session descriptions presented were clustered in two phases: co-creation/design phase and fine-tuning/rehearsal phase. The design of the interactive sonic artifact was mostly concluded in the first phase. The structure of the activities was not predefined, rather it followed the needs emerging in each session.

7.2.4.1 Co-creation/Design Phase

In theco-creation/design phase, the three main instruments that corresponded to the three main scenes in the dance piece, and the dance piece itself were developed in parallel.

Figure 7.4: The Captury computer alongside the computer running Unity3D for avatar rendering (The screen is showing the avatar on OBS)

The first instrumentwas characterised by acrackling soundwhose volume and density was mapped to the movement amount of the dancer. Such an amount was calculated using a computer vision approach. It was computed by subtracting two sequential frames of positional data of the avatars in Unity, based on the information coming from the motion capture system. The crackling sound was implemented using a set of variable time delays, reading in different points at variable speed a percussive sound stored in a buffer, resulting in a granulation of the original sound. This instrument was used in the first scene of the dance piece, in which the choreography consisted of an open task: D1 was required to improvise alternating movements with moments of stillness; the second dancer (D2) was required to imitate the first dancer’s static pose. Only movements of D1 were used for the creation of sound, originating alternate moments of sound and silence.

The two dancers are physically distant in this first scene, as they are in different parts of the stage.

The second instrument was a percussive sound that was triggered by dancers’ hand movements in a gesture that imitates hand clapping. Initially the percussive sound was triggered when the dancers’ hands were closer than 10 centimetres to the floor. However, the movement of this interaction highly impacted the physicality of the dancers and the dramaturgy of the choreography, imposing an action that interrupts the fluidity of other movements. To reach the floor, the majority of the body is involved. Therefore, their interaction was subsequently improved by changing it to a hand-clapping gesture that

would act as a trigger. These changes were based on the observation that the choreogra-pher made by seeing the dancers interacting with the system, and the new interaction modality emerged from discussion between the choreographer and the sound designer.

Delays and reverberation have also been added to these sounds for aesthetic improve-ment. The sounds were similar for the two dancers, but the actual pitch and timbre of the two sounds were different, so each performer had her own sound. This was used in the second scene, where the choreographic task required the dancers to maintain a continuous movement, while sharing the same part of the stage. The two dancers are closer compared to the first scene.

The third instrument was a cello drone. The volume of the cello sound was linearly mapped to the average movement amount of the two dancers. The movement amount was calculated by using the same approach developed for the crackling sound. This instrument was used in the third scene. In this scene, the choreographic task was to explore the body of the other dancer, by grazing it with any body part while maintaining a continuous movement. As a consequence, the two dancers shared an intimate space, mutually influencing their movements.

In the first phase, the choreographer tried the hand-clapping interaction for the per-cussion herself. She also tested the choreographic tasks with the dancers, and modified them according to the feedback.

Overall, dramaturgically the piece progressively goes from physical distance of the two dancers to intimate connection in a close dance that shares the same space. The sound design accompanies this progression by transitioning from noisy spectra and irregular envelopes, to harmonic sounds with a smooth envelope. From this element we can already observe how the dramaturgy of the piece, which has specific choreographic tasks, was embedded in the sounds.

7.2.4.2 Fine-tuning/Rehearsing Phase

After session 2, the main design process was concluded and the features of the final artifact were defined, as well as the three instruments (crackling sound,percussion, and cello). In the remaining sessions, the sonic features were fine-tuned: the delay time in the percussion was calibrated and the timbre of the cello was improved (a second sound, one octave lower, was added to the original cello). In this phase, the calibration of the motion capture system was also fine tuned. This activity included a specific work and tasks involving the dancers. In order to test the threshold of the percussion and the crackling sound, the sound designer asked the dancers to perform the specific corresponding ges-tures and adapted the thresholds accordingly. Apart from these minor sound details and system calibration, the last sessions focused mainly on the development and rehearsal of the choreography.

The sound designer also explained to the dancers the functioning of both the motion capture system and the sound computing engine, showing where the motion capture was

more efficient, and identifying movements that created issues with it. During this process, the dancers became more aware of the motion capture area and of how their movement influenced the various sounds. This process produced a very positive effect, the dancers started to dance with more confidence in the system.