Components Integration

As explained in the description of the Audio Engine in Section 4.4.1, VORPAL fits Pure Data Patches made through the Soundtrack Creation Kit into the Object-Oriented Programming paradigm, by expressing them through the vorpal::SoundtrackEvent class in the Au-dio Engine. To do this, the Kit provides the Command Objects. They have a creation argu-ment indicating the name of the Command. When the programmer pushes a Command to a vorpal::SoundtrackEvent instance, he or she must specify the Command name and parameters. The Engine then maps the Pure Data Message to the Command Object carry-ing the same name in the associated Patch and sends the given parameters to it. For in-stance, a soundtrack Patch could have a Command Object with creation argument “play”, and when the programmers pushes the “play” Command with no parameters to the corresponding vorpal::SoundtrackEvent instance, the Command Object outputs a simple “bang” mes-sage (since no parameters were given) in the Patch, and this could be used to actually start playing something in it. Likewise, the Output Bus Object provided in the Kit goes the other way around. It receives audio signals generated inside the Patch (be it from samples or syn-thesis) and sends them back to the matching vorpal::SoundtrackEvent in the Engine.

Then, when the Audio Engine is serviced during the Game Loop, all audio signal captured by instances of vorpal::SoundtrackEventare properly transferred to the sound card for play-back. Since Pure Data works in Ticks of 64 samples, the Engine can control how much signal is processed every frame to guarantee that the soundtrack is played inside real-time constraints.

Again, note that all of these concepts explained here are detailed in Chapter5, where we discuss the implementation and usage of each of the architecture parts.

Chapter 5

Implementation

Chapter 4 defined the architecture of the VORPAL middleware and the soundtrack format it uses based on the System Requirements raised and the concepts explained in previous chapters.

The next step is to delve into the implementation of the core aspects of the middleware, which is the role of this chapter. Here, we discusshow we achieve the proposed architecture and which technologies, data structures, algorithms, and conventions we utilized to do so. We also address the resulting workflow of our tool, that is, how users should proceed to actually make use of it. As stated in Section4.1, we developed the system following agile principles. One of the consequences of this is that, before arriving on the current design of the middleware, we had already iterated over it a number of times. Here, we omit all the intermediate steps the development process took, but we will briefly discuss one of the prototypes we built, since it was the first to present relevant results in terms of real-time soundtrack support.

From the middleware specifications in Sections 4.3 and 4.4, we know that VORPAL is im-plemented using a mixture of C++ and Pure Data. Pure Data has its own means of producing sound playback, but the same could not be said of the parts written inC++– namely, the VOR-PAL Audio Engine. For that purpose, we chose to use OpenAL, which is a cross-platform free software programming library for sound playback. We have already shown some of it in the code examples of Section 3.3.4. Besides the reasons we just mentioned, we also decided to use Ope-nALbecause it supports 3D spatialized audio out-of-the-box, which brings us closer to meeting System Requirement 7 (“3D Audio Spatialization”). Since we preferred to avoid too many de-pendencies on our system, there are no other librariesVORPAL needs aside fromOpenAL and libpd. It does require support for theC++11standard, since most modern compilers provide it and it relieves us of further dependencies for multi-threading and time synchronization features.

Outside of the system itself, it is worth noting that we use Git for source code version control (hosted at https://github.com/vorpal-project) and CMake¹ as our building framework. These are dependencies only developers need, not the end users. Besides, allC++ code written for the middleware, including the examples and excerpts found in this thesis, follow a slight variation of theGoogleC++ Style Guide².

Section 5.1 presents the first real-time soundtrack prototype we developed using a more

1 https://cmake.org/

2 https://google.github.io/styleguide/cppguide.html

81

unstable integration withPure Data. Next, Sections5.2and5.3address the implementations of theVORPALAudio Engine and Soundtrack Creation Kit, respectively. In Section5.4, we describe in details: the workflow programmers and sound designers must follow to use our middleware;

game engine support for VORPAL, demonstrating how it works by exposing implementations of this support made for two free software engines; and two usage examples that show theVORPAL middleware in action.

5.1 Prototype

In the early process of designing VORPAL, when we were pondering the choice of soundtrack format, we decided to develop a quick prototype to validate whether Pure Data could work as a language that expresses real-time soundtracks. To avoid spending time in the development of a brand new game for the sole purpose of this prototype, we searched for an open source title we could modify and insert the new soundtrack into. Our middleware did not have a single line of code at this moment, and we wanted to be able to show people what our research was about without having to actually implement the whole system beforehand. Since our methodology involved gathering feedback from musicians and sound designers, having a demonstration of what we meant by “real-time soundtrack in games” was paramount.

For the game itself, we decided on modifying Mari0 (Stabyourself, 2012)³, distributed under theCreative Commons BY-NC-SA 3.0⁴ license. It is a parody game that reconstructs the classic Super Mario Bros.(Nintendo, 1985) but mixes in the portal mechanics fromPortal (Valve, 2007).

The fact that it resembles a relatively famous game helps get the point across when showcasing the prototype, but that was not the only reason that made us chooseMari0. The other reason is that it was developed using theLÖVE framework, which we have extensive experience working with due to previous projects we participated in atUSPGameDev⁵, a student group from University of São Paulo that researches game development. That saved us the time of learning a new development tool just to change a few dozen lines of code to add the new soundtrack. Figure5.1 shows a screen capture of the game.

The next step was to determine how we would embed a Pure Data Patch into the game. We were aware oflibpd, but since we wanted the fastest way to achieve a real-time time soundtrack, we took a different approach. We wanted an alternative where we did not have to implement sound playback, since that required time and more software dependencies (in fact, we did add theOpenAL dependency for this part later). Thus, inspired by one of the features from FMOD where the sound designer could tweak the soundtrack while playing the game at the same time, we chose to play everything from thePure Data application itself, while receiving real-time data from the game through a direct socket connection. Since both programs would be running on the same machine, we could assume there would be no issues such as packet loss, significant delays, etc. But, to do that, we needed to define a communication protocol. Again, as to not waste too much time on a relatively lesser detail of a prototype, we searched for something we could use out-of-the-box. The sufficiently good alternative we found was theOpen Sound Control

3 http://stabyourself.net/mari0/

4 https://creativecommons.org/licenses/by-nc-sa/3.0/

5 https://uspgamedev.org/

5.1 PROTOTYPE 83

Figure 5.1: An in-game screenshot ofMari0 (Stabyourself, 2012) [Sta12]. In this parody game, Mario can shoot portals and use them to traverse the classic challenges of the original title fromNintendo.

(OSC) protocol⁶, for which we promptly arrived at an accessible and lightweight implementation, liblo⁷. On thePure Data side, we ended up temporarily moving toPure Data Extended (refer to Section3.4.5) to find Externals that could decode OSC messages. After that, all we had to do was write a Lua⁸ binding for liblo (another task we already had experience with [Miz]) and use it in Mari0 to send messages containing relevant real-time data from gameplay, which, in turn, we used to control the soundtrack insidePure Data.

For the actual soundtrack, we focused on the first stage of the game and started by sim-ply playing a shortened version of its original theme, “Overworld Theme” by Koji Kondo. We implemented it inPure Data using simple synthesized timbres and a custom sequencer. Then, we added two real-time effects to it. The first was inspired by the battle theme variations from Faster Than Light: the percussion track of the theme was removed, and we added a new one with hits that played according to the quantity of enemies nearby. The more enemies there were around Mario, the more hits the percussion track would play. There was a total of four levels of percussion intensity. Notice how we would have needed at least five different sound samples (the melody plus the four percussion variations) if we had not implemented it through a pro-cedural sequencer. The other real-time effect we added to the soundtrack was a modulation in the melody. When the player managed to grab theFire Flower item in the game (which makes Mario much stronger than normal), the theme key would rise a whole tone. If he got hurt and lost theFire Flower bonus, the melody would go back to normal. Again, this would have been much harder to do with a sample-based implementation without incurring in a time-stretch of the sample (or using a second sample). Thus, both these effects served the purpose of not only illustrating what we envisioned as a real-time soundtrack, but also explicitly highlighting the advantages of procedural audio over sample-based audio.

The first “released” version of the prototype was finished around the middle of 2015⁹. Later, after theVORPALmiddleware achieved its own initial release, we ported most of the prototype to the new system, dropping OSC and liblo in the process. The revised prototype was used

6 http://opensoundcontrol.org/

7 http://liblo.sourceforge.net/

8 Lua is a Brazillian scripting language used worldwide by the community of game developers, being the language of choice of many game engines, such asLÖVE.

9 https://github.com/vorpal-project/mari0/releases/tag/prototype/zero

to verify the other side of the coin: we composed our own arrangement of “Overworld Theme”

using the LMMS tool and inserted it into the game using a sample-based approach. Due to the restrictions of this choice, we reduced the real-time effects in the new version of the soundtrack to a variable mixing (System Requirement 5) effect. Now, the percussion track merely increases in volume when enemies are close, and the bass track increases in volume according to Mario’s size (that is, how many “power ups” he currently has in effect).

The original prototype served as a solid indication of the possibilities brought by procedural audio and confirmed that usingPure Data as a soundtrack creation interface was a viable option.

It was an important first step in our research on top of which we based the current middleware implementation. At the same time, we learned that using Pure Data Extended could become a burden to the sound designer, since it came with a considerable number of dependencies and had actually not been updated for a while. The agile approach allowed us to detect this issue and move back toPure Data “Vanilla”, at the same time that it showed us that supporting real-time soundtracks in games was an achievable and fruitful goal.

The following sections present a more in-depth description of the implementation details and usage protocols of the middleware. Readers more interested in a broader view of this work may skip directly to Chapter6.