5
I V R U X - A n a l y z i n g X R I m m e r s i v e N a r r a t i v e s
This chapter is focused on the construction of severalartifactsexploring differentXR systems. Following aResearch through Design (RtD)approach, the construction of narrativeartifactsled to a parallel construction ofartifactsused to analyze user experience,IVRUX. The knowledge gained in the constuction of theseartifacts, and aligned with theoretical background and related works (PartI), was applied to the artifactsdescribed in chapters6-8.a
aConspicuous parts of the text below have appeared in co-authored publications [Bal+16;Bal+17;BNN17;Dio+16;Dio+17a;Dio+17b;Dio+18]
5 . 1 . F R AG M E N T S O F L AU R A
for tourists. FoLwas developed as part of two EU-funded projects (Future Fabulators &
Beanstalk) and was intended to raise awareness ofparticipantsregarding the natural and cultural heritage of Madeira Island. This was intended through two distinct main media channels (see fig. 5.1):
• FoL Location Aware Multimedia Stories (LAMS)- a location-aware multimedia story that letsparticipantsexplore a route in Funchal, Madeira Island. Asparticipants physically explored several locations in the historic parish of São Pedro, they would be able to unlock videos, audios, and interactive scenes, about the life of a fictional character. The story, mixing fictional events with real-life events and characters of the 18th century, follows Laura Silva, an orphan girl, learning the medicinal powers of the local endemic forest.
• Há-vita2 - a hypermedia platform connecting the fictional facts of the FoLstory world to scientific facts about Madeira Island’s natural heritage through interviews with locals and experts.
FoLresearch followed a Research through Design (RtD) approach "where designers produce novel integrations of HCI research in an attempt to make the right thing: a product that transforms the world from its current state to a preferred state" [ZFE07].
This involves a balance between making and research with several cycles of iteration; in the end, the product prototype itself is the research contribution and an instrument of design knowledge [ZFE07].
One way thatRtDis reflected inFoLis the iterative and incremental development of one of the story points,The Old Pharmacy, a virtual environment for exploration. These iterations not only explored differentXRplatforms and storytelling approaches but were motivated by different research questions on how this story point could be included in LAMS. For example, Vosmeer and Schouten [VS14] identify different experience styles when consuming media from a passive style ("lean back") of someone who is watching a movie to the active style ("lean forward") of someone playing a videogame. 360º video, CVRorVRexperiences are a "lean in" style, since the user is not completely "lean back"
nor completely "lean forward" [VS14]. In initial iterations ofFoL LAMS, in theThe Old Pharmacyscene (see fig.5.2), Laura is working on a healing infusion when a local gentle-man, Adam, interrupts her with an urgent request; the experience ends in a cliffhanger as a landslide falls upon our characters. The user is merely spectating the story, although immersively through the chosen technologies; this corresponds to a "lean in" style. In later iterations ofFoL LAMS, in theThe Old Pharmacyscene (see fig. 5.3), the user, em-bodying the main character, is asked by another virtual character (an old woman) to make
the creation of multimedia content and application. My contribution to this project was threefold: (1) as part of the team developing the overall concept and user experience of the narrative; (2) as a programmer developing multiple iterations of anXRscene; (3) as an author/co-author in the dissemination of scientific articles emerging from this project.
2https://havita.arditi.pt/
Figure 5.1: Media channels in theFoLTransmedia. From left to right: below theFoLlogo, two screens of theFoL LAMSmobile application; below theHá-vitalogo, a screenshot of the web based hypermedia platform
a medicinal drink by gathering four objects, spread around the virtual environment. To fulfill this task, the user must navigate and orient themselves in the virtual world and examine the objects within it; this corresponds to a "lean forward" style.
The following subsections describe theThe Old Pharmacy scene at different points of the development process. When existing, summarized pilot/evaluation studies are described in the appendixes.
5.1.1 The Old Pharmacyas mobileVR- Influence of Location and Medium applied to mobileVRStorytelling
The increasing availability, power, and portability of mobile computing, has allowed for exploration in locative media. LAMS, a sub-genre within the wider field of locative media, explore this by "telling stories that unfold in real space" through the use of location-aware mobile technologies [Kar+12]. The assumption underlying all LAMS is that the overlaying of virtual information (the narrative) over the existing physical space increases the level of immersion in the narrative. For FoL, we were motivated to understand the affordances and barriers of mobile VR platforms in LAMS. Poppe et al. [PGS17]
defined mobileVRas a "system that creates the illusion of participation in a simulated environment, rather than external observation of such an environment, by replacing real
5 . 1 . F R AG M E N T S O F L AU R A
sensory signals that the user perceives with simulated sensory signals through the use of portable technology".
A prototype ofThe Old Pharmacy scene was implemented using the Unity 5 game engine3, and the Google Cardboard Unity plugin4forVRsupport. Google Cardboard is a low-cost cardboard headset, where the display is a compatible Android device with minimum requirements (gyroscope and accelerometer). For smartphones, a stereoscopic view is shown on the screen matching the lenses of the headset, and the device is inserted into the foldable frame. The Google Cardboard Unity is also compatible with tablets matching minimum requirements; in this case, the screen is not stereoscopic.
As previously mentioned, this prototype corresponds to the first iteration ofThe Old Pharmacy scene, where the main character of Laura (see fig. 5.2(b)) is working in the pharmacy (see fig. 5.2(a)) and is interrupted by a secondary character, Adam (see fig.
5.2(c)). This pharmacy is based on an existing location, a pharmacy in the 19th century that now functions as a coffee shop (named "Old Pharmacy").
A pilot evaluation of this prototype, described in appendix sectionB.1, was carried out with two independent variables:location(existence and absence of links between the test location and the narrative location) andmedium(support platform used for delivering the experience; tablet andHMD).
This pilot was useful in determining the research direction ofFoL. By focusing on one location, we were able to ponder the benefits and disadvantages of using immersive technologies to tell stories with links to the real environment. This knowledge permeated through the project with the inclusion of the "Old Pharmacy" asHMDexperience in the final prototype, and with linking the physical locations, not only in the interactive story points but also in other media types such as videos.
5.1.2 The Old PharmacyinMR- Navigation Styles in Mixed Reality Entertainment
Technological advances in mobile systems have allowed improved graphic rendering and runtime intensive environment scanning to be possible in devices across Milgram and Kishino’s taxonomy ofMRvisual displays [MK94]. An example of this was Google’s Project Tango (2014-2018)5, anMRtablet. Using depth perception information and com-puter vision algorithms, Project Tango reconstructed mathematical models of the real world over time. The system estimates the movement of the device in relation to the real world, allowing for motion tracking (navigation and orientation) of the user holding the device. Abstracting from the technology behind it, this type of system showcased the potential of using knowledge of the surrounding world as input. While Project Tango
3https://unity3d.com/pt/unity/whats-new/unity-5.0 4https://arvr.google.com/cardboard/
5https://developers.google.com/tango
(a)3DEnvironment ofThe Old Pharmacy, from theparticipant’s position
(b) Laura, the main character (c) Adam, the secondary character
Figure 5.2: First iteration ofThe Old Pharmacy: while two characters have a dialogue about recent events, theparticipantis located in the middle of the room
is currently deprecated, its development was essential to the creation of the Google AR-Core development kit6, which now allows depth-perception, area learning and motion-tracking over a large set of Android devices. Forecasting this interest,FoLwas used as a use case of how these technologies could be used. A second iteration of the The Old Pharmacy(see fig.5.3) was built using the Unity 5 game engine7for the Tango platform.
In this iteration, the user is asked by a virtual character to prepare a Madeiran medic-inal drink (poncha) by gathering four objects distributed around the virtual pharmacy (see fig. 5.4). To accomplish this task, the user must navigate and orient themselves in the virtual world and examine the objects within it. The pharmacy is a visually complex environment with many objects distributed around the space both horizontally and verti-cally (e.g., on furniture). The search task requires the user to move around and explore different viewpoints, with a total of 15 selectable objects. When a user is within reaching distance of one of these, the object is highlighted visually with a glow effect, and a user
6https://developers.google.com/ar
7https://unity3d.com/pt/unity/whats-new/unity-5.0
5 . 1 . F R AG M E N T S O F L AU R A
(a) Google Tango (b) Selectable tutorial object with on-screen in-structions
(c) 3Dexplorable environment ofThe Old Pharmacy
(d) Selectable objects according to proximity (e) Correct selected object with visual feedback
Figure 5.3: Second iteration ofThe Old Pharmacy: participantsare able to move around the environment and collect objects to make a local drink
D
C
B T
A
Figure 5.4: The Old Pharmacyroom layout: white dots are selectable objects, the yellow dot (T) is a tutorial object (book), and green objects are selectable objects that need to be collected (A to D, respectively, bee honey, mexelote, lemons, and rum)
can select it with an on-screen tap. An audio dialogue between the customer and the main characters elaborates on the properties of the object. When a correct object (in the set of ingredients needed to make the drink) is selected, the user receives encouraging on-screen and auditory feedback.
This prototype supported different navigation styles (see fig.5.5) covering the reality-virtuality continuum: Screen(virtual based),Hybrid(involving on-screen and in-world) andSpatial(in-world). A evaluation of this prototype, described in appendix sectionB.2, was carried out with one independent variable,Navigation Style (NS).
As a summary of this study, the exploration task (matching a ("lean forward") style [VS14]) was well received byparticipants; this was particularly relevant forparticipants who experienced theSpatialnavigation style. In this case, the natural navigation style of walking helped to create immersion in the task and the story world. This follows a design concern ofReality-Based Interaction(RBI)[Jac+08], where the participant must be aware of their surroundings and their body to be able to navigate. Ultimately, due to safety concerns related to awareness of surroundings, theHybridnavigation style was adopted for the final implementation ofFoL, as it still allowed a degree of natural reality-based interaction with rotation of the body, but minimized risks related to real world context of use.
5.1.3 The Old PharmacyasVR- Remediating a locative tour to Virtual Reality
An advantage of locative media is that it allows for "telling stories that unfold in
5 . 1 . F R AG M E N T S O F L AU R A
(a)Screen (b) Hybrid (c)Spatial
Figure 5.5: Navigation styles:Screen,HybridandSpatial. Green represents navigation ac-tions and red represents looking acac-tions. Objects are selectable by touch in all condiac-tions
real space" through the use of location-aware mobile technologies [Kar+12]. However, the strong link between FoLand location was a barrier when needing to showcase the prototype in exhibitions and demo sessions, where the ability to travel was not feasi-ble. Building on the concept of remediation, representation of one medium in another medium [BG99], we reconceptualizedFoLfor when physical travel was not possible.
One remediation for physical installations involved the use of tangible objects in a physical installation (see fig. 5.6). The original mobile application was designed to trigger the content upon arrival at specific locations, by using a combination of GPS coordinates and Bluetooth beacons. In the physical installation, tokens created in a laser cutter and depicting the real buildings that they were replacing, were used to encase the Bluetooth beacons; the Bluetooth detection range was also changed to allow them to be scanned by the mobile device. A map of the city center was used as the base for the physical tokens.
Another alternative for physical installations was the use ofVR. The incorporation ofVRallows for aggressive remediation, as the use of the new medium absorbs the old medium and creates a new experience. This is supported byVR’s power as a metamedium,
Figure 5.6: Fragments of Laura installation atICIDS2017
a medium that can encapsulate all others [BL95, p. 16].
MappingFragments, building on the previous installation’s use of maps, uses a VR HMD (Samsung Galaxy 6 with Gear VR SM-R322) to immerse theparticipant in the experience by placing them in a multilayered topographical map. This map is intended to recreate the actual locations where the locative tour takes place. Participants can explore the map in two levels:
• The first, "Room level" (see fig. 5.7(a)), allows theparticipantto walk around and over the map, getting a bird’s eye view of the landscape. Representative icons of the storypoints (see fig. 5.7(c)) and other selectable content were interactable using the HMDcontroller’s raycast and trigger button (see fig.5.7(d)).
• The second level, "Map Level" (see fig. 5.7(b)), places theparticipantin the map itself, being able to virtually walk by using the touchpad of the HMDcontroller or by automatic pathfinding by raycasting and selecting a location in the map. An adaptive Field of View (FoV)(see fig. 5.7(e)) was controlled using the velocity of the movement, as done by Fernandes and Feiner [FF16]. This remediation ofFoL includes a third iteration of theThe Old Pharmacyscene (see fig.5.7(d)), using the previously described interactions for selecting and moving in the virtual environ-ment.
Although this iteration was never evaluated formally, the experience of building it was beneficial to the development ofIVRUX(described in the next section), and inspired work described in chapter7and8.
5 . 1 . F R AG M E N T S O F L AU R A
(a) "Room Level"
(b) "Map Level"
(c) Representative icons (d) Object Selection (e) AdaptiveFoV
Figure 5.7: MappingFragments, aVRremediation of theFoLlocative tour