Virtual Reality Framework for
Collaborative Applications
Vasco Magalhães Pereira
Integrated Master in Informatics and Computing Engineering
Supervisor: Rui Rodrigues Co-Supervisor: Rui Nóbrega Co-Supervisor: Teresa Matos
Applications
Vasco Magalhães Pereira
Integrated Master in Informatics and Computing Engineering
Virtual Reality aims at immersing the users into a 360oenvironment that can either be simulated or recorded. This can be done using Head-Mounted Displays that the user wears, or by projecting images of the environments into the walls of a room. VR is part of a mixed-reality continuum that describes the different levels of reality each system offers. Augmented Reality is also part of the same continuum, much closer to a real environment than VR.
Each mixed-reality technology has its own purposes which, when used together with other devices in a single collaborative online application, could allow for different design paths that would not be possible using only one mixed-reality device.
Collaborative applications benefit from multiple users working together to accomplish a com-mon goal. Interface design for collaborative applications should promote user cooperation. Some applications apply collaborative mechanisms to provide users with the needed tools to work to-gether.
This dissertation aims at developing a framework for the development of collaborative VR applications. An additional objective is to allow users from different devices, such as AR, to join VR users in the same shared environment. As such, this project sets out to document, analyze and solve most of these challenges in a collaborative environment, while, at the same time, maintaining a synchronized state of the environment across all users, making it possible for them to change and see the changes of others on the world around them.
Using the implemented tools, a prototype was developed for testing purposes. The tests were conducted in pairs of users. The pair would go through an initial phase that would help them get familiarized with the controls. Afterwards, they followed a set of instructions which should have them interact using the various tools.
The resulting framework allows for the development of virtual worlds where multiple users can interact in real-time. The framework, through the use of different interface methodologies, facilitates the development of these types of applications. The prototype that was developed pro-vided some insights into how to make these mechanisms work for VR devices. We conclude that the tools developed highly increase user immersion in a shared environment and helped them collaborate with each other.
Realidade Virtual ou VR tem como objetivo imergir um utilizador num ambiente 360o, que pode ser simulado ou pré-gravado, através da utilização de Head-Mounted Displays ou da projeção de imagens desse ambiente em paredes de uma divisão. VR, deste modo, faz parte do continuum referente a realidade mista (mixed-reality), que descreve os diferentes níveis de realidade que cada sistema oferece. Realidade Aumentada faz, também, parte deste continuum, sendo um tipo de realidade mais próximo de um ambiente real e verdadeiro do que VR.
Cada tecnologia relativa a mixed-reality tem o seu propósito, o qual quando usado em conjunto com outros dispositivos numa única aplicação colaborativa online permite diferentes caminhos de design que não seriam possíveis de ser obtidos usando apenas um dispositivo de mixed-reality.
Aplicações colaborative beneficiam ao ter multiplos utilizadores a interagir juntos para atingir um objectivo em comum. O design de interfaces para aplicações colaborativas deve promover a cooperação entre utilizadores. Algumas aplicações implementam ferramentas de colaboração para ajudar os utilizadores a trabalhar juntos.
O objectivo desta dissertação é o desenvolvimento de uma framework para o desenvolvimento de aplicações colaborativas em VR. Um outro objectivo é permitir que utilizadores de outros dis-positivos, tal como AR, se juntem aos utilizadores de VR num ambiente partilhado. Desta forma, este projeto visa documentar, analisar e solucionar a maioria dos desafios relativos a ambientes co-laborativos, e, ao mesmo tempo, manter um estado sincronizado do ambiente entre todos os seus utilizadores, fazendo com que seja possível que estes mudem e vejam as mudanças que outros utilizadores fizeram no mundo que os rodeia.
Usando as ferramentas implementadas, um protótipo foi desenvolvido para fins de teste. Os testes foram realizados em pares de utilizadores. O par passou por uma fase inicial que os ajuda a familiarizarem-se com os controlos. Depois disso, eles seguem uma série de instruções que lhes pede que usem as ferramentas para interagirem.
A framework implementada permitirá o desenvolvimento de mundos virtuais onde múltiplos utilizadores podem interagir em tempo real. As diferentes ferramentas implementadas facilitarão o desenvolvimento deste tipos de aplicaticações. O protótipo desenvolvido forneceu diferentes prespetivas sobre as ferramentas colaborativas que funcionam melhor para os dispositivos de VR. Concluímos que a ferramentas desenvolvidas aumentam a imersão dos utlizadores em ambientes colaborativos e ajudam os utlizadores a colaborar uns com os outros.
Firstly, I would like to thank the Faculty of Engineering of the University of Porto, especially to the teachers with whom I had the privilege of knowing and acquiring different knowledge.
A special thanks to my Supervisor, Rui Rodrigues, and my Co-Supervisors, Rui Nóbrega and Teresa Matos, for the constant help and availability. Especially on those final days when we were all so tired but they kept pushing anyway.
I would also like to thank my family for all the support given. A big “Thank You” for being there through all the stress and mood swings.
I would also like to thank my friends and classmates for their help and support. Especially to all the people that shared train rides with me throughout these 5 years. Because of you missed or delayed trains were not so bad.
1 Introduction 1
1.1 Context and Motivation . . . 1
1.2 Research Questions . . . 2 1.3 Objectives . . . 3 1.4 Contributions . . . 3 1.5 Document Structure . . . 4 2 Literature Review 5 2.1 Virtual Reality . . . 5
2.1.1 VR Games, Applications and Tools . . . 6
2.1.2 VR Interfaces . . . 13
2.1.3 Discussion . . . 19
2.2 Interchangeable Reality . . . 19
2.2.1 Augmented Reality . . . 19
2.2.2 Reality-Virtuality Continuum . . . 21
2.2.3 Interchangeable Reality applications . . . 22
2.2.4 Discussion . . . 22
2.3 Collaborative Virtual Reality . . . 23
2.3.1 Collaboration mechanisms . . . 24
2.3.2 Collaborative Virtual Reality applications . . . 25
2.3.3 Collaborative IR Tools . . . 26
2.3.4 Discussion . . . 28
2.4 Summary . . . 28
3 Proposed Architecture for Collaborative Virtual Reality Framework 31 3.1 Requirements . . . 32 3.1.1 VR Interactions . . . 33 3.1.2 VR Collaboration . . . 34 3.1.3 Tool Selection . . . 36 3.1.4 Networking . . . 36 3.2 Functionalities . . . 37 3.3 Solution Architecture . . . 38 3.3.1 System Architecture . . . 39 3.3.2 Interaction Diagram . . . 40 3.3.3 Input Hardware . . . 41 3.4 Summary . . . 43
4 Development of Collaborative Virtual Reality Framework 45
4.1 Tools and Technologies . . . 45
4.2 VR Interaction Techniques . . . 47
4.2.1 Object Manipulation . . . 47
4.2.2 Teleportation and Actions at a Distance . . . 50
4.2.3 Tools . . . 50 4.3 VR Collaboration Techniques . . . 52 4.3.1 Avatars . . . 52 4.3.2 Highlights . . . 55 4.3.3 Previews . . . 57 4.4 VR and AR Integration . . . 59 4.5 Hardware Solution . . . 60 4.5.1 Development Environment . . . 61 4.5.2 Input Solution . . . 61 4.6 Summary . . . 63 5 Evaluation 65 5.1 Validation Protocol . . . 65 5.2 Test Sessions . . . 67 5.3 Gathered Data . . . 68 5.3.1 Demographics Questionnaire . . . 69 5.3.2 Usability Questionnaire . . . 69 5.3.3 Other Questionnaires . . . 69
5.3.4 Notes and Measurements . . . 70
5.4 User Studies . . . 70
5.4.1 Description . . . 70
5.4.2 Stage 1 Test . . . 70
5.4.3 Stage 2 Test . . . 76
5.4.4 System Usability Scale . . . 81
5.5 Discussion . . . 82 6 Conclusion 85 6.1 Future Work . . . 86 References 89 A User Questionnaire 95 B Consent Form 99
2.1 Oculus Rift . . . 6
2.2 HTC Vive . . . 6
2.3 Playstation VR . . . 6
2.4 Samsung’s GearVR headset . . . 7
2.5 LumiSpace shown through Unity3D’s editor . . . 7
2.6 Wakeboard interactions . . . 8
2.7 Setup for the game Rift-a-Bike . . . 9
2.8 Tilt Brush . . . 10
2.9 Unbound . . . 11
2.10 Unity VR Editor . . . 12
2.11 Unreal Engine VR Editor . . . 12
2.12 Head Lock main steps . . . 14
2.13 HTC Vive and Oculus Rift’s motion controllers . . . 15
2.14 Leap Motion . . . 17
2.15 Haptic Gloves example . . . 18
2.16 Accessibility: Movement and Rotation . . . 18
2.17 Accessibility: Controller Position Shift . . . 18
2.18 Accessibility: Virtual Controller . . . 18
2.19 AR showing information through phone . . . 20
2.20 AR showing information through tablet . . . 20
2.21 A picture of the application Pokemon Go . . . 21
2.22 Mixed-Reality Continuum . . . 21
2.23 Lag compensation algorithm . . . 24
2.24 Interactions in “The MagicBook” . . . 27
2.25 “CoVAR”’s architecture . . . 28
3.1 VR Multiple Users . . . 32
3.2 System diagram . . . 39
3.3 Interaction diagram . . . 40
3.4 Motion controller buttons . . . 42
4.1 Pivot Point . . . 48
4.2 Scaling of a lamp . . . 48
4.3 Invalid drop . . . 49
4.4 Teleport . . . 51
4.5 Hand color depending on the selected tool . . . 52
4.6 Tool selection sequence . . . 53
4.8 Occluded user . . . 55
4.9 Final avatar design . . . 56
4.10 Send Highlight . . . 57 4.11 Highlight Sequence . . . 58 4.12 Highlight Effects . . . 59 4.13 Preview object . . . 60 4.14 Accepting Preview . . . 61 4.15 VR and AR Integration . . . 62 4.16 IR Test . . . 63 5.1 Test environment . . . 66 5.2 Test session . . . 68
5.3 Histogram object manipulation . . . 72
5.4 Histogram locomotion . . . 73
5.5 Histogram tool selection . . . 74
5.6 Histogram tool selection . . . 75
5.7 Histogram user visualization . . . 78
5.8 Histogram highlight . . . 79
2.1 Virtual Reality applications comparison . . . 29
2.2 Collaborative IR applications comparison . . . 29
3.1 PAINTER Actions Taxonomy . . . 37
3.2 Tool Selection Taxonomy . . . 38
3.3 Networking Taxonomy . . . 38
5.1 Gender distribution. . . 71
5.2 Age distribution . . . 71
5.3 Experience of the users with VR. . . 71
5.4 Experience of the users with video-games. . . 71
5.5 Stage 1, Object Manipulation . . . 72
5.6 Stage 1, Locomotion . . . 73
5.7 Stage 1, Tool Selection . . . 74
5.8 Stage 1, Tool Selection By Device . . . 75
5.9 Stage 2, User Visualization . . . 78
5.11 Measurements of the time in seconds it took users to find the couch. . . 79
5.10 Stage 2, Highlight . . . 79
5.12 Stage 2, Preview . . . 80
AR Augmented Reality HMD Head-Mounted Display IAR Industrial Augmented Reality IR Interchangeable Reality MAR Mobile Augmented Reality MC Motion controller
MR Mixed Reality
PAINTER Procedurally Assisted INTErchangeable Reality VE Virtual Environment
Introduction
Many of our daily tasks are done through collaboration with other people. With the growth of technology and the virtualization of much of the work we do, the demand for collaborative appli-cations has never been higher. Collaborative appliappli-cations use a series of techniques to enhance the cooperation between users. We believe that these aspects can be improved through Virtual Reality (VR). VR technology has been growing lately. VR places users in a virtual environment through specialized hardware, e.g. headsets that track the head’s orientation. New devices tend to close the gap between real world actions and their representation in the virtual world. This increases immersion has it gives users the feeling of being present in the virtual environment, which can be beneficial in the context of collaborative applications.
This dissertation studies ways of increasing collaboration in VR. The feeling of a shared space is one of the most important aspects that are explored, as well as communication between users.
1.1
Context and Motivation
Collaboration is involved in any application that requires users to work together towards a common goal. But collaboration needs to be embedded into the design of such applications. Collaborative applications should increase users’ awareness of various aspects of the environ-ment [TNG+16]. These involve being able to visualize other users, as well as their interactions with the virtual world. If the application requires explicit communication between users, some tools need to be provided depending on the type of interactions expected.
Introducing collaboration is not a straightforward process. It requires full knowledge of the expected interactions between users, as well as a documented set of tools and actions that should be visible across everyone [RB06]. These tools highly depend on the type of application and platform it is being developed on. That is partly due to the difference in interaction devices and their limitations.
The interface design of a VR application is different from the design of that same application in desktops. In VR, the point-of-view of users is tied to their real movements, and, most likely, they will be using motion controllers to interact with the application. On the other hand, in desktop applications, users can freely move the camera, if the application allows for that. While VR is more dependant on physical movements, desktops are not, and that completely changes the interface design.
When designing collaboration mechanics to VR that same logic applies. The tools should be inspired in already existing mechanisms. But there is a lack of documentation and research around collaboration in VR environments. That forces developers to find new ways of integrating collaboration features into the respective applications. A specialized tool for VR might be able to cover some of the most relevant collaboration mechanisms.
To research and study some of those mechanisms might allow future developers to focus on interface designs for more specific applications. Also, a tool that implements collaborative features that are easy to modify and iterate upon should help new developers understand the requirements of building collaborative applications.
The goal with VR is providing an immersive experience in which users can interact the same as they would in the real world. That is, using their hands, even the whole body. Taking that into ac-count, and to further guide the development of a solution, this dissertation focuses on collaborative mechanisms around object manipulation in VR.
1.2
Research Questions
Applications that require user collaboration use mechanisms to promote that. Some applica-tions include verbal communication, however, for this dissertation, we are focusing on non-verbal communication. Non-verbal communication involves the use of both passive and explicit features to provide users with enough information about others and their surroundings.
To help accomplish those tasks, the following research questions are defined:
Research Question 1 (RQ1) - What are the different techniques that increase user immersion of a shared virtual environment? This research question explores how to provide users with the feeling of a shared environment. This involves mechanisms such as the representation of avatars, among others.
Research Question 2 (RQ2) - How can multiple users collaborate in a VR environment with-out verbal communication? This dissertation does not tackle verbal communication in the con-text of collaborative applications. It mostly implements techniques that allow users to cooperate through intuition, as there is no specific meaning to the actions they perform.
These research questions will be referred to multiple times throughout this document. They should help with the focus of the research. From now on, each research question will be referred to as RQ1 and RQ2, respectively.
1.3
Objectives
This dissertation, as mentioned above, focuses on studying and developing collaborative tools for VR applications. The main focus is in collaboration, but some interaction features were needed. These tools allow users to interact with the environment, but they also helped guide the develop-ment of the collaborative mechanisms.
Focusing on the research questions defined above, a set of objectives is listed below:
• Research on the current state of virtual reality applications, collaborative mechanisms, and collaborative virtual reality applications.
• Detail the list of collaborative mechanisms that will be implemented
• Implement general VR tools that allow for interaction with the virtual environment • Implement VR tools for manipulating objects in the environment
• Implement the detailed collaborative mechanisms in VR
• Through user tests, analyze the usability of the implemented tools, as well as if all the systems integrate into a cohesive application
A goal with the development of these tools is to help guide other developers that are trying to build collaborative VR applications. The tools detailed and concepts explored in this document should provide a basis for the implementation of collaborative features. The developed systems need to be flexible enough so that other developers can adapt the tools to new applications.
This dissertation is integrated in the context of project PAINTER1. The goal of PAINTER is to develop tools that allow users with different devices to connect to the same virtual environment and interact with it.
1.4
Contributions
The development of a framework around collaboration in VR environments which will help other developers implement these types of applications. The tools were implemented in such a way that developers can simply adjust the existing systems to better fit their applications. A prototype was put together using the different mechanisms.
As stated before, this dissertation is part of the PAINTER project. Alongside this dissertation, Diogo Duque’s master thesis’ AR prototype is also part of PAINTER. It centered itself on de-veloping collaborative tools for augmented reality (AR) applications. Ultimately, the goal was to allow for two clients, one VR and another AR, to connect to the same room and interact with the environment, which was accomplished by the end of this dissertation.
The review of the state of the art provides a detailed overview of what as been done and new paths of research that can be explored. The conducted user tests provide a validation of the tools implemented, which should help new researchers determine their efficacy for new collaborative VR applications. This document also explains the development of the tools, as well as the chal-lenges that are faced when trying to develop these types of systems.
1.5
Document Structure
This chapter (chapter1) provides a brief explanation of the project. It also details the reasons why the project is relevant and the objectives that need to be accomplished.
Subsequently, chapter2, reviews the state of the art. It starts by exploring virtual reality, both hardware and current software solutions for some of its problems. This is followed by an ex-planation of interchangeable reality that is a concept is being introduced in this thesis. Finally, a study of collaborative applications and collaborative VR unveils some strategies for accomplishing collaboration between users.
Chapter3presents the proposed solution for addressing the research questions posed in this chapter. It includes a high-level diagram of the different components and a list of requirements for the solution.
Chapter4starts by detailing the tools and technologies that are used in the development of the project. Afterward, a detailed description of each implemented system is provided alongside the challenges faced during development. The chapter closes with a description of the development environment, the hardware used, and input solution used.
In chapter5the testing phase is detailed. It starts with a description of the tests and how they were executed, followed by an analysis of the results to validate the implemented solution.
Literature Review
The literature review explores the current technologies surrounding virtual reality and discusses some different topics related to the main subject of this dissertation.
Section 2.1 introduces the topic of Virtual Reality, including some concepts related to this theme, as well as some tools. Section2.2is about Interchangeable Reality, a concept introduced in the context of this thesis. It includes Augmented Reality, the Reality-Virtuality Continuum, and some Interchangeable Reality applications. Section 2.3, Collaborative Virtual Reality, includes information about collaborative interfaces, VR and IR tools. At the end of each section, there is a discussion of all the information gathered.
At the end of this chapter, the conclusions drawn from the analysis are presented (Section2.4), as well as a summary of the topics covered and a brief analysis of the various tools that were analyzed.
2.1
Virtual Reality
Virtual Reality places users in a 360o degrees environment that can either be a pre-recorded video or a simulated virtual environment. VR offers an increased immersion, specially when com-pared with other technologies, such as desktop applications. That is accomplished by mimicking the real movements of the user into their virtual experience, i.e. by moving the virtual camera of the users depending on the movements executed by the user in the real world. It offers an interactive fly-through environment that can be explored by the user [IPW+05].
The use of the expression “Virtual Reality” was pioneered by Jaron Lanier, in the 1980s, when referring to the field of research dedicated to the stimulation of the human senses and the simula-tion of virtual environments. However, the first virtual reality device is considered to have been invented in 1962 by Morton Heilig [Hei62]. Shortly after, Sutherland I. et al. developed the first Head-Mounted Display [Sut68].
Things have not evolved much until the release of the Oculus Rift1(Figure2.1) on the March 28th, 2016, and, shortly after, of the HTC Vive2 (Figure 2.2) and Playstation VR3 (Figure 2.3) marked the start of a new era for VR. Ever since then it has been a growing technology4.
Figure 2.1: Oculus Rift Figure 2.2: HTC Vive Figure 2.3: Playstation VR
The gaming industry took an interest in Virtual Reality, but VR has proven to be useful in other fields too. Fields such as industry [OHT+19], healthcare [APY+19], and education. VR has been shown to improve students’ attention span and interest on the subject, however it is unclear whether or not the system also works as a distraction [FB99,AZDA15].
This section of the Literature Review explores existing VR tools and applications, along with different interfaces, to determine what is used in terms of interactions in VR. Since this dissertation explored object manipulation, that is also going to be reviewed.
2.1.1 VR Games, Applications and Tools
As it has been pointed out before, VR is a growing technology, mostly due to the vast applica-tions it can be used for. This section will be focusing on existing VR applicaapplica-tions that exist today as a way to show what is possible to do in VR.
Virtual Reality applications
LumiSpace is a project that brings a daylighting design system to indoor architectural de-signs [HM16]. Hong Y et al. [HM16] explain that the “innate properties of VR”, more specifically immersion and interaction, improve users’ experience as it helps them perceive lighting and space in a different way. The tool was developed in Unity3D5using Samsung’s GearVR headset (Figure
2.4) and a gamepad as the input device.
1Oculus Rifthttps://www.oculus.com/rift/, Last accessed on January 28th, 2019 2HTC Vivehttps://www.vive.com, Last accessed on January 28th, 2019
3Playstation VRhttps://www.playstation.com/playstation-vr/, Last accessed on January 28th, 2019 4Statistica: Number of active virtual reality users worldwide from 2014 to 2018https://www.statista.com/
statistics/426469/active-virtual-reality-users-worldwide/, Last accessed on January 31st, 2019
Figure 2.4: Samsung’s GearVR headset
Unity3D6 is mostly known for being a game development engine, even though it can be used to develop other applications [JYH11]. It incorporates an easy-to-use scene editor and the ap-plications’ behavior is mostly programmed through the use of scripts that are imported into the projects. Unity3D also offers direct support for VR devices, as it already handles camera move-ment, therefore, developers only have to program the user interface.
Figure 2.5: LumiSpace shown through Unity3D’s editor. Adapted from [HM16]
The validation methods used for the research were meant to analyze whether users had a better perception of the space around them. The environment was both built in Unity3D for VR and also in Rhino, a 3D modeling software. Through the analysis of the user studies the research concludes that VR improves the user experience by allowing for a better perception of space.
Chen X. el al. [CSL18] propose the use of a haptic simulator to rehearse the drilling process in a surgery to the craniomaxillofacial region, known for being an anatomically complex region.
The haptic force-feedback device used was the Omega.67which interacted with the application through the use of third-party open source libraries. The interaction device is controlled in the
6Unity3Dhttps://unity.com/, Last accessed on July 1st, 2019
7Omega.6http://www.forcedimension.com/products/omega-6/overview, Last accessed February
application and the force-feedback is actuated in the opposite direction of the drilling. The mesh will then be subdivided so that the drilled holes can be simulated.
The research concludes, through the validation of user studies, that the operation could be rehearsed in VR instead of having to do it on cadavers. That would also lower the expenses of training, which would, in turn, help surgeons improve their skills. M. Fraser et al. [FGV+00] explain that haptic force-feedback is essential for training medical skills: without it, the sensibility to touch is not properly trained.
Virtual Reality games
In 2017, a research paper was published about a game named Wakeboarding [HHYC17]. It worked by having the user perform 4 actions: sliding, leaning, squatting and jumping. What makes this game so relevant, is that the interaction device was a balance board ski, making this an haptic-enhanced virtual reality application (Figure2.6).
Haptic-enhanced virtual reality uses interaction devices that offer haptic feedback as a way to improve engagement [RM14, FGV+00]. In this example, the haptic feedback is given by an actual board ski instead of interacting with the application through the use of buttons or other metaphors that do not necessarily relate to the tasks executed.
The board was built from scratch. A motion controller, an input device that is explored in section 2.1.2, was placed on top of the board to allow for tracking, four tee balls, along with a vibrospeaker, were placed on the bottom. The balls allow to steer the board on the floor, and the speaker increases the haptic-feedback.
The creative design of this game shows that building blocks for any VR creation framework should be set on flexibility. Although this is the most extravagant example shown, the develop-ment of these types of applications should not be limited by the tools that try to facilitate the development of VR applications. In section2.1.2more interface designs in VR are explored.
Figure 2.6: Wakeboard interactions: sliding, leaning, squatting and jumping, from left to right, respectively. Adapted from [HHYC17]
Haptic-enhanced virtual reality can be used for many applications. These mechanisms help users have a sense of presence in the virtual world that would otherwise be lost if using other interaction techniques [RM14,FGV+00].
E. Tuveri et al. [TMS+16] use a concept, which they are calling “Fitmersive” games, to help users be engaged in physical activities through the use of VR and haptic feedback devices.
The developed game, Rift-a-bike, used a regular exercise bicycle adapted to be able to send input to the game. This input would then be used to determine the user’s movement through the virtual track.
Figure 2.7: The setup used by E. Tuveri et al. [TMS+16] for the game Rift-a-Bike
In order to incentivize users to do physical activity, some gamification techniques were applied. Progression through levels is one of them. The levels are what allows the user to progress along the plot of the game and, as a result, gives the user a goal to work towards. Challenges are also another gamification technique implemented in this game. The use of challenges gives a short-term goal for the user to work towards as it rewards them with experience.
The goal of the research is to determine if the gamification techniques used helped users com-plete the challenges and, therefore, do physical exercise. To determine these metrics a user study was conducted using the documented PACES [KD91], Physical Activity Enjoyment Scale, test.
Gamification techniques have been used in several projects as a way to incentivize users to complete certain tasks, in this case, it has to do physical activity, but it can be, for example, to help patients deal with mental health issues [DBV18,FBS+17], and to help in education [HF15,
DSdNDM+13].
Content Creation Tools
Content creation tools in VR require complex interfaces that allow for the customization of the creation mechanisms. The analyzes of some of these applications might unveil key design choices for good VR interfaces.
Some of these tools work as a 3D painting playground. The user chooses a brush, color, adjusts its size and starts painting. Since it is in VR, users can paint depth and in 360 degrees. If the
VR mechanism also tracks position, the movement allows them to play with designs that change radically depending on the users point-of-view.
Tilt Brush8 (figure2.8) is one such application. It is a highly customizable painting appli-cation for VR. Tilt Brush offers a great number of brushes for different effects, from traditional 2D brushes to various meshes that deform according to the movement. The color of the strokes can also change according to the lighting of the scene. The environment is also customizable, from basic colored backgrounds to pre-defined skyboxes and even environments. The features are endless and enable diverse creations that can be seen on the website.
Figure 2.8: Tilt Brush.
For this research project, it is important to analyze the interface designs that enable all those different behaviours. In Tilt Brush one hand is a permanent menu, while the other is the brush that paints. The menu is divided into sections. These rotate around the user’s hand by interacting with either the Trackpad or Joystick in the controllers. Menu options are selected using the free hand by pointing at the desired button. Some buttons open new menus for additional options.
The controls are supported by a representation of the controllers with icons that remind the purpose of each button. That helps mitigate the initial complexity of the application. Tilt Brush also provides several levels of interaction, that enable more features depending on the experience of the user. More complex tools will only be present if the user desires it.
Unbound 9 (figure2.9) is another content creation tool for 3D objects. Unbound lets users
create models using 3D primitives as brushes. Not only that, but it also provides tools to paint the models and change material properties, such as Color, Metalness, Brightness, among others.
Unbound is still in alpha, therefore, the interface design still needs to improve. But, Unbound already provides support for collaborative modeling. This mode allows multiple users to interact in real-time with the same model and collaborate to work on different aspects of the design at the
8Tilt Brush. https://www.oculus.com/experiences/gear-vr/1123989124339476/?locale=pt_
PT, Last accessed on June 15th, 2019
Figure 2.9: Unbound.
same time. The tools the other user is using are all properly represented. The mechanisms that are enabled, such as Mirror mode, are visible for all users, which increases clarity.
Tools for Developing VR Applications
This research project will be developed for VR. While it is possible to develop VR applications from the ground up, there would not be any benefits to do so for this research project. Instead, an existing platform will be used to allow the development to be mostly focused on research questions.
Unity and Unreal Engine are some of the most well-known game development engines avail-able. These two, in particular, offer more features when compared to other engines [RKF+18]. Unreal Engine has the benefit of being open-source and not requiring a fee to have access to all the engine functionalities. Unity, on the other hand, restricts access to some of its features by requiring the acquisition of the professional license.
D. Roth et al. [RKF+18] describe in great detail the benefits of using one engine over the other. While Unreal Engine offers more features than its competition, it is also more specification heavy, requiring more computational power. Unity is a more lightweight option. However, recent changes to Unity’s rendering pipeline provide more customization allowing for the development of great looking applications as well. Both engines have massive user bases that provide support for developers in need of help.
Unreal Engine and Unity both support VR development. Moreover, both engines also support VR editors where developers can build the environment in VR.
Unity’s10 VR editor (figure 2.10) works similarly to Tilt Brush, which was previously ex-plored. An enhancement is that users can open and close the menu instead of always having one hand as the menu. The other will interact with the menu, which rotates around the user’s hand. New windows can be opened an placed around the user for quick access to functionalities.
Figure 2.10: Unity VR Editor.
Unreal Engine’s11 VR editor (figure 2.11) differs from the aforementioned Unity editor, where the menu is just a flat window that resembles Unreal’s desktop interface. One hand still works as the menu while the other interacts with it.
Both VR editors support a snapping feature that sticks objects to other surfaces when close to them. Therefore, objects will not be placed inside others. The fact that this feature exists in both editors might indicate that it is a good feature to have in environment manipulation applications. Like many other features, this can be turned on or off depending on the situation.
2.1.2 VR Interfaces
Interface designs in VR are varied due to a couple of factors, including:
• Interaction devices: there are multiple interaction devices for VR (which will be explored further bellow) making it difficult to decide which ones to support. Each developer will have to choose from a catalog of options and adapt those devices to the application. However, more commercial application, tend to resort to the most mainstream interaction devices. • Increasing immersion: the trend in interaction devices for VR follows 2 trends, which are
not exclusive to each other. First, the devices try to solve the issues inherent to the system, such as cybersickness, and second, the devices try to offer increasingly more immersion to the user.
For virtual environments, which might be larger than the physical space available to the users, there needs to exist a locomotion mechanism that allows them to move around the environment. Standard locomotion methods are generally walking/running or teleporting through the environ-ment.
Teleportation has been shown to decrease the chances of feeling the symptoms of cybersickness while improving enjoyment in experiencing VR [BRKD19]. It works by simply pointing at the intended destination and, if it is a valid place, teleporting there. Therefore, while the users are not moving in real life, they are also not perceiving movement in the virtual world. On the other hand, walking/running increases the chances of feeling the symptoms of cybersickness, and studies show that users are less productive using this navigation technique when compared to desktop users [SSDP+09].
However, applications such as the video game The Climb12 use different locomotion tech-niques. The Climb is a climbing game where players have to use their hands to grab certain spots along the climbing wall to push forward, showcasing that the locomotion systems should be adapted to the context of each application.
Rotations are considered to be the biggest cause for cybersickness [KDK10,FF16,BMMF17]. Cybersickness is a type of motion sickness that will be explored further bellow.
11Unreal Enginehttps://www.unrealengine.com, Last accessed on June 1st, 2019 12The Climbhttp://www.theclimbgame.com/, Last accessed on February 4th, 2019
Rotations should be handled with special care. While soft rotation for changing directions works fine, for sharper turns, it is preferred to use Head Lock mode [AGMC17]. Head Lock mode changes users orientation by a pre-defined amount. That predictability is what helps reduce the chances of cybersickness. The steps of Head Lock mode can be seen in Figure2.12.
Figure 2.12: Head Lock main steps. Adapted from [AGMC17]
The focus of this dissertation is in developing a framework for the development of collaborative interfaces for VR applications. As such, the framework should not limit the interface design of developers.
The collaborative mechanisms developed for this dissertation focus on object manipulation. Object manipulation in VR is of the utmost importance. The manipulation refers to a set of inter-actions users can perform on elements of the environment. These can involve:
• Push: Apply forces on objects in whatever direction a hand is coming from • Grab: An object can be attached to a hand
• Drag: The grabbed object will follow the pose of the hand
• Throw: When an object is let go of a force is applied to it that corresponds to the movement it had before
These are the basic interactions a user can have with a virtual environment. However, the use of devices that keeps track of the pose of a user’s hand, such as Hand tracking devices, allows for more realist interactions.
K.Nasim et al. [NK18] try to solve the problem of real-time grab manipulation in VR. The method uses a collider around the user’s hand that indicates that any object inside that collider is grabbable. If there is an object and the distance between the fingers is diminishing, indicating that the user is trying to grab something, the object will be pushed toward the palm of the hand. M. Holl et al. [HOAL18] developed a physics-based model for interactions with virtual objects. The model computes several contact points between a user’s hands and the object and then calculates the forces at each of those points. The number of control points can be adjusted to adapt to the devices’ computational power.
To interact with elements in a menu or distant objects is normally done by pointing at those el-ements. The use of gamepads, for example, does not allow for the use of hands to point, however, menus can easily be adapted to work with gamepads just like desktop applications. VR applica-tions can be adapted to support a number of input devices, but some of these devices might work better then others for specific types of VR applications.
Input Devices
Basic VR headsets update the users’ head orientation in applications. However, there are more advanced displays that keep track of even more parameters, such as, the head’s position or the area of the display the eyes are looking at [PZBG16].
That input is not enough, for most applications, as it would only allow the users to use their heads to interact with the environment. Since the goal of VR is to increase user immersion in the virtual environment, one of the most used input devices, which most times comes packaged with the headsets, are motion controllers.
Motion controllers (MC) keep track of a user’s hand position and orientation making it possible to both represent them and use them in the virtual world (Figure2.13). Tracking the controllers is done through two different methods, outside-out or inside-out tracking. Outside-out tracking requires the use of external trackers that constantly check the position of the controllers. Inside-out tracking makes use of trackers placed on the headset to determine the position of the controllers. Inside-out tracking has the benefit of requiring less setup work than outside-out, but it increases the blind spots when the controllers are not visible to the headset’s cameras.
The visualization of the hands in the virtual world highly increases the immersion. But this system is not without its flaws. The use of buttons and analog sticks to perform certain tasks reduces immersion since the hands are never a true representation of the users’ real hands. The following list provides an overview over other compatible input devices:
• Hand Trackers - Hand-tracking devices (figure2.14), capture the skeleton structure of the hands. That is, each finger can be tracked separately from the rest of the hand, allowing them to be accurately represented in the virtual world.
• Keyboard and mouse - These are not the most appropriate devices to use with VR. Both require a surface to work and users can not visualize the keyboard when using headsets that do not provide a visual representation of the real world. However, for more traditional controls, i.e., the use of the WASD keys for movement, it is not a problem, but still very restricting.
• Gamepad - The Gamepad offers the freedom for the users to move around as it does not require a surface to work on. If the tasks required from the user do not imply the use of their hands to directly interact with the world gamepads are a good option instead of using more sophisticated input devices. However, the use of Gamepads still impact the experience in terms of immersion, and they are limiting for applications that do make use of virtual hands. • Haptic Gloves - Haptic Gloves (Figure2.15) try to fix the disconnect between the visual perception of touching objects in VR and the sensation of touch in the real world. Some companies are currently developing this technology, however no device is available for the general public. The simulation of haptics is a developing technology still not available to the general consumer. So, researchers have used real objects, in what they are calling direct haptics [CNdSR16], re-created into the virtual environment to provide some type of feedback to the user. They used hand trackers to detect the position of the hands that adjusted the position of the virtual object according to the real one. This method of interaction provides real feedback to the user while also allowing for some basic interaction with the virtual objects.
The interaction devices in VR are many, and as the technology grows new devices start emerge. However, this dissertation was developed for MC devices since they are the most common among VR users and offer the possibility of using two hands, which is important in the context of object manipulation.
Interaction Challenges
VR is still a developing technology. As such, developers have to account for the challenges that come with the system.
One potential barrier to the adoption of VR is cybersickness, that manifests the same symptoms as motion sickness. Motion sickness occurs when there is a disconnect between the eyes and the vestibular system, meaning, for example, some people feel motion sickness when riding on a
Figure 2.14: Leap Motion13hand-tracking.
car since the eyes see movement outside the car but the rest of the body does not perceive that movement. Cybersickness occurs through the use of VR, usually when the user is stationary while visually experiencing motion [LaV00,GGH+19,PKVCvdG18]. This condition can cause nausea, dizziness, fatigue, among others [LaV00]. Cybersickness occurs due to exposure to VR and is thought to be a sub-type of motion sickness since it does not involve the vestibular system [LaV00], although it triggers the same symptoms.
Cybersickness should not be ignored, as first time VR users that experience these symptoms are immediately put off from using the technology again [LaV00]. Studies show that increasing frame-rate and reduced input lag lowers the chances of experiencing cybersickness [FF16]. Other techniques, such as placing static 3D elements in front of the camera, like, for example, using a virtual nose [FF16], or applying motion blur upon head rotation [BMMF17], help in reducing the chances of experiencing cybersickness.
Another potential barrier is for disabled people, which suffer more hardships than most others. These come from barriers, of different nature, imposed on them by their disability. The World Health Organization describes these barriers as:
“(...) factors in a person’s environment that, through their absence or presence, improve func-tioning and reduce disability. These include aspects such as a physical environment that is ac-cessible, the availability of relevant assistive technology, and positive attitudes of people towards disability, as well as services, systems and policies that aim to increase the involvement of all people with a health condition in all areas of life.” [Wor07].
VR can be used for therapeutic reasons [DBV18,FBS+17,TPC17], but for users that have a first bad experience with VR they usually prefer to abandon the VR environment rather than seek assistance [TPC17].
For movement impaired users, most of the traditional interactions in VR are not suitable [TPC17], and so there are multiple interaction devices created for these users, such as a wheelchair as an input device.
WalkinVR Driver14 is a software that provides assistance for disabled users by offering a
Figure 2.15: Haptic Gloves example
range of features that should help them interact with the virtual environment. These include, among others:
• Movement and rotation in the virtual world without performing any movement in the real world (see Figure2.16)
• Controller Position Shift is the adjustment of a controller’s position in the virtual world so that the user can reach further away than would otherwise be possible (see Figure2.17) • Virtual controller is a second controller existing only on the virtual environment, that will
follow the real controller or the head. This allows for the user to interact using only one hand (see Figure2.18)
Figure 2.16: Accessibility: Movement and Rotation. Adapted from https: //www.walkinvrdriver.com
Figure 2.17: Accessibility: Controller Position Shift. Adapted from https: //www.walkinvrdriver.com
Figure 2.18: Accessibility: Virtual Controller. Adapted from https://www. walkinvrdriver.com
Tools, such as the one mentioned above, allow for the integration of people with disabilities into the virtual worlds. The features offered close the gap between what the application requires from the users and what the users can actually execute in the real world.
2.1.3 Discussion
Virtual Reality (VR) transports a user to a new environment, real or simulated. Through the study of the examples mentioned above a conclusion can be made that VR offers the versatil-ity for researchers and developers to explore their ideas. VR can be used to surf down a body of water [HHYC17], but it can also be used for helping patients deal with mental health is-sues [DBV18,FBS+17].
This dissertation explores collaboration in VR environment. Throughout the section we ex-plored VR interfaces and tools to develop mechanisms that adapt well to VR.
2.2
Interchangeable Reality
Interchangeable Reality (IR) is a concept that is being introduced in this dissertation. IR refers to collaborative applications that allow different devices from multiple levels of reality, i.e. VR and AR devices, to interact with both real and virtual objects in the same shared environment. For instance, an IR application could allow for a VR user to join the shared environment of an Augmented Reality, or desktop user.
Augmented Reality (AR) is the topic that will be explained next, along with some tools and examples that might give some insight into how to create an AR application and in what situations it can be used. This topic will then transition into Reality-Virtuality continuum [MTUK95] which joins AR and VR under one spectrum of technologies. Finally, IR will be introduced, along with a few applications and tools that might help unveil the purpose behind this concept.
The PAINTER project, in which this dissertation is inserted, aims at exploring the concept of IR. It starts by having VR and AR clients shared the environment.
2.2.1 Augmented Reality
Augmented Reality consists of layering virtual objects on top of the real world by using a camera and a digital device to process and display that information.
Industrial Augmented Reality
The use of AR has many applications, many of which are related to industrial processes. In 1994, Boeing release a research paper [Miz94] that analyzed the use of AR to superimpose di-agrams and text onto the surface of an identified piece in an effort to provide information and, subsequently, reduce the number of errors during the manufacturing process.
IAR, industrial augmented reality, applications are driven by a few characteristics [BNFCFLVM18]:
• User-friendly: any user must be capable of performing the tasks required from these sys-tems
• Scalable: reproduction and distribution of these systems should be possible
O. Blanco-Novoa et al. [BNFCFLVM18] offer great insight into the capabilities of such sys-tems, describing the use cases of IAR syssys-tems, and then developing one for an Industry 4.0 Ship-yard. The paper concludes that, even though IAR can already be integrated into many processes, it still needs to be improved, especially in low lighting ambients, for it to be reliably used in their case study.
Figure 2.19: Information about a pipe ob-tained through AR and a phone. Adapted from [BNFCFLVM18]
Figure 2.20: Information obtained through AR and a tablet. Adapted from [BNFCFLVM18]
Mobile Augmented Reality
MAR, mobile augmented reality, is AR driven by devices, such as smartphones, tablets, among others. This is only possible because these devices are now equipped with increasingly more pow-erful processors [HZR16], cameras, and inertial sensors [YGXC18]. In Pokemon GO15, shown in Figure2.21, MAR was used to render popular fictional characters into the real world.
X. Yang et al. [YGXC18] propose the adaption of 3D object detection and pose tracking al-gorithms that rely on computer vision to be able to take advantage of inertial sensors present in mobile devices as a way to predict the position of the objects in the previous frame, reducing the processing power previously required at every frame. The research concludes that it is possible to achieve good results of pose tracking a 3D object in real-time using the purposed method.
D. Chatzopoulos et al. [CBHH17] do an overview of the developments in MAR, its current lim-itation and future projects that might take place as mobile devices get increasingly more powerful, and equipped with more tools useful for AR, and as the technology evolves and matures.
Figure 2.21: A picture of the application Pokemon GO rendering a fan favorite Pokemon, Charizard, into the real world
2.2.2 Reality-Virtuality Continuum
VR immerses the user in a completely virtual environment that they can explore. VR can even mimic the real world, through the rendering of photorealistic objects, or in realistic physics simulations. But everything a user sees in VR is virtual.
On the other hand, there is no AR without real objects, since this technology is the suprapo-sitioning of virtual objects, such as labels or 3D models, into a captured real-time video of the environment.
A problem arose when VR was being used to describe AR applications. P. Milgram et al. [MTUK95] noticed the issue and published the aforementioned paper that differentiated both sys-tems. The research paper documents a spectrum of technologies that go from the real environment to the virtual environment.
Figure 2.22: Mixed-Reality Continuum as described by P. Milgrim et al. [MTUK95]
while the right-most point is the virtual environment, meaning there are no real objects. In that sense, VR falls into the right-most side of the spectrum, as it involves the simulation of a com-pletely virtual world. At the middle of the spectrum, there are technologies, such as, AR, that use both real and virtual environments in its applications.
MR is the set of technologies that fall into the spectrum. However, the research paper does not mention the use of several systems, such as VR and AR, in a single collaborative application. There does not seem to be a lot of emphasis on applications that join the benefits of different systems. The next topic will be about that concept, in what we are calling Interchangeable Reality (IR).
2.2.3 Interchangeable Reality applications
As mentioned in the previous section2.2.2, Interchangeable Reality refers to the systems and applications that allow for interaction between different devices with both real and virtual objects in the same shared environment.
AR allows for the visualization of data relevant, or not, to the environment being recorded, which involves the visualization of diagrams and manuals [Miz94], the visualization of architec-tural models on site [DW05], among many other contexts. On the other hand, VR is an immersive visualization tool, that lets users interact with virtual environments, for examples, visualization architectural models at different times of the day [HM16], or the visualization of software code through a city metaphor [MGAN17].
Architectural Modeling using Interchangeable Reality
Consider the idea of a software program that allows for different visualization methods of architectural models.
The modeling phase would mainly be done in a Desktop, as it offers better control over the process. The VR system would be used to allow for better visualization of the space, in different lighting conditions [HM16], for both architects and clients. And finally, AR, as a final step on the development process, which would be used to visualize the building on the intended construction site [DW05].
Such an application would help clients understand the architect’s vision, promoting communi-cation and interaction between both parties.
2.2.4 Discussion
IR is a concept that promotes the design of multi-purpose applications whose services can vary depending on the interaction intended at any given time. Interchangeable Reality is related to topics such as Augmented Reality and Virtual Reality, for example.
Augmented reality (Section 2.2.1) the visualization of virtual objects, as well as interacting with them, layered on top of the real world. Many of the uses of this type of reality are related to the industry, in order to improve their processes [Miz94,BNFCFLVM18]. In addition, AR applied to industrial processes allowed the creation of Industry 4.0.
These types of systems have also allowed for the development of different mobile applications, which allow the user to have a better experience. AR applications have increased in popularity, one example that was presented was the game Pokemon Go, which was a success and it is still a popular game, demonstrating that it is possible to present 3D objects in mobile systems, correctly. Reality-Virtuality Continuum (Section2.2.2) presents the idea of combining technologies, such as VR and AR, in one spectrum that describes the level of reality and virtuality in a system.
There are several advances regarding Interchangeable Reality, in which AR and VR concepts are conjugated. These systems tend to be collaborative applications because it allows for the po-tential of having several users interact and visualize the environment in different ways. Interacting with the same environment in different ways at the same time foments communication and team-work to accomplish a task.
2.3
Collaborative Virtual Reality
Organizations have been using collaborative strategies to gain a competitive advantage, through the use of software applications. M. Maybury et al. [MDH01] describe, that the use of software can enhance an organization by allowing it to become more agile in its processes.
Collaboration between multiple users requires, above all, information transparency [WN09,
NB03]. For users to be able to come up with solutions for a given problem the information has to be laid out for everyone. This can be done through interface designs [Xia19] that make it possible for shared data to be displayed across everyone. For users that come from different backgrounds, this becomes even more complex, as raw information might not be enough for all user to come to a conclusion, meaning that data as to be mixed together by specialized users into new information [Xia19].
However, the software mentioned above does not require users to interact in real-time. The same can not be applied to shared VR environments.
Immersive social interactions in VR require low latency communications between the clients in that server [PPS18]. The latency of network communications is defined as the time it takes for a command sent by the client to reach the server, which will then be broadcast to all other users [CH01,Hen01]. The physical distance between a user and the server is one of the factors that go into minimizing latency issues [PH18], and that is why most games, such as, League of Legends16and Fortnite17, join users by regions.
M. Fraser et al. [FGV+00] explain that latency issues can be disrupted in communications between users. For clients with higher latency, it will also be more difficult to interact with the environment as commands will take more time to reach and be processed by the server. In the same publication, M. Fraser et al., reference incompatible updates between several users. For these issues, commands have to be timed correctly and the server will then decide based on the event that took place first and rejects the others. Meaning that clients need to wait for the confirmation
16League of Legendshttps://leagueoflegends.com, Last accessed on June 27th, 2019 17Fortnitehttps://www.epicgames.com/fortnite/home, Last accessed on February 5th, 2019
of the server to a command, affecting even further slow clients that receive that confirmation with an even higher delay.
There are algorithms that compensate for the communication delay between clients, such as the Lag Compensation algorithm [LC18], which rewinds time for every command, to predict the state of the game the player was experiencing at the time of issuing the command.
Figure 2.23: Lag compensation algorithm predicting a player’s hitbox18
Collaborative applications for real-time scenarios have to face issues mostly related to commu-nication between clients. Although the focus of this research project is not on solving these issues, it is still important to know the effects they might have on a Virtual Reality application.
2.3.1 Collaboration mechanisms
A collaborative application has users interacting to accomplish a goal. One example of such an application would be team-based multiplayer games. These types of games normally require strategy to overcome challenges. Coordination between the users of the same team is the key to success. Games normally provide users with tools for communication. Since the purpose of each game differs, the provided tools need to change as well. Communication tools should focus on the potential challenges faced throughout the game.
Collaborative applications can be used in every setting that benefits from having multiple users working together in a virtual environment. These applications need to find a balance between providing sufficient communication tools and maintaining an intuitive and cohesive system. The main mindset when developing these tools is to increase usability, and not the other way around.
As referenced above, the method of communication between users will greatly depend on the purpose of the application. Moreover, it also depends on the type of application. Going back to the example of team-based games, even if two games present users with similar challenges the tools provided might be completely different if the genre of game is not the same. Communication systems will depend on the hardware, interface design, among others.
18Lag Compensation algorithm https://store.steampowered.com/app/730/CounterStrike_
Teruel M. et al. [TNG+16] defend that awareness is an important factor for the success of collaborative applications. The lack of awareness still allows the use of these types of applications but collaboration becomes clumsy. Providing users with enough information about the state of others increases collaboration.
One such example is, in the previously mentioned (section2.1.1) application, Unbound, when a user activates Mirror mode others can visualize it. Being able to perceive what tools others are using also increases awareness. In games like League of Legends, players can send out notifica-tions with pre-defined meaning at locanotifica-tions in the map. This is a very important mechanic in the game to communicate individual knowledge to players of the same team.
2.3.2 Collaborative Virtual Reality applications
Shared experiences in VR have to be designed so that the interactions between the multiple users do not break immersion from the environment, but enhance it [LM16]. Social interactions require the use of several visual cues to feel right, without taking the users out of the experi-ence [BBL+05].
Visual cues would include, the presence of other users’ avatars, but also head-pose and eye-gaze [BBL+05, RKF+18]. These help users deal with the fact that they are interacting with a virtual avatar. However tracking eye-gaze requires the use of eye tracking devices which is a technology that is not found in most HMD [PZBG16], in which case, it is better not to render an avatar’s eyes, and rely on the head-movement, rather than having eyes with no eye move-ment [BBL+05,RKF+18]. D. Roth et al. [RKF+18] used particle systems as visual cues to indi-cate an interaction between the various entities.
For local multiplayer experiences, the presence of other users on the same real environment can break immersion. However, S. Liszio et al. [LM16] developed a research project that designs real-world interactions as part of the project’s design. They also added that, for local multiplayer experiences, there must be a combination of multiple platforms since it is really unusual for today’s households to have more than one HMD.
Collaborative applications can take advantage of having multiple users in one single environ-ment to have them interact with the system in different ways that compleenviron-ment each other. L. Li et al. [LZ16] created a VR system for enhancing science museum exhibits by creating virtual ex-periences that used the real equipment on the museums. The research project would have 5 users distributed across 5 roles, each one using different interaction devices.
The possibility of having users distributed across multiple roles that use different input de-vices, or different interfaces [LZ16], are an important part of collaborative VR. There are no tools to easily manage these systems, therefore these functionalities have to be integrated into the appli-cations’ development cycle, and such was the case of the previously mentioned research project.
This concept, of multi-purpose applications, adapts to accessibility issues within VR. Develop-ing interfaces and designDevelop-ing roles around different disabilities could help integrate disabled people on these applications. The roles and interfaces would be adapted to the disability.
This dissertation does not implement different interfaces for each user, however, they are able to select different ways of interacting with the environment depending on the situation.
VRChat19
Gaming benefits from the study of collaborative VR. VRChat is one example of a game de-veloped for letting players interact in a social environment. VRChat lets players choose an avatar of their liking. Users will then choose a lobby to join, which will take them to a virtual environ-ment that they can explore and interact with other users. Users can communicate with each other through voice chat, and they can interact with certain objects scattered across the scene.
The locomotion implemented into this game works by the player pointing to the location they want to move to, and their avatar will start moving into that position, while the camera stays on the same place. This is an adaptation of the teleporting mechanic, the difference is that other players see the user’s character moving in the environment. This helps combat cybersickness and, at the same time, other players can still visualize player movement without just seeing them disappear in one spot and reappear in another, as is the case with teleporting.
2.3.3 Collaborative IR Tools
IR tools focus on the modes of interaction between the users sharing the same environment across different systems. These tools as specialized at adapting to the device a user is using and from there control the behavior of the application.
There are tools for creating VR collaborative applications where users can experience and interact at the same time [DOD+09,NB03,LM16,LZ16]. And there are also AR collaborative applications that allow the users to interact within the same environment [YKO15]. But these use cases can be complementary [KTY99,PDE+17].
K. Kiyokawa et al. [KTY99] describe a method for integrating both approaches, which they call seamless view-mode switching. This technique involves a HMD that can both perform VR and AR. The HMD has two modes of operation, one which is see-through for AR, and a blind mode, that only displays the virtual environment. The purposed method involves localized collaboration between the users. which is very restricting. Also, now-a-days, HMD are either focused for VR or AR.
The MagicBook
Localized experiences offer the benefits of users being able to interact in the real environment with each other.
The MagicBook [BKP01] uses that same concept to have users experience a book through both AR and VR. This application lets users interact with the environments along the story of the book.
Without the use of the device, users can read the book and see the images. Upon looking at the book through AR displays, users will be able to see the virtual environment on top of the book. If
the user switches to VR they will be able to move around the environment. Figure2.24shows the different interaction modes, going from left to right, it shows the normal view without the use of technology, the AR view, and the VR view, respectively.
Figure 2.24: The 3 different interaction modes in “The MagicBook”. Adapted from [BKP01]
This research project focused on using the benefits of each technology to enhance the experi-ence of reading a book. The decision to make this a collaborative localized experiexperi-ence allowed them to integrate several layers of environment interaction [BKP01].
“Physical objects, AR interfaces and immersive VR experiences have different advantages and disadvantages for supporting collaboration. As shown by Benford’s classification, there has been a proliferation of collaborative interfaces, but it has traditionally been difficult to move between the shared spaces they create. For example, users in an immersive virtual environment are separated from the physical world and cannot collaborate with users in the real environment.” [BKP01]
CoVAR
The application mentioned above does not give insight into what makes a good IR collaborative tool, it was created in an IR environment, but is very specialized for what it does.
CoVAR [PDE+17] is a remote collaboration tool for IR systems developed in Unity3D. The system architecture (Figure2.25) for CoVAR can be described as a Client-Server architecture in which any of the users is the Server and all others connect to him. In this way, CoVAR allows having only AR or only VR use cases, if that what the application being develop requires.
This tool integrates a few interface features:
• Head-Gaze: the head gaze is updated by the HMD, in AR the HoloLens is used, and in VR the HTC Vive is utilized
• Eye-Gaze: CoVAR integrates eye-tracking devices to be able to show the direction of a user is looking at. CoVAR represents the eye-gaze with a ray
• Hand Gestures: the use of Leap Motion allows the tool to keep track of a user’s hand pose, and, with that, a set of possible interactions were implemented
Figure 2.25: “CoVAR”’s architecture. Adapted from [PDE+17]
The problem with this research project is that is restricting on the devices it requires and inter-actions it provides between the multiple users. This tool is more focused on building a tool that allows for the creation of a specific type of IR applications, ones that demand the use of all the hardware CoVAR requires.
2.3.4 Discussion
Collaborative applications have to deal with the network speed of the time at which they are being developed [FGV+00]. That includes taking the necessary steps toward overcoming latency issues.
Collaboration in VR is a concept that is starting to emerge, most notably in VR games. How-ever, these only provide insights into how to provide the sensation of shared environment, since they are lacking in more explicit collaborative mechanisms. The growing demand for collabo-rative applications will demand researchers to explore new paths, such as collaboration in VR environments.
IR allows multiple users to collaborate across different types of systems. For this, it is necessary to have transparency between the different types of realities [WN09,NB03], in order to be able to operate an application through AR or VR, for example. This is the main difficulty related to collaborative IR, handling the different layers of reality and interactions between all users. This is a challenges for the PAINTER project and will be referenced multiple times during the planning and development chapters of this document.
2.4
Summary
Some of the information collected for each VR application, presented in Section2.1.1, is sum-marized below. This analysis takes into account not only the nomenclature of each system but also
