Leveraging eXtented Reality & Human-Computer Interaction for User Experi- ence in 360◦ Video

DEPARTMENT OF COMPUTER SCIENCE

LEVERAGING EXTENTED REALITY &

HUMAN-COMPUTER INTERACTION

FOR USER EXPERIENCE IN 360 ^◦ VIDEO

PAULO ALEXANDRE CÂMARA BALA Master in Informatics Engineering

DOCTORATE IN DIGITAL MEDIA

NOVA University of Lisbon

COMPUTER SCIENCE

LEVERAGING EXTENTED REALITY &

HUMAN-COMPUTER INTERACTION FOR USER EXPERIENCE IN 360 ^◦ VIDEO

PAULO ALEXANDRE CÂMARA BALA Master in Informatics Engineering

Adviser: Valentina Nisi

Associate Professor, Instituto Superior Técnico, University of Lisbon

Co-advisers: Nuno Manuel Robalo Correia

Full Professor, NOVA School of Science & Technology

Duarte Nuno Jardim Nunes

Full Professor, Instituto Superior Técnico, University of Lisbon

Examination Committee:

Chair: Pedro Manuel Corrêa Calvente Barahona

Full Professor, NOVA School of Science & Technology

Rapporteurs: Steve Benford

Full Professor, Department of Computer Science, University of Nottingham

Rui Pedro Amaral Rodrigues

Assistant Professor, Faculty of Engineering, Universidade of Porto

Adviser: Valentina Nisi

Associate Professor, Instituto Superior Técnico, University of Lisbon

Members: Johannes Schöning

Full Professor, School of Computer Science, University of St. Gallen

Maria Teresa Caeiro Chambel

Associate Professor, Faculty of Science, University of Lisbon

Teresa Isabel Lopes Romão

Associate Professor, NOVA School of Science & Technology

Rui Pedro da Silva Nóbrega

Assistant Professor, NOVA School of Science & Technology

DOCTORATE IN DIGITAL MEDIA

Leveraging eXtented Reality & Human-Computer Interaction for User Experi- ence in 360^◦Video

Copyright © Paulo Alexandre Câmara Bala, NOVA School of Science and Technology, NOVA University of Lisbon.

The NOVA School of Science and Technology and the NOVA University of Lisbon have the right, perpetual and without geographical boundaries, to file and publish this dissertation through printed copies reproduced on paper or on digital form, or by any other means known or that may be invented, and to disseminate through scientific repositories and admit its copying and distribution for non-commercial, educational or research purposes, as long as credit is given to the author and editor.

A c k n o w l e d g e m e n t s

This research would not have been possible without the advice, support, and collaboration of many people.

I want to start by acknowledging my supervisor Prof. Dr. Valentina Nisi, for her guidance along this journey, especially knowing when to push me forward. I also want to acknowledge my co-advisors Prof. Dr. Nuno Jardim Nunes and Prof. Nuno Correia for their insights and time. I wish to thank the research proposal committee, Prof. Dr.

Johannes Schöning, Prof. Dr. Teresa Chambel, and Prof. Dr. Teresa Romão, for their feedback and suggestions on how to make my work stronger.

I wish to acknowledge the support of the following funding entities during the devel- opment of this work: the Fundação Ciência e Tecnologia (FCT) Grant PD/BD/128330/2017 (under the UTAustin-Portugal framework); the MADEIRA 14-20 FEDER funded project Beanstalk (2015-2020) MITIExcell (M1420-01-0145-FEDER-000002); LARSys-FCT pluri- annual funding (UID/EEA/50009/2019 and UIDB/50009/2020); the internal FCT-UNL funding for Digital Media. These funding opportunities not only made it possible for me to sustain myself, but helped to sustain this research through conferences registrations, travel, equipment, etc.

I thank the support of my host institution, Interactive Technologies Institute, and my main institution, Faculdade de Ciências e Tecnologias of Universidade Nova de Lisboa, including both faculty, staff, and research colleagues.

This research involved several collaborations. I particularly would like to thank my co-authors Prof. Dr. Ian Oakley, Prof. Dr. Mara Dionísio, and Raul Masu; the experience of writing with others was the most enjoyable and rewarding part of this journey. A note of gratitude as well to the members of Beanstalk team (Dina Dionísio, Rui Trindade, and Sandra Olim) for their support in prototyping and evaluation studies. I would like to thank as well, the many participants of these studies, for their feedback and time.

Lastly, I would like to thank my family and friends. Through a mix of life events and world events, doing this research and writing a thesis was not a breeze; it often felt like a hurricane. Thank you all for keeping me grounded.

Randy Pausch, The Last Lecture

A b s t r a c t

EXtended Reality systems have resurged as a medium for work and entertainment. While 360º video has been characterized as less immersive than computer-generated VR, its realism, ease of use and affordability mean it is in widespread commercial use. Based on the prevalence and potential of the 360º video format, this research is focused on improving and augmenting the user experience of watching360º video. By leveraging knowledge fromExtented Reality (XR)systems andHuman-Computer Interaction (HCI), this research addresses two issues affecting user experience in 360º video: Attention Guidance andVisually Induced Motion Sickness (VIMS).

This research work relies on the construction of multipleartifactsto answer the de- fined research questions: (1)IVRUX, a tool for analysis of immersiveVRnarrative expe- riences; (2)Cue Control, a tool for creation of spatial audio soundtracks for360º video, as well as enabling the collection and analysis of captured metrics emerging from the user experience; and (3) VIMSmitigation pipeline, a linear sequence of modules (including optical flow and visual SLAM among others) that control parameters for visual modi- fications such as a restricted Field of View (FoV). These artifactsare accompanied by evaluation studies targeting the defined research questions. Through Cue Control, this research shows that non-diegetic music can be spatialized to act as orientation for users.

A partial spatialization of music was deemed ineffective when used for orientation. Addi- tionally, our results also demonstrate that diegetic sounds are used for notification rather than orientation. ThroughVIMSmitigation pipeline, this research shows that dynamic restrictedFoVis statistically significant in mitigatingVIMS, while mantaining desired levels of Presence. Both Cue Controland theVIMSmitigation pipeline emerged from a Research through Design (RtD)approach, where theIVRUXartifactis the product of de- sign knowledge and gave direction to research. The research presented in this thesis is of interest to practitioners and researchers working on 360º video and helps delineate future directions in making360º videoa rich design space for interaction and narrative.

Keywords:EXtended Reality, Virtual Reality, 360º Video, Visually Induced Motion Sick- ness, Attention Guidance, Human-Computer Interaction, Perceptual Systems

Sistemas de Realidade EXtendida ressurgiram como um meio de comunicação para o tra- balho e entretenimento. Enquanto que o vídeo 360º tem sido caracterizado como sendo menos imersivo que a Realidade Virtual gerada por computador, o seu realismo, facili- dade de uso e acessibilidade significa que tem uso comercial generalizado. Baseado na prevalência e potencial do formato de vídeo 360º, esta pesquisa está focada em melhorar e aumentar a experiência de utilizador ao ver vídeos 360º. Impulsionado por conhecimento de sistemas de Realidade eXtendida (XR) e Interacção Humano-Computador (HCI), esta pesquisa aborda dois problemas que afetam a experiência de utilizador em vídeo 360º:

Orientação de Atenção e Enjoo de Movimento Induzido Visualmente (VIMS).

Este trabalho de pesquisa é apoiado na construção de múltiplos artefactos para res- ponder as perguntas de pesquisa definidas: (1)IVRUX, uma ferramenta para análise de experiências narrativas imersivas emVR; (2)Cue Control, uma ferramenta para a criação de bandas sonoras de áudio espacial, enquanto permite a recolha e análise de métricas capturadas emergentes da experiencia de utilizador; e (3)canal para a mitigação deVIMS, uma sequência linear de módulos (incluindo fluxo ótico eSLAMvisual entre outros) que controla parâmetros para modificações visuais como o campo de visão restringido. Estes artefactos estão acompanhados por estudos de avaliação direcionados para às perguntas de pesquisa definidas. Através do Cue Control, esta pesquisa mostra que música não- diegética pode ser espacializada para servir como orientação para os utilizadores. Uma espacialização parcial da música foi considerada ineficaz quando usada para a orientação.

Adicionalmente, os nossos resultados demonstram que sons diegéticos são usados para notificação em vez de orientação. Através docanal para a mitigação deVIMS, esta pesquisa mostra que o campo de visão restrito e dinâmico é estatisticamente significante ao mitigar VIMS, enquanto mantem níveis desejados de Presença. AmbosCue Controle ocanal para a mitigação deVIMSemergiram de uma abordagem dePesquisa através do Design (RtD), onde o artefactoIVRUXé o produto de conhecimento de design e deu direcção à pesquisa.

A pesquisa apresentada nesta tese é de interesse para profissionais e investigadores tra- balhando em vídeo 360º e ajuda a delinear futuras direções em tornar o vídeo 360º um espaço de design rico para a interação e narrativa.

Palavras-chave: Realidade eXtendida, Realidade Virtual, Vídeo 360º, Enjoo de Movi- mento Induzido Visualmente, Orientação de Atenção, Interação Humano-Computador, Sistemas de Perceção

List of Figures xv

List of Tables xviii

Acronyms xxi

Glossary xxiv

1 Introduction 2

1.1 Motivation . . . 2

1.2 Research Goals and Questions . . . 3

1.3 Research Overview/Methodology . . . 4

1.4 Contributions and Publications . . . 5

1.5 Document Structure. . . 13

I Theoretical Background & Related Work 14 2 Theoretical Background 15 2.1 Human-Computer Interaction . . . 15

2.1.1 Computing and HCI . . . 16

2.1.2 HCI as Paradigms and Waves . . . 17

2.1.3 HCI as Periods . . . 19

2.2 Perceptual Systems . . . 23

2.2.1 Visual System . . . 23

2.2.2 Auditory System. . . 24

2.2.3 Vestibular System . . . 25

2.2.4 Multisensory Integration & Misalignment . . . 26

2.3 EXtended Reality . . . 29

2.3.1 Forms of Reality . . . 29

2.3.2 EXtended Reality Systems . . . 32

C O N T E N T S

2.3.3 360º video . . . 33

2.4 Conclusion . . . 35

3 Sound for Attention Guidance 40 3.1 Sound in HCI . . . 40

3.1.1 Spatial Auditory Interfaces. . . 41

3.2 Attention Guidance . . . 42

3.3 Conclusion . . . 44

4 Visually Induced Motion Sicknesss 46 4.1 Symptomatology, Epidemiology and Pathophysiology of VIMS . . . 46

4.2 VIMS Mitigation . . . 47

4.2.1 Rest-frame/Independent Visual Background . . . 48

4.2.2 FoV Restriction . . . 48

4.3 Conclusion . . . 49

II Primary Research - Augmenting 360º Video 51 5 IVRUX - Analyzing XR Immersive Narratives 52 5.1 Fragments of Laura . . . 52

5.1.1 The Old Pharmacyas mobile VR - Influence of Location and Medium applied to mobile VR Storytelling . . . 54

5.1.2 The Old Pharmacyin MR - Navigation Styles in Mixed Reality Enter- tainment . . . 55

5.1.3 The Old Pharmacyas VR - Remediating a locative tour to Virtual Reality . . . 58

5.2 IVRUX - Immersive Virtual Reality User Experience . . . 62

5.2.1 First iteration of IVRUX . . . 62

5.2.2 Second iteration of IVRUX . . . 64

5.2.3 Third iteration of IVRUX . . . 65

5.3 Conclusion . . . 68

6 Cue Control - Using Spatial Audio for Orientation in 360º Videos 70 6.1 Introduction . . . 70

6.2 Design ofCue Control . . . 72

6.3 Evaluation Study . . . 73

6.3.1 Media . . . 74

6.3.2 Experimental Design . . . 75

6.3.3 Measures . . . 75

6.3.4 Analysis . . . 76

6.4 Results . . . 78

6.4.1 Sample . . . 78

6.4.2 City Video . . . 79

6.4.3 Safari Video . . . 81

6.4.4 Post Study Interview . . . 82

6.5 Discussion . . . 84

6.5.1 Methodological considerations . . . 84

6.5.2 RQ1 - Can spatialized music guide viewers’ attention in 360º video? 85 6.5.3 RQ2 - Can the use of diegetic cues be reinforced by the audio spa- tialization to guide viewers in 360º video? . . . 86

6.5.4 Implication for Design . . . 87

6.5.5 Future Work . . . 87

6.6 Conclusion . . . 88

7 Staying on Track - Use of Optical Flow in 360º Video to Mitigate VIMS 89 7.1 Introduction . . . 89

7.2 Design of Dynamic Field of View Restriction. . . 91

7.2.1 Optical Flow . . . 91

7.2.2 Implementation . . . 93

7.3 Evaluation Study . . . 93

7.3.1 Media . . . 95

7.3.2 Experimental Design . . . 95

7.3.3 Measures . . . 97

7.3.4 Sample . . . 98

7.3.5 Analysis . . . 98

7.4 Results of Self-Reported Measures. . . 99

7.4.1 Sample . . . 99

7.4.2 IPQ & SSQ . . . 100

7.4.3 Visual Questionnaire . . . 101

7.5 Discussion of Self-reported Measures . . . 103

7.6 Results of Objective Measures . . . 104

7.7 General Discussion . . . 108

7.7.1 RQ3 - Can optical flow of a 360º video be used to mitigate VIMS? 108 7.7.2 RQ4 - Can the combination of different VIMS-mitigating techniques for VR be sucessful in 360º video?. . . 109

7.7.3 Implications for Design & Future Work . . . 109

7.8 Conclusion . . . 110

8 Dynamic Field of View Restriction - Aligning Optical Flow and Visual SLAM to Mitigate VIMS 111 8.1 Introduction . . . 111

8.2 Design of Dynamic Field of View Restriction. . . 112

C O N T E N T S

8.2.1 Optical Flow . . . 114

8.2.2 Visual SLAM. . . 114

8.2.3 Restricted FoV . . . 115

8.3 Evaluation Study . . . 116

8.3.1 Media . . . 116

8.3.2 Experimental Design . . . 118

8.3.3 Measures . . . 119

8.3.4 Sample . . . 120

8.3.5 Analysis . . . 121

8.4 Results . . . 122

8.4.1 Sample . . . 122

8.4.2 Self-Reported Measures . . . 122

8.4.3 Objective Measures . . . 126

8.4.4 Thematic Analysis . . . 129

8.5 Discussion . . . 130

8.5.1 Implications for Design, Limitations and Future Work . . . 132

8.6 Conclusion . . . 133

9 Conclusion 134 9.1 Research Summary and Findings . . . 134

9.2 Limitations . . . 139

9.3 Future Directions . . . 140

Bibliography 142 Appendices A Appendix for Chapter 2 195 A.1 Classical HCI. . . 195

A.2 Modern HCI . . . 196

A.3 Contemporary HCI . . . 198

B Appendix for Chapter 5 200 B.1 Pilot Study on Medium and Location in the first iteration ofThe Old Phar- macy. . . 200

B.1.1 Experimental Design . . . 200

B.1.2 Measures & Procedure . . . 200

B.1.3 Sample . . . 202

B.1.4 Results . . . 202

B.1.5 Discussion . . . 202

B.2 Study on Navigation Styles in the second iteration ofThe Old Pharmacy 204 B.2.1 Experimental Design . . . 204

B.2.2 Measures & Procedure . . . 204

B.2.3 Sample . . . 205

B.2.4 Analysis . . . 205

B.2.5 Quantitative Results . . . 205

B.2.6 Qualitative Results . . . 206

B.2.7 Discussion . . . 209

B.3 Study on the use of IVRUX in the first iteration ofThe Old Pharmacy . . 210

B.3.1 Experimental Design . . . 210

B.3.2 Measures & Procedure . . . 210

B.3.3 Sample . . . 210

B.3.4 Analysis . . . 210

B.3.5 Results . . . 211

B.3.6 Discussion . . . 213

B.4 Online Supplementary Material . . . 214

C Appendix for Chapter 6 215 C.1 Online Supplementary Material . . . 215

D Appendix for Chapter 7 217 D.1 Results for the Upper 75th percentile Subpopulation . . . 217

D.2 Results for the Female Subpopulation. . . 220

D.3 Online Supplementary Material . . . 223

E Appendix for Chapter 8 226 E.1 Supplementary Results . . . 226

E.2 Online Supplementary Material . . . 229

L i s t o f F i g u r e s

1.1 Research scope . . . 3

2.1 Infographic on User Experience disciplines fromenvis precisely[pre21] . . 16

2.2 Visual system . . . 24

2.3 Timeline of relevant events in the evolution of XR systems, literature, film, and XR concepts, based on [Jer15a; MNB14; Sta16; Ste16a], up to 1980. . . 30

2.4 Timeline of relevant events in the evolution of XR systems, literature, film, and XR concepts, based on [Jer15a; MNB14; Sta16; Ste16a], from 1980 onwards. 30 2.5 Narrowing of research scope into Attention Guidance (chapter 3) and VIMS (chapter 4) . . . 39

5.1 Media channels in the FoL Transmedia . . . 54

5.2 First iteration ofThe Old Pharmacy . . . 56

5.3 Second iteration ofThe Old Pharmacy . . . 57

5.4 The Old Pharmacyroom layout . . . 58

5.5 Navigation styles:Screen,HybridandSpatial. . . . 59

5.6 Fragments of Laura installation at ICIDS 2017 . . . 60

5.7 MappingFragments, a VR remediation of the FoL locative tour . . . 61

5.8 IVRUX Interface . . . 63

5.9 IVRUX reimplementation for the second iteration ofFoL, running on a NVIDIA Shield K1 tablet . . . 66

5.10 High fidelity wireframes for a reconceptualized third iteration of IVRUX . 67 5.11 Diagram of different artifacts resulting from RtD approach described in this chapter . . . 69

6.1 Cue Controlapplication showing the timeline interface for soundtrack creation, and heatmaps for analysis . . . 72

6.2 Hotspots for scene S9 in City Video . . . 80

6.3 Hotspots for scene S9 in Safari video . . . 82

7.1 Optical flow pipeline . . . 92

7.2 Optical Flow (sum of absolute optical flow vectors, abs(xVel) + abs(yVel)) across time (frames) . . . 93 7.3 Relation between Optical Flow magnitude and FoV for IVB, r-FoV & r-FoV+IVB

. . . 94 7.4 Top-down representation of the roller coaster track at Seoul Grand Park. Seg-

ments colors correspond to subsegments in table 7.2 . . . 95 7.5 Boxplots with absolute (SSQ-PreTS and SSQ-PostTS) and relative (SSQ-TS) for

the SSQ component across conditions, considering the full population . . . 100 7.6 Time series for S2 and S3, considering the full population: x-axis is represen-

tative of frames; y-axis, from top to bottom: yaw, roll, pitch, distance to POI (DP) . . . 105 7.7 Time series for accumulated distances, considering the full population . . 106 7.8 Time series of hotspots for subsections of S2, considering the full population. 107 7.9 Time series of hotspots for subsections of S3, considering the full population. 108 8.1 Dynamic FoV Restriction pipeline . . . 113 8.2 Time series of optical flow (top), tracked landmarks inOpenVSLAM(center),

and automatic movement classification based onOpenVSLAM’s tracking (bot- tom) . . . 118 8.3 Stacked barplot for Visual Questionnaire (VS1 to VS7), where response 0 is

equal to "Did not notice or did not happen" and 6 to "Very obvious" . . . . 125 8.4 Mean FoV size fordynamic-FoVacross time, using a binning of 2 seconds . 129 B.1 Median Scores for IPQ and GEQ components . . . 207 B.2 Clustered column charts for participants scores in relation to the semi-structured

interviews questions . . . 212 B.3 IVRUX data mapping the plot points of the scene, coded alphabetically from

A to S . . . 213 C.1 Screenshot for Video 1 in online repository . . . 215 C.2 Screenshot for Video 2 in online repository . . . 216 D.1 Parallel coordinate plots for mean scores, considering the upper 75th per-

centile subpopulation . . . 218 D.2 Boxplots with absolute (SSQ-PreTS and SSQ-PostTS) and relative (SSQ-TS)

for the SSQ component across conditions, considering only the upper 75th percentile subpopulation . . . 218 D.3 Time series for S2 and S3, considering only the upper 75th percentile subpop-

ulation . . . 219 D.4 Time series for accumulated distances, considering only the upper 75th per-

centile subpopulation . . . 219

L I S T O F F I G U R E S

D.5 Parallel coordinate plots for mean scores, considering only the female subpop-

ulation . . . 220

D.6 Boxplots with absolute (SSQ-PreTS and SSQ-PostTS) and relative (SSQ-TS) for the SSQ component across conditions, considering only the female subpopu- lation . . . 221

D.7 Time series for S2 and S3, considering only the female subpopulations . . . 222

D.8 Time series for accumulated distances, considering only the female subpopu- lation . . . 222

D.9 Thumbnail grid for Video 1 in online repository . . . 223

D.10 Thumbnail grid for Video 2 in online repository . . . 224

D.11 Thumbnail grid for Video 3 in online repository . . . 225

E.1 Mean scores and standard errors for IPQ components for the full population (top) and upper 75th percentile subpopulation (bottom) . . . 227

E.2 Boxplots for SSQ and VRSQ components for the full population (top) and upper 75th percentile subpopulation (bottom). . . 228

E.3 Thumbnail grid for Video 1 in online repository . . . 229

E.4 Thumbnail grid for Video 2 in online repository . . . 230

1.1 Research questions . . . 4

1.2 Research contributions in HCI [WK16] . . . 5

1.3 Research problems types and subtypes in HCI [OH16] . . . 6

1.4 Research contributions. Publications are ordered in terms of significance to- wards the overall research . . . 7

1.5 Research contributions connected to chapter 5. . . 10

2.1 Evolution of technology from invention to maturity, and links to Interaction Design (IxD) and key research approaches (adapted from Grudin [Gru17, p. 122]) . . . 17

2.2 HCI Paradigms (adapted from [HTS07]) . . . 18

2.3 Roles of theory applied to HCI [BS03; Rog05; Rog12] . . . 20

2.4 Theories and approaches of HCI Periods [Rog12]. See appendix A for a de- tailed description . . . 20

2.5 Reality-Based Interaction(RBI) framework [Jac+08] . . . 21

2.6 Sensory Receptors classification [Kan+13, p. 460] . . . 27

2.7 Artificial sensory stimulation [Mar+19] . . . 28

2.8 Grand Challenges in HCI [Shn+16; Ste+19] . . . 36

2.9 Key Barriers in Authoring AR/VR Applications (adapted from [Ash+20]) . 36 3.1 Different dimensions of guiding methods and possible values [RBH19] . . 43

6.1 Audio Spatialization for the different levels of Spatial Manipulation (C1, C2, and C3) . . . 72

6.2 Measures for chapter 6 . . . 77

6.3 Mean scores and standard deviations for IPQ and PANAS components across conditions forCityandSafarivideo . . . 79

7.1 Factorial Design with two independent variables. The color scheme of the combination cells are replicated across the chapter . . . 90

L I S T O F TA B L E S

7.2 Roller coaster subsegments with track description (see fig. 7.4) and optical flow (see fig. 7.2) . . . 96 7.3 Measures for chapter 7 . . . 98 7.4 Mean scores and standard deviations for IPQ and SSQ components across

conditions, considering the full population . . . 99 7.5 Mean scores and standard deviations for distance and distance to POI for S2

and S3, considering the full population . . . 105 8.1 Manual classification of the level of motion in the video used in the study . 117 8.2 Measures for chapter 8 . . . 121 8.3 Median scores and standard deviations for SSQ and VRSQ components across

conditionsUnrestricted-FoV,Dynamic-FoV, andFixed-FoV, considering the full population (top) and upper 75th percentile subpopulation (bottom) . . . . 123 8.4 Non-parametric tests for SSQ and VRSQ components across conditionsUnrestricted-

FoV,Dynamic-FoV, andFixed-FoV, considering the full population (top) and upper 75th percentile subpopulation (bottom). . . 123 8.5 Mean scores and standard error for IPQ components across conditions, con-

sidering the full population (top) and upper 75th subpopulation percentile (bottom) . . . 124 B.1 Four scenarios resulting from the combination of two variables: location (ex-

istence and absence of links between the physical location and the virtual location) and medium (tablet and HMD) . . . 201 B.2 Comparisons of mean values of relevant measures according to the variable

location. N=20 . . . 202 B.3 Comparisons of mean values of relevant measures according to the variable

medium. N=20 . . . 202 B.4 Comparisons of mean values of relevant measures according to the combina-

tions of both variableslocationandmedium . . . 202 B.5 Semi-Structured Interview Questions . . . 211 D.1 Mean scores and standard deviations for IPQ and SSQ components across

conditions, considering only the upper 75th percentile subpopulation . . . 217 D.2 Mean scores and standard deviations for distance and distance to POI for S2

and S3, considering only the upper 75th percentile subpopulation . . . 217 D.3 Mean scores and standard deviations for IPQ and SSQ components across

conditions, considering only the female subpopulation . . . 220 D.4 Mean scores and standard deviations for distance and distance to POI for S2

and S3, considering only the emale subpopulation . . . 221

E.1 Mean scores and standard errors for SSQ and VRSQ components across con- ditions unrestricted-FoV(u-FoV), dynamic-FoV(d-FoV), and fixed-FoV(f-FoV), considering the full population (top) and upper 75th percentile subpopula- tion (bottom) . . . 226

A c r o n y m s

2D Two-Dimensional

3D Three-Dimensional

ACE Advances in Computer Entertainment Technology ACM Association for Computing Machinery

AI Artificial Intelligence ANOVA Analysis of variance

AoS Awareness of Surroundings

AR Augmented Reality

ART Aligned Rank Transform

AV Augmented Virtuality

CHI Conference on Human Factors in Computing Systems COVID-19 Coronavirus Disease 2019

CS Computer Systems

CSCW Computer Supported Cooperative Work

CSV Comma Separated Values

CVR Cinematic Virtual Reality

ER Experienced Realism

FoL Fragments of Laura

FoV Field of View

GEQ Game Experience Questionnaire

GP General Presence

GPS Global Positioning System GUI Graphical User Interface

HCI Human-Computer Interaction

HF Human Factors

HMD Head-Mounted Display

ICIDS International Conference on Interactive Digital Storytelling IEEE Institute of Electrical and Electronics Engineers

IMX ACM International Conference on Interactive Media Experiences

INV Involvement

IPQ Igroup Presence Questionnaire iqr Interquartile range

IS Information Systems

ISMAR International Symposium on Mixed and Augmented Reality IVB Independent Visual Background

IVRUX Immersive Virtual Reality User Experience

IxD Interaction Design

IxDA Interaction Design Association IxDF Interaction Design Foundation LAMS Location Aware Multimedia Stories LIS Library and Information Science

M Mean

MANOVA Multivariate Analysis of Variance

mdn Median

MNVC Most Noticed Visual Change

MR Mixed Reality

MSSQ-Short Motion Sickness Susceptibility Questionnaire Short-form

MUM International Conference on Mobile and Ubiquitous Multimedia

NA Negative Affect

NIME New Interfaces for Musical Expression

NS Navigation Style

NTS Narrative Transportation Scale

PA Positive Affect

PANAS Positive and Negative Affect Schedule POI Point of Interest

RBI Reality-Based Interaction

AC R O N Y M S

r-FoV restricted-FoV

RQ Research Question

RtD Research through Design

S[1-4]-DP Accumulated distance to POI for segment [1-4]

S[1-4]-D Accumulated distance for segment [1-4]

SD Standard Deviation

SIGCHI Special Interest Group on Computer-Human Interaction SII Sensory and Imaginative Immersion

SLAM Simultaneous Localization and Mapping SMNVC Second Most Noticed Visual Change

SP Spatial Presence

SSQ Simulator Sickness Questionnaire SSQ-D Disorientation

SSQ-O OculoMotor

SSQ-N Nausea

SSQ-TS Total Severity

SSQ-PostTS Absolute Total Severity after exposure SSQ-PreTS Absolute Total Severity before exposure

UX User Experience

UXD User Experience Design

UXPA User Experience Professionals Association

VE Virtual Environment

VIMS Visually Induced Motion Sickness

VO Visual Odometry

VR Virtual Reality

VRSQ Virtual Reality Sickness Questionnaire VRSQ-TS Total Severity

VRSQ-O OculoMotor VRSQ-D Disorientation

WIMP Windows, icons, menus, and pointing XML Extensible Markup Language

XR Extented Reality

xVel Optical Flow vector component yVel Optical Flow vector component

360º video In this thesis, 360º video, also known as immer- sive video, spherical video, omnidirectional video, or panoramic video, is a VR form in which real-world scenes (recorded using omnidirectional cameras) or computer-generated scenes (rendered), are mapped to a geometry (normally, a sphere) for playback. The user experiences the playback from the center of the geome- try.

approach This thesis follows Rogers’s definition of "approach" as a

"set of interrelated concepts and/or a set of specific ques- tions that is intended to inform a particular domain area, [...] or an analytic method [...]" [Rog12, p. 4]. This am- biguity in form is expressed through "steps, questions, concepts, challenges, principles, tactics and dimensions"

[Rog12, p. 4].

artifact Artifacts (or artefacts) are objects created or manipu- lated by humans (in opposition to natural objects that have suffered no intervention). InHCI, the term is used to describe outcomes of generative design-driven activi- ties such as systems, tools, architectures, and techniques [WK16].

attention guidance In this thesis, attention guidance refers to an interven- tion on the participant’s experience that tries to (re)focus their attention to a selected element of the experience.

G L O S S A RY

design To "design" is to "to create, fashion, execute, or con- struct according to plan" [Mer]. While some definitions accentuate the outcome of the design (e.g., "a created object, a process, and an action. The entire creation of a VR experience from information gathering and goal setting to delivery and product enhancement/support"

[Jer15c]), in this thesis, I consider not only the artifacts that emerge, but also the user activities they enable and the experience formed by the interaction between user and artifacts.

experience In the context ofHCI, "experience" emerges "from the integration of perception, action, motivation, and cog- nition into an inseparable, meaningful whole” [Has10].

Therefore, "experience is a story, emerging from the dia- logue of a person with her or his world through action"

and is subjective, holistic, situated, dynamic, and worth- while [Has10].

extented reality The termExtented Reality (XR)has been used as an um- brella term forAR/VR/CVR/MRsystems [MT19]. As such, it can be considered a superset of different forms of the Virtuality Continuum [MK94]. A common need for spatial computing between these forms has been the basis for OpenXR ¹, an open royalty-free standard for platforms and systems.

framework This thesis follows Rogers’s definition of "framework"

as a "simplification of some aspect ofHCI, intended to make it easier for designers to predict and evaluate alter- native designs" [Rog12, p. 4].

human-computer interaction HCIis defined by theAssociation for Computing Ma- chinery (ACM)’s Special Interest Group on Computer- Human Interaction (SIGCHI)curriculum as "a discipline concerned with the design, evaluation and implementa- tion of interactive computing systems for human use and with the study of major phenomena surrounding them" [Hew+92]. Human-Computer Interaction (HCI) has been portrayed as "moving target" [Gru12], since its been expanded over time to include all aspects of human life through computing [Rog12, p. xi]. HCIis also used to describe the interdisciplinary researchers applying its practices, and to refer to a number of selectedAssocia- tion for Computing Machinery (ACM)conferences and journals, in particular, the premierConference on Hu- man Factors in Computing Systems (CHI).

immersion In this thesis, "immersion" is the degree to which a sys- tem stimulates the sensory system without interference from the external environment, therefore describing the technical capabilities of a system [BL95].

interaction This thesis follows Hornbæk and Oulasvirta’s definition of interaction concerning "two entities that determine each other’s behavior over time " [HO17]. InHCI, these entities are either users or artifacts (e.g., a computing system), but "users, with their goals and pursuits, are the ultimate metric of interaction"[HO17]. Through an analysis of 35 years ofCHIproceedings, Hornbæk et al.

[Hor+19] note the close relationship between styles of interaction and technology, but accentuate that styles or qualities (e.g., structure, feel, effectiveness, and effi- ciency) of interaction are taking an increasingly promi- nent role inHCIdiscourse.

G L O S S A RY

interaction design In this thesis, the term "interaction design" (IxD) is used as a discipline of design overlapping other disciplines of design (e.g., Communication Design, Architecture, among others). Emerging fromHCI,IxDwas prompted by moving away from the concept of single-user and computer [Rog12, p. 3], and applying design thinking approaches [Ben19b, p. xv]. Practitioners ofIxD orga- nize themselves in local chapters of theInteraction De- sign Association (IxDA), supported with conferences (In- teraction and Interaction Latin America). Furthermore, organizations like the Interaction Design Foundation (IxDF)provide open-source educational literature and courses for members.

model This thesis follows Rogers’s definition of "model" as a

"well-substantiated explanation of some aspect of a phe- nomenon" [Rog12, p. 4]. For example, the model human processor [CMN86] is a cognitive model focusing on the abstraction of information flow.

paradigm In this thesis, "paradigm" may be used in two ways:

• in the Kuhnian sense, where science happens between paradigms; a paradigm shift is the re- sult of a scientific revolution, where the domi- nant paradigm is deemed incompatible, a new paradigm is adopted and normal science proceeds [Kuh96].

• as used in HCI, where a paradigm is a "general approach that has been adopted by a community of researchers and designers for carrying out their work, in terms of shared assumptions, concepts, values and practices" [Rog12, p. 4].

participant InHCI, a person is commonly referred to as a user since they are using an artifact, system, or service. InXR, the user is sometimes referred to as a participant because they are participating in an extended reality. For exam- ple, Slater and Sanches-Vives posit that "VRis different from other forms of human-computer interface since the human participates in the virtual world rather than uses it" [SS16]. Furthermore, some frameworks like theTra- jectories framework define "participant" as "a member of the public who is the main target for the experience"

[BG11]. In this thesis, "participant" may refer to two non-exclusive meanings:

• Participant as a person who has participated in a study (e.g. in chapters in PartII)

• Participant as a user that is participating in a expe- rience mediated by aXRsystem.

perceptual system While sensation is "simple stimulation of a sense or- gan" [Sch+19, p. 237], perception is the "organization, identification, and interpretation of a sensation in order to form a mental representation" [Sch+19, p. 237]. As such, perceptual systems are systems (either biological or artificial) that transduce information from at least one sensory organ.

presence Although multiple definitions exist, depending on the field of origin, in this thesis, "presence" is the "subjective experience of being in one place or environment, even when one is physically situated in another" [WS98].

theory This thesis follows Rogers’s definition of "theory" as a "theoretical approach or perspective that refers more generally to the assumptions about a phenomenon being studied or designed for, the lenses used to analyze it and the questions asked, that are grounded in a theoretical tradition, for example, within social psychology, design or engineering" [Rog12, p. 4].

G L O S S A RY

user experience design In this thesis, the term "user experience design" (UXD) is used as a discipline of design overlapping other disci- plines of design (e.g., Industrial Design, Architecture, etc.). Emerging from HCI, UXD was prompted by new interfaces (e.g., web, tangible, spatial computing, among others) and the need to make the experience of using these interfaces enjoyable and engaging [Ben19b, p. xv]. Practitioners ofUXDorganize themselves in local chapters of User Experience Professionals Association (UXPA), supported with an annual international confer- ence and a journal (Journal of Usability Studies).

user experience In this thesis, the term "user experience" (UX) is used to refer to the experience created and shaped through tech- nology. In this sense, technology (interactive systems) act as mediators of the experience [Has10].

virtual reality Definitions ofVirtual Reality (VR), such as "A computer- generated digital environment that can be experienced and interacted with as if that environment were real"

[Jer15c], follow idealized visions of what such systems can be (e.g., [Con+89; Har98;Mor93]). Current imple- mentations of these visions are limited by technical ca- pabilities ofXRsystems. In this thesis, unless specified, the term "virtual reality" refers to computer-generated digital environments experienced through systems such as a head-mounted display. These computer-generated digital environments can be purely virtual or can be cap- tured from the real world (e.g.,360º video).

wave In this thesis, "wave" is used following Bødker’s [Bød06;

Bød15] characterization ofHCIas waves. This grouping ofHCIwork as waves is expressive of how research ques- tions emerge, how methods are introduced, and how challenges are addressed [Ban11]. Waves are not about setting a standard practice but rather representing how the practice of research inHCIevolves.

1https://www.khronos.org/openxr/

G L O S S A RY

1

I n t r o d u c t i o n

This chapter addresses the motivation behind this research work, an overview of the goals and questions ushering the research process, and the main contributions that arise from it.

1.1 Motivation

While Extented Reality (XR)systems have been a focus of research since the early days of computing [Sut68], previous attempts of commercialization failed to gain traction [Jer15a]. Due to technology barriers, these systems failed to live up to idealized systems set by literature, film, and researchers’ computing visions [Jer15a]. Recently, new and more robust XR systems have rekindled the expectations and financial investment for XRas a computing platform [BBB19]. Subsequently, this has led to an increase in the availability of XR systems for practitioners and researchers alike. Many practitioners, exploring a new rich design space, are now left dealing with design hurdles of a space that is very different from what they are accustomed to [Ash+20]. Human-Computer Interaction (HCI)researchers have addressed this by not only focusing on the usability issues connected toXRsystems [McG+15], but also extending research on how to design and evaluate theUser Experience (UX)for these platforms [Ash+20].

On a personal note, starting this research, I was particularly interested in how emer- gent technologies supported Digital Storytelling, specifically Immersive Journalism using 360º videoas a medium [Peñ+10]. While360º video’s potential for Digital Storytelling was discussed in academia and industry, it was also often described as a lesser form of Virtual Reality (VR), static and limited, or as a translation of film into a new medium [Buc18;Mat17;SS16]. Additionally, usability issues ofVRwere still present in360º video and many methods that addressed them were not compatible with360º video. This moti- vated me to research the interplay between different forms ofXRand how these could be applied to uplift360º video.

1 . 2 . R E S E A R C H G OA L S A N D Q U E S T I O N S

Combining previous work fromHCIresearchers onXRand personal motivation, this research work is motivated by a desire to improve and augment the experience of watch- ing360º video. Through the creation ofartifactsand evaluation studies, this research work is applicable for both researchers and practitioners working on360º video.

1.2 Research Goals and Questions

The aim for this research is to explore how to leverage knowledge from the creation of XR artifacts to improve user experience in 360º video. As seen in figure 1.1, the relevance and rationale for this research are grounded in the knowledge gaps and research challenges within the area of HCI,perceptual systems, andXR. This research scope is further described and discussed in chapter2.

XR

Perceptual Systems

Human Computer Interaction VR _360º

Video

User Experience in 360º Video

Figure 1.1: Research scope

Based on previous work on360º video(see subsection2.3.3), two areas of action were identified where the experience of watching a360º videocan be augmented: Attention GuidanceandVisually Induced Motion Sickness (VIMS). Each of these areas of action are further described in chapter3and4. Based on the related work in the areas, the research questions (RQ1-RQ4) in Table1.1were defined.

Table 1.1: Research questions Area Research Question

Attention Guidance (chapter3)

RQ1 -Can spatialized music guide viewers’ attention in360º video?

RQ2 -Can the use of diegetic cues be reinforced by the audio spatialization to guide viewers in360º video?

VIMS (chapter4)

RQ3 -Can optical flow of a360º videobe used to mitigateVIMS?

RQ4 -Can the combination of differentVIMS-mitigating techniques forVR be sucessful in360º video?

1.3 Research Overview/Methodology

Parallel to the initial secondary research (chapters2-4), I collaborated in the develop- ment ofFragments of Laura (FoL), a location-based transmedia experience, using mobile VR, that was part of another student’s Digital Media Ph.D [Dio20]. During this period, I developed and tested different iterations of aVRandMRscene and remediation of the location-based tour as aVRexperience. While this work does not necessarily apply to the research questions mentioned in table1.1, the practical and methodological knowledge gained from this involvement helped shape this research. More importantly, during this period, I developedIVRUX[Bal+16], a tool for the analysis of immersiveVRstory-driven experiences. This tool helped contextualize user behavior with the narrative (story events, audio events, character events, among others). Both FoLandIVRUX were developed considering a Research through Design (RtD)approach [ZFE07], where the design arti- facts are containers for design knowledge and can help determine the research direction.

Thisartifactsystem was the base for subsequent studies in360º video, and therefore is described in chapter5.

To answer the research questions mentioned in table 1.1, I developed artifact sys- tems and empirical evaluation studies using those systems. Forattention guidance, this resulted in Cue Control [Bal+19], an audio cue spatializer described in chapter6. For VIMS, this resulted in a pipeline (pre-computed and runtime) using optical flow (de- scribed in chapter7), and iteration of the pipeline using optical flow andSimultaneous Localization and Mapping (SLAM) (described in chapter 8). Studies on these systems involved a mixed-methods approach, using self-reported measures, objective measures, and interviews.

Since the work developed in chapters5,6,7, and8, was the result of collaborations or resulting from a multidisciplinary team, the pronoun "we" is used. In chapter5, my contribution is specified in footnotes. In the remaining chapters, my contribution extends to the whole work described in those chapters: development of artifacts, planning of evaluation studies, analysis and discussion of results.

1 . 4 . C O N T R I B U T I O N S A N D P U B L I C AT I O N S

1.4 Contributions and Publications

Since the overall framing of this work is in the field ofHCI, this section reflects firstly on two examples of how knowledge produced is discussed and presented inHCI.

Firstly, based on contribution types for the CHI conference from 2006 to 2016, Wob- brock and Kientz identified seven contribution types in HCI, how they are judged and example forms of each [WK16]. Table1.2shows a synthesis of contribution types: empir- ical, artifact, methodological, theoretical, dataset, survey, and opinion.

Table 1.2: Research contributions in HCI [WK16]

Contribution Description

Empirical

Arises from descriptive discovery-driven activities and provides new knowledge through findings based on observation and data- gathering (e.g., interviews, diaries, quantitative lab studies, crowd- sourced studies, qualitative field studies, etc.).

Artifact

Arises from generative design-driven activities and provides new knowledge embedded in and manifested byartifacts(e.g., systems, tools, architectures, techniques, etc.) and the supporting materials that describe them or evaluate them.

Methodological

Creates new knowledge that informs and improves research or prac- tice (such as new methods, methods adaptation, new instruments, new measures, etc.).

Theoretical

Provides theoretical knowledge in the form of concepts, definitions, models, principles, or frameworks, with descriptive and/or predic- tive powers, that inform what we do, why we do it, and what to expect.

Dataset Provides knowledge as a representative corpus, which is of interest to the research community.

Survey Provides reviews and synthesis of work done on a specific research topic, highlighting trends, patterns, and gaps.

Opinion Provides persuasive and informative arguments on a topic, com- pelling reflection, discussion, and debate.

Secondly, Oulasvirta and Hornbæk frameHCIas a problem-solving discipline [OH16], based on Laudan’s view of problem and solution as the foundational concepts of science [Lau78]. A problem is defined via inabilities and absences occurring in descriptions, knowledge, or constructive solutions. Laudan, based on natural and social sciences, de- scribes only empirical and conceptual problems; Oulasvirta and Hornbæk include con- structive problems to account for engineering and design, therefore, problems in HCI are empirical, conceptual and design/constructive problems [OH16]. A solution (even if partial) changes the inabilities/absences and therefore improves the problem-solving capacity [OH16]. These solutions can be evaluated [OH16] according to their:

• Significance - does the solution solve a problem of interest to researchers, practi- tioners, or users?

• Effectiveness - does the solution solve the essential aspects of the problem?

• Efficiency - is the cost of creating or deploying the solution greater than its gains?

• Transfer - is the solution transferable to other problems?

• Confidence - is the proposed solution likely to hold?

Oulasvirta and Hornbæk present three types of research withinHCI(see table1.3):

empirical research (concerned with the description of phenomena occurring from the use of computing), conceptual research (concerned with explaining phenomena occur- ring from interaction), and constructive research (concerned with the construction of interactiveartifacts) [OH16]. HCIoften involves multiple problem types and types of research can be pairwise mixed (e.g., an empirical study with implications for design is empirical–constructive) [OH16]. Contribution pairing is also described in Wobbrock and Kientz [WK16].

Table 1.3: Research problems types and subtypes inHCI[OH16]

Research Subtypes

Empirical- Creating or elaborating descriptions of real-world phenomena related to human use of computing

Unknown phenomena Unknown factors Unknown effects Conceptual- Explaining previously

unconnected phenomena occurring in interaction

Implausibility - phenomenon is unrea- sonable, improbable, or lacking an expla- nation

Inconsistency- a position is inconsistent with data, with itself, or with some other position

Incompatibility - two positions have as- sumptions that cannot be reconciled Constructive- Producing understanding

on construction of an interactiveartifact for some purpose in human use of computing

No known solution- work on novel con- cepts for interaction

Partial, ineffective, or inefficient solu- tion- work on improved solutions

Insufficient knowledge or resources for implementation or deployment - work on not yet implemented solutions

While Wobbrock and Kientz [WK16] and Oulasvirta and Hornbæk [OH16] approaches have different origins (one that is more practice-based versus another that is more con- ceptual), both highlight the role of artifactsor constructive research in HCI. For this dissertion, contributions are inline with most work done inVR[BCK17] with construc- tive research onartifacts:

1 . 4 . C O N T R I B U T I O N S A N D P U B L I C AT I O N S

• Adopting Wobbrock and Kientz [WK16]’s view, the main contribution of my re- search is anartifact, since its describes systems and techniques developed; the minor contribution is therefore asempiricalresearch with its quantitative lab studies.

• Adopting Oulasvirta and Hornbæk [OH16]’s view, the main contribution of my research is constructive, as it provides knowledge through the construction of an interactiveartifact, with the subtype ofPartial, ineffective, or inefficient solution, since it is trying to improve existing solutions.

These contributions are supported by framing 360º video/XRsystems within HCI (see chapter2). This framing and the work developed contribute by placing360º video as rich design space for interaction, instead of the passive video format it is sometimes described at.

Different research contributions described in this thesis were published in peer-reviewed research articles as follows. Table1.4presents contributions that are directly connected to the research theme and are ordered in terms of significance.

Table 1.4: Research contributions. Publications are ordered in terms of significance towards the overall research

Publication Description

[Bal+19] P.Bala, R. Masu, V. Nisi, and N.

Nunes. “"When the Elephant Trumps": A Comparative Study on Spatial Audio for Orientation in 360º Videos”. In: Proceed- ings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 04-09, 2019.

CHI ’19. New York, NY, USA: ACM, 2019.

i s b n: 978-1-4503-5970-2. d o i: 10.1145 /3290605.3300925

Full paper at CHI (SIGCHI main con- ference) describing Cue Control, an ar- tifact system for audio spatialization in 360º videosand accompanying evaluation study. This research contribution is de- scribed in chapter6.

[Bal+20b] P. Bala, I. Oakley, V. Nisi, and N. Nunes. “Staying on Track: a Compar- ative Study on the Use of Optical Flow in 360º Video to Mitigate VIMS”. in:Proceed- ings of the ACM International Conference on Interactive Media Experiences. IMX 2020, Barcelona, Spain, June 17-19, 2020. ACM, 2020, pp. 82–93. d o i: 10.1145/3391614 .3393658.u r l:https://doi.org/10.11 45/3391614.3393658

Full paper atIMX(SIGCHIsponsored con- ference) describing the first iteration of pipeline using optical flow for controlling techniques known to mitigate VIMSand accompanying evaluation study. This re- search contribution is described in chap- ter7.

Continued on next page

Table 1.4: continued from previous page

Publication Description

[Bal+21] P. Bala, I. Oakley, V. Nisi, and N. Nunes. “Dynamic Field of View Re- striction in 360º Video: Aligning Optical Flow and Visual SLAM to Mitigate VIMS”.

in:Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.

CHI ’21. Yokohama, Japan: Association for Computing Machinery, 2021. d o i: 1 0. 1145 /3411764 . 3445499. u r l: https : //doi.org/10.1145/3411764.3445499

Full paper atCHI (SIGCHI main confer- ence) describing the second iteration of pipeline using optical flow andSLAMfor controlling restrictedFoVknown to miti- gateVIMSand accompanying evaluation study. This research contribution is de- scribed in chapter8.

[Bal+16] P.Bala, M. Dionísio, V. Nisi, and N. Nunes. “IVRUX: A Tool for Analyz- ing Immersive Narratives in Virtual Real- ity”. en. In: Interactive Storytelling. Ed.

by F. Nack and A. S. Gordon. Vol. 10045.

Lecture Notes in Computer Science 10045.

Springer International Publishing, Nov.

2016, pp. 3–11. i s b n: 978-3-319-48279-8.

d o i:10.1007/978-3-319-48279-8\_1

Short paper atICIDS describing IVRUX, anartifactsystem fo viewing and explor- ing syntetized recorded data from user’s experience in a HMD narrative. This ar- tifact system was used as basis for the previous publications [Bal+19; Bal+20b;

Bal+21]. This research contribution is de- scribed in chapter5and an evaluation of the IVRUXprototype is described in ap- pendix sectionB.3.

[Bal+18a] P. Bala, D. Dionisio, V. Nisi, and N. Nunes. “Visually Induced Mo- tion Sickness in 360º Videos: Comparing and Combining Visual Optimization Tech- niques”. In:IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2018 Adjunct, Munich, Germany, October 16-20, 2018. IEEE, 2018, pp. 244–249.

d o i: 10 . 1109 / ISMAR - Adjunct . 2018 . 0 0077

Poster atIEEE ISMARon first iteration of pipeline using optical flow for controlling techniques known to mitigate VIMSand a pilot evaluation study, that later led to [Bal+20b].

Continued on next page

1 . 4 . C O N T R I B U T I O N S A N D P U B L I C AT I O N S

Table 1.4: continued from previous page

Publication Description

[Bal+18b] P.Bala, R. Masu, V. Nisi, and N.

Nunes. “Cue Control: Interactive Sound Spatialization for 360º Videos”. In: Inter- active Storytelling - 11th International Con- ference on Interactive Digital Storytelling, ICIDS 2018, Dublin, Ireland, December 5- 8, 2018, Proceedings. Ed. by R. Rouse, H.

Koenitz, and M. Haahr. Vol. 11318. Lec- ture Notes in Computer Science. Springer, 2018, pp. 333–337. d o i:10.1007/978-3- 030-04028-4\_36

Poster at ICIDS describing an initial im- plementation ofCue Controlartifactand a pilot evaluation, that later led to [Bal+19].

[BNN17] P. Bala, V. Nisi, and N. Nunes.

“Evaluating User Experience in 360º Sto- rytelling Through Analytics”. In: Interac- tive Storytelling - 10th International Con- ference on Interactive Digital Storytelling, ICIDS 2017, Funchal, Madeira, Portugal, November 14-17, 2017, Proceedings. Ed.

by N. Nunes, I. Oakley, and V. Nisi.

Vol. 10690. Lecture Notes in Computer Science. Cham: Springer, 2017, pp. 270–

273. i s b n: 978-3-319-71026-6 978-3-319- 71027-3. d o i: 10.1007/978-3-319-7102 7-3\_23. (Visited on 10/12/2018)

Poster at ICIDS on methodology of ob- serving user experience on 360º video, in- cluding a proof-of-concept prototype iter- ation ofIVRUX, which was later used for [Bal+18b] and [Bal+19]. This is described in section5.2.3.

[Mas+20] R. Masu, P. Bala, M. A. Ah- mad, N. Do Nascimento Correia, V. Nisi, N. Nunes, and T. Romão. “VR open scores:

scores as inspiration for VR scenarios”. In:

NIME New Interfaces for Musical expres- sion. 2020

Full paper at NIME on aleatoric score- based virtual scenarios, where an aleatoric score is embedded in a virtual environ- ment. This publication describes two sce- narios, one of those usingCue Control.

Table1.5presents contributions emerging from research done on transmedia story- telling (described in chapter5). Although not central to the main theme of the work, these contributions resulted in artifacts that shaped the previous contributions, namely the development ofIVRUX.

Table 1.5: Research contributions connected to chapter5.

Publication Description

[Dio+16] M. Dionísio, V. Nisi, N. J. Nunes, and P.Bala. “Transmedia Storytelling for Exposing Natural Capital and Promoting Ecotourism”. In: Interactive Storytelling - 9th International Conference on Interactive Digital Storytelling, ICIDS 2016, Los Ange- les, CA, USA, November 15-18, 2016, Pro- ceedings. Ed. by F. Nack and A. S. Gordon.

Vol. 10045. Lecture Notes in Computer Science. 2016, pp. 351–362. d o i: 10.100 7/978-3-319-48279-8\_31

Full paper at ICIDS on the use of trans- media storytelling for engaging users in thinking of ecosystems and their underly- ing biodiversity. This publication includes a initial description of the experience pro- totype of Fragments of Laura, a location- based transmedia experience.

[Dio+17b] M. Dionísio, P. Bala, V. Nisi, and N. Nunes. “Fragments of laura: in- corporating mobile virtual reality in lo- cation aware mobile storytelling experi- ences”. In: Proceedings of the 16th Inter- national Conference on Mobile and Ubiq- uitous Multimedia, MUM 2017, Stuttgart, Germany, November 26 - 29, 2017. Ed.

by N. Henze, P. Wozniak, K. Väänänen, J. Williamson, and S. Schneegass. ACM, 2017, pp. 165–176. d o i:10.1145/315283 2.3152868

Full paper at MUM (SIGCHI in- cooperation conference) on the de- scription and evaluation of Fragments of Laura, a location-based transmedia experience, incorporating mobile VR environments.

[Dio+17a] M. Dionisio, P. Bala, V. Nisi, I. Oakley, and N. Nunes. “Step by Step:

Evaluating Navigation Styles in Mixed Re- ality Entertainment Experience”. In: Ad- vances in Computer Entertainment Technol- ogy - 14th International Conference, ACE 2017, London, UK, December 14-16, 2017, Proceedings. Ed. by A. D. Cheok, M. In- ami, and T. Romão. Vol. 10714. Lec- ture Notes in Computer Science. Springer, 2017, pp. 32–45. d o i: 10.1007/978-3-3 19-76270-8\_3

Full paper atACEon navigation styles for the second iteration of theThe Old Phar- macy, a virtual environment inFragments of Laura. This study is described in ap- pendix sectionB.2.

Continued on next page

1 . 4 . C O N T R I B U T I O N S A N D P U B L I C AT I O N S

Table 1.5: continued from previous page

Publication Description

[Dio+18] M. Dionísio, P. Bala, V. Nisi, and S. Câmara. “Bringing Locative Me- dia Indoors: Strategies For Remediation”.

In: Looking Forward, Looking Back: Inter- active Digital Storytelling and Hybrid Art Approaches. Carnegie Mellon University:

ETC Press, Pittsburgh, PA, Jan. 1, 2018, pp. 71–92. u r l: https : / / press . etc . cmu.edu/index.php/product/looking- forward-looking-back/

Book chapter on the remediation of trans- media artifacts for physical exhibitions, in- cluding remediating the locative tour of Fragments of Laura asVR. This remedia- tion is described in section5.1.3.

[Bal+17] P.Bala, M. Dionísio, R. Trindade, S. Olim, V. Nisi, and N. Nunes. “Evaluat- ing the influence of location and medium applied to mobile VR storytelling”. In:

Proceedings of the 16th International Confer- ence on Mobile and Ubiquitous Multimedia, MUM 2017, Stuttgart, Germany, Novem- ber 26 - 29, 2017. Ed. by N. Henze, P.

Wozniak, K. Väänänen, J. Williamson, and S. Schneegass. ACM, 2017, pp. 371–378.

d o i: 10 . 1145 / 3152832 . 3156617. u r l: https://doi.org/10.1145/3152832.31 56617

Poster at MUM (SIGCHI in-cooperation conference) describing the pilot study of a first iteration of The Old Pharmacy, ex- perienced in physical locations with and without links to the narrative location and with different platforms (tablet andHMD).

This pilot study is described in appendix sectionB.1.

Finally, the following contributions were also accomplished during the research pe- riod but are not connected to the main research theme. Since these explore alternative research threads, they are not described in this document.

[Nis+19] V. Nisi, M. S. Dionisio, P.Bala, T. Gross, T. Up, and N. J. Nunes. “Lucid Peninsula, a Physical Narrative Art Installation Comprising Interactive 360º Virtual Reality Components”. In:IJCICG10.1 (2019), pp. 1–15.d o i: 10.40 18/IJCICG.2019010101.

[Bal+20a] P.Bala, M. Dionísio, T. Andrade, and V. Nisi. “Tell a Tail 360º: Immersive Storytelling on Animal Welfare”. en. In:Interactive Storytelling. Ed. by A.-G.

Bosser, D. E. Millard, and C. Hargood. Vol. 12497. Series Title: Lecture Notes in Computer Science. Cham: Springer International Publishing, 2020, pp. 357–360. i s b n: 978-3-030-62515-3 978-3-030-62516-0. d o i: 10.1007 /978-3-030-62516-0\_35.

[Bal+20b] P.Bala, M. Dionísio, S. Oliveira, T. Andrade, and V. Nisi. “Tell a Tail: Lever- aging XR for a Transmedia on Animal Welfare”. en. In: Entertainment Computing – ICEC 2020. Ed. by N. J. Nunes, L. Ma, M. Wang, N. Correia, and Z. Pan. Vol. 12523. Series Title: Lecture Notes in Computer Science.

Cham: Springer International Publishing, 2020, pp. 223–239. i s b n: 978-3- 030-65735-2 978-3-030-65736-9. d o i:10.1007/978-3-030-65736-9\_19.

[Dio+20] M. Dionísio, P.Bala, S. Oliveira, and V. Nisi. “Tale of T(r)ails: The Design of an AR Comic Book for an Animal Welfare Transmedia”. en. In: Interactive Storytelling. Ed. by A.-G. Bosser, D. E. Millard, and C. Hargood. Vol. 12497.

Series Title: Lecture Notes in Computer Science. Cham: Springer Interna- tional Publishing, 2020, pp. 281–284.i s b n: 978-3-030-62515-3 978-3-030- 62516-0.d o i: 10.1007/978-3-030-62516-0\_25.

[Bal+21] P.Bala, V. Nisi, M. Dionisio, N. J. Nunes, and S. James. “Square Peg, Round Hole: A Case Study on Using Visual Question & Answering in Games”.

In: Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play. CHI PLAY ’21. Virtual Event, Austria: Association for Computing Machinery, 2021, pp. 133–139. i s b n: 9781450383561. d o i: 10.1145/3450337.3483466.

[Dio+21] M. Dionísio, V. Nisi, J. Xin, P.Bala, S. James, and N. J. Nunes. “Amnesia in the Atlantic: An AI Driven Serious Game on Marine Biodiversity”. en.

In: Entertainment Computing – ICEC 2021. Ed. by J. Baalsrud Hauge, J.

C. S. Cardoso, L. Roque, and P. A. Gonzalez-Calero. Vol. 13056. Series Title: Lecture Notes in Computer Science. Cham: Springer International Publishing, 2021, pp. 427–432.i s b n: 978-3-030-89393-4 978-3-030-89394- 1. d o i: 10.1007/978-3-030-89394-1_35.

1 . 5 . D O C U M E N T S T R U C T U R E

1.5 Document Structure

The remainder of this document is structured in two parts:

• PartIcorresponds to secondary research and contains chapters2-4on foundational concepts and related work on the research areas. Chapter2presents foundational concepts ofHCI(subsection2.1),perceptual systems(subsection2.2) andXR(sub- section2.3). Chapter3addresses the use of sound forAttention Guidance, by first looking at how sound is used inHCI(subsection3.1) and then focusing on attention and techniques used to guide users (subsection 3.2). Chapter 4adresses what is VIMS(subsection4.1) and how it can be mitigated (subsection4.2)

• PartIIcorresponds to primary research and contains chapters5-9on theartifacts developed and evaluation studies of suchartifacts. Chapter5briefly summarizes the work done inFragments of Laura, that lead to the creation ofIVRUX, used as a basis forartifactson subsequent chapters. Chapter6describesCue Control, an audio cue spatializer, addressing RQ1 and RQ2. Chapter 7describes a pipeline using optical flow to control techniques known to mitigate VIMS (independent visual background and restricted FoV), addressingRQ3 andRQ4. Chapter8describes an iteration of the previous pipeline, now using optical flow andSLAM, to control a technique known to mitigate VIMS(restricted FoV), addressing RQ3. Finally, chapter9presents a research summary and findings of the work done, delineating limitations and future directions.

Theoretical Background & Related

Work

2

T h e o r e t i c a l B a c k g r o u n d

This chapter presents foundational concepts ofHuman-Computer Interaction (HCI), Perceptual Systems, andExtented Reality (XR). Firstly, this chapter starts with an overview of the evolution ofHCI, highlighting the influence of technology,

interdisciplinary practices, and theory. Secondly, the chapter focuses on human perception, and how technology supports and replaces it. The chapter then describes fundamental concepts ofExtented Reality (XR), focusing on360º video. Lastly, we contextualize the current state of the art inXRwithinHCI, identifying research opportunities.

2.1 Human-Computer Interaction

The field ofHuman-Computer Interaction (HCI)gained prominence in the 1980s and be- came a burgeoning field of study. Fueled by fast computing technology,HCIas a field has been described as a "moving target" [Gru12], expanding its original concern of interfacing humans and machines to addressing all aspects of human life through computing [Rog12, p. xi]. Alan Dix has also singled out this expansive nature, describing "HCIas an aca- demic discipline has always been positioned, sometimes uneasily, sometimes creatively, in the tension between solid intellectual rigor and the excitement in new technology"

[Dix17]. This can also be perceived in Figure2.1where the different disciplines ofUser Experience Design (UXD)andInteraction Design (IxD)are mapped [pre21;Saf10]. The core tenet of usability (easy to use, easy to learn [Ben19b, p. xv]) inHCIis central toUXD andIxDas both fields were exports from traditionalHCIinto the world of Design. IxD was prompted by moving away from the concept of single-user and computer [Rog12, p. 3], and applying design thinking approaches [Ben19b, p. xv].UXDwas prompted by new interfaces (e.g., web, tangible, spatial computing, etc.) and the need to make the ex- perience of using these interfaces enjoyable and engaging [Ben19b, p. xv]. This expansion of focus has ledHCIto outgrow its initial concerns, become an interdisciplinary field and reflect onto itself on how it is studied, designed, and evaluated. Several researchers have

characterized this evolution throughparadigms[HTS07],waves[Bød15], circles [Gru17]

and periods [Rog12].

User Experience Design

Interaction Design Industrial

Design

Human Computer Interaction

Usability Engineering Ubiquitous

Computing

Interactive Controls Mechanical

Engineering

Electrical Engineering

Psychology Cognitive

Science

Sociology Philosophy

Human Factors

& Ergonomics Architecture

Information Architecture

Communication Design

Motion Design

Spatial Experience

Contextual Requirements

Data & Info Visualization Functional

Requirements

Generative Design Marketing

Audio Engineering Sound

Design

Writing

Computer Science

Interactive Environments

User Interface Design Application

Design

Software Development

Navigation Design

Guidance Systems

User Interface Scenography

Scenario Design Digital

Signage Media

Installations

Figure 2.1: Infographic on User Experience disciplines fromenvis precisely[pre21] (Cre- ative Commons Attribution-ShareAlike 3.0 Unported), based on original concept from Saffer [Saf10, p. 21]

2.1.1 Computing andHCI

Grudin [Gru17] adopted a historical perspective to the evolution ofHCI, by mapping landmark research and technological advances that are fundamental to modern HCI.

Through reflection on the advances in artifacts (e.g., vacuum tubes, mainframes, com- puters, mobile/hanheld, embedded devices) [Gru17, p. 55], and visionary visions (e.g., Sutherland’s concept of Computer Graphics that lead toGraphical User Interface (GUI)