1.4 Contributions and Publications
Since the overall framing of this work is in the field ofHCI, this section reflects firstly on two examples of how knowledge produced is discussed and presented inHCI.
Firstly, based on contribution types for the CHI conference from 2006 to 2016, Wob-brock and Kientz identified seven contribution types in HCI, how they are judged and example forms of each [WK16]. Table1.2shows a synthesis of contribution types: empir-ical, artifact, methodologempir-ical, theoretempir-ical, dataset, survey, and opinion.
Table 1.2: Research contributions in HCI [WK16]
Contribution Description
Empirical
Arises from descriptive discovery-driven activities and provides new knowledge through findings based on observation and data-gathering (e.g., interviews, diaries, quantitative lab studies, crowd-sourced studies, qualitative field studies, etc.).
Artifact
Arises from generative design-driven activities and provides new knowledge embedded in and manifested byartifacts(e.g., systems, tools, architectures, techniques, etc.) and the supporting materials that describe them or evaluate them.
Methodological
Creates new knowledge that informs and improves research or prac-tice (such as new methods, methods adaptation, new instruments, new measures, etc.).
Theoretical
Provides theoretical knowledge in the form of concepts, definitions, models, principles, or frameworks, with descriptive and/or predic-tive powers, that inform what we do, why we do it, and what to expect.
Dataset Provides knowledge as a representative corpus, which is of interest to the research community.
Survey Provides reviews and synthesis of work done on a specific research topic, highlighting trends, patterns, and gaps.
Opinion Provides persuasive and informative arguments on a topic, com-pelling reflection, discussion, and debate.
Secondly, Oulasvirta and Hornbæk frameHCIas a problem-solving discipline [OH16], based on Laudan’s view of problem and solution as the foundational concepts of science [Lau78]. A problem is defined via inabilities and absences occurring in descriptions, knowledge, or constructive solutions. Laudan, based on natural and social sciences, de-scribes only empirical and conceptual problems; Oulasvirta and Hornbæk include con-structive problems to account for engineering and design, therefore, problems in HCI are empirical, conceptual and design/constructive problems [OH16]. A solution (even if partial) changes the inabilities/absences and therefore improves the problem-solving capacity [OH16]. These solutions can be evaluated [OH16] according to their:
• Significance - does the solution solve a problem of interest to researchers, practi-tioners, or users?
• Effectiveness - does the solution solve the essential aspects of the problem?
• Efficiency - is the cost of creating or deploying the solution greater than its gains?
• Transfer - is the solution transferable to other problems?
• Confidence - is the proposed solution likely to hold?
Oulasvirta and Hornbæk present three types of research withinHCI(see table1.3):
empirical research (concerned with the description of phenomena occurring from the use of computing), conceptual research (concerned with explaining phenomena occur-ring from interaction), and constructive research (concerned with the construction of interactiveartifacts) [OH16]. HCIoften involves multiple problem types and types of research can be pairwise mixed (e.g., an empirical study with implications for design is empirical–constructive) [OH16]. Contribution pairing is also described in Wobbrock and Kientz [WK16].
Table 1.3: Research problems types and subtypes inHCI[OH16]
Research Subtypes
Empirical- Creating or elaborating descriptions of real-world phenomena related to human use of computing
Unknown phenomena Unknown factors Unknown effects Conceptual- Explaining previously
unconnected phenomena occurring in interaction
Implausibility - phenomenon is unrea-sonable, improbable, or lacking an expla-nation
Inconsistency- a position is inconsistent with data, with itself, or with some other position
Incompatibility - two positions have as-sumptions that cannot be reconciled Constructive- Producing understanding
on construction of an interactiveartifact for some purpose in human use of computing
No known solution- work on novel con-cepts for interaction
Partial, ineffective, or inefficient solu-tion- work on improved solutions
Insufficient knowledge or resources for implementation or deployment - work on not yet implemented solutions
While Wobbrock and Kientz [WK16] and Oulasvirta and Hornbæk [OH16] approaches have different origins (one that is more practice-based versus another that is more con-ceptual), both highlight the role of artifactsor constructive research in HCI. For this dissertion, contributions are inline with most work done inVR[BCK17] with construc-tive research onartifacts:
1 . 4 . C O N T R I B U T I O N S A N D P U B L I C AT I O N S
• Adopting Wobbrock and Kientz [WK16]’s view, the main contribution of my re-search is anartifact, since its describes systems and techniques developed; the minor contribution is therefore asempiricalresearch with its quantitative lab studies.
• Adopting Oulasvirta and Hornbæk [OH16]’s view, the main contribution of my research is constructive, as it provides knowledge through the construction of an interactiveartifact, with the subtype ofPartial, ineffective, or inefficient solution, since it is trying to improve existing solutions.
These contributions are supported by framing 360º video/XRsystems within HCI (see chapter2). This framing and the work developed contribute by placing360º video as rich design space for interaction, instead of the passive video format it is sometimes described at.
Different research contributions described in this thesis were published in peer-reviewed research articles as follows. Table1.4presents contributions that are directly connected to the research theme and are ordered in terms of significance.
Table 1.4: Research contributions. Publications are ordered in terms of significance towards the overall research
Publication Description
[Bal+19] P.Bala, R. Masu, V. Nisi, and N.
Nunes. “"When the Elephant Trumps": A Comparative Study on Spatial Audio for Orientation in 360º Videos”. In: Proceed-ings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 04-09, 2019.
CHI ’19. New York, NY, USA: ACM, 2019.
i s b n: 978-1-4503-5970-2. d o i: 10.1145 /3290605.3300925
Full paper at CHI (SIGCHI main con-ference) describing Cue Control, an ar-tifact system for audio spatialization in 360º videosand accompanying evaluation study. This research contribution is de-scribed in chapter6.
[Bal+20b] P. Bala, I. Oakley, V. Nisi, and N. Nunes. “Staying on Track: a Compar-ative Study on the Use of Optical Flow in 360º Video to Mitigate VIMS”. in: Proceed-ings of the ACM International Conference on Interactive Media Experiences. IMX 2020, Barcelona, Spain, June 17-19, 2020. ACM, 2020, pp. 82–93. d o i: 10.1145/3391614 .3393658.u r l:https://doi.org/10.11 45/3391614.3393658
Full paper atIMX(SIGCHIsponsored con-ference) describing the first iteration of pipeline using optical flow for controlling techniques known to mitigate VIMSand accompanying evaluation study. This re-search contribution is described in chap-ter7.
Continued on next page
Table 1.4: continued from previous page
Publication Description
[Bal+21] P. Bala, I. Oakley, V. Nisi, and N. Nunes. “Dynamic Field of View Re-striction in 360º Video: Aligning Optical Flow and Visual SLAM to Mitigate VIMS”.
in:Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.
CHI ’21. Yokohama, Japan: Association for Computing Machinery, 2021. d o i: 1 0. 1145 /3411764 . 3445499. u r l: https : //doi.org/10.1145/3411764.3445499
Full paper atCHI (SIGCHI main confer-ence) describing the second iteration of pipeline using optical flow andSLAMfor controlling restrictedFoVknown to miti-gateVIMSand accompanying evaluation study. This research contribution is de-scribed in chapter8.
[Bal+16] P.Bala, M. Dionísio, V. Nisi, and N. Nunes. “IVRUX: A Tool for Analyz-ing Immersive Narratives in Virtual Real-ity”. en. In: Interactive Storytelling. Ed.
by F. Nack and A. S. Gordon. Vol. 10045.
Lecture Notes in Computer Science 10045.
Springer International Publishing, Nov.
2016, pp. 3–11. i s b n: 978-3-319-48279-8.
d o i:10.1007/978-3-319-48279-8\_1
Short paper atICIDS describing IVRUX, anartifactsystem fo viewing and explor-ing syntetized recorded data from user’s experience in a HMD narrative. This ar-tifact system was used as basis for the previous publications [Bal+19; Bal+20b;
Bal+21]. This research contribution is de-scribed in chapter5and an evaluation of the IVRUXprototype is described in ap-pendix sectionB.3.
[Bal+18a] P. Bala, D. Dionisio, V. Nisi, and N. Nunes. “Visually Induced Mo-tion Sickness in 360º Videos: Comparing and Combining Visual Optimization Tech-niques”. In:IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2018 Adjunct, Munich, Germany, October 16-20, 2018. IEEE, 2018, pp. 244–249.
d o i: 10 . 1109 / ISMAR - Adjunct . 2018 . 0 0077
Poster atIEEE ISMARon first iteration of pipeline using optical flow for controlling techniques known to mitigate VIMSand a pilot evaluation study, that later led to [Bal+20b].
Continued on next page
1 . 4 . C O N T R I B U T I O N S A N D P U B L I C AT I O N S
Table 1.4: continued from previous page
Publication Description
[Bal+18b] P.Bala, R. Masu, V. Nisi, and N.
Nunes. “Cue Control: Interactive Sound Spatialization for 360º Videos”. In: Inter-active Storytelling - 11th International Con-ference on Interactive Digital Storytelling, ICIDS 2018, Dublin, Ireland, December 5-8, 2015-8, Proceedings. Ed. by R. Rouse, H.
Koenitz, and M. Haahr. Vol. 11318. Lec-ture Notes in Computer Science. Springer, 2018, pp. 333–337. d o i: 10.1007/978-3-030-04028-4\_36
Poster at ICIDS describing an initial im-plementation ofCue Controlartifactand a pilot evaluation, that later led to [Bal+19].
[BNN17] P. Bala, V. Nisi, and N. Nunes.
“Evaluating User Experience in 360º Sto-rytelling Through Analytics”. In: Interac-tive Storytelling - 10th International Con-ference on Interactive Digital Storytelling, ICIDS 2017, Funchal, Madeira, Portugal, November 14-17, 2017, Proceedings. Ed.
by N. Nunes, I. Oakley, and V. Nisi.
Vol. 10690. Lecture Notes in Computer Science. Cham: Springer, 2017, pp. 270–
273. i s b n: 71026-6 978-3-319-71027-3. d o i: 10.1007/978-3-319-7102 7-3\_23. (Visited on 10/12/2018)
Poster at ICIDS on methodology of ob-serving user experience on 360º video, in-cluding a proof-of-concept prototype iter-ation ofIVRUX, which was later used for [Bal+18b] and [Bal+19]. This is described in section5.2.3.
[Mas+20] R. Masu, P. Bala, M. A. Ah-mad, N. Do Nascimento Correia, V. Nisi, N. Nunes, and T. Romão. “VR open scores:
scores as inspiration for VR scenarios”. In:
NIME New Interfaces for Musical expres-sion. 2020
Full paper at NIME on aleatoric score-based virtual scenarios, where an aleatoric score is embedded in a virtual environ-ment. This publication describes two sce-narios, one of those usingCue Control.
Table1.5presents contributions emerging from research done on transmedia story-telling (described in chapter5). Although not central to the main theme of the work, these contributions resulted in artifacts that shaped the previous contributions, namely the development ofIVRUX.
Table 1.5: Research contributions connected to chapter5.
Publication Description
[Dio+16] M. Dionísio, V. Nisi, N. J. Nunes, and P.Bala. “Transmedia Storytelling for Exposing Natural Capital and Promoting Ecotourism”. In: Interactive Storytelling -9th International Conference on Interactive Digital Storytelling, ICIDS 2016, Los Ange-les, CA, USA, November 15-18, 2016, Pro-ceedings. Ed. by F. Nack and A. S. Gordon.
Vol. 10045. Lecture Notes in Computer Science. 2016, pp. 351–362. d o i: 10.100 7/978-3-319-48279-8\_31
Full paper at ICIDS on the use of trans-media storytelling for engaging users in thinking of ecosystems and their underly-ing biodiversity. This publication includes a initial description of the experience pro-totype of Fragments of Laura, a location-based transmedia experience.
[Dio+17b] M. Dionísio, P. Bala, V. Nisi, and N. Nunes. “Fragments of laura: in-corporating mobile virtual reality in lo-cation aware mobile storytelling experi-ences”. In: Proceedings of the 16th Inter-national Conference on Mobile and Ubiq-uitous Multimedia, MUM 2017, Stuttgart, Germany, November 26 - 29, 2017. Ed.
by N. Henze, P. Wozniak, K. Väänänen, J. Williamson, and S. Schneegass. ACM, 2017, pp. 165–176. d o i:10.1145/315283 2.3152868
Full paper at MUM (SIGCHI in-cooperation conference) on the de-scription and evaluation of Fragments of Laura, a location-based transmedia experience, incorporating mobile VR environments.
[Dio+17a] M. Dionisio, P. Bala, V. Nisi, I. Oakley, and N. Nunes. “Step by Step:
Evaluating Navigation Styles in Mixed Re-ality Entertainment Experience”. In: Ad-vances in Computer Entertainment Technol-ogy - 14th International Conference, ACE 2017, London, UK, December 14-16, 2017, Proceedings. Ed. by A. D. Cheok, M. In-ami, and T. Romão. Vol. 10714. Lec-ture Notes in Computer Science. Springer, 2017, pp. 32–45. d o i: 10.1007/978-3-3 19-76270-8\_3
Full paper atACEon navigation styles for the second iteration of theThe Old Phar-macy, a virtual environment inFragments of Laura. This study is described in ap-pendix sectionB.2.
Continued on next page
1 . 4 . C O N T R I B U T I O N S A N D P U B L I C AT I O N S
Table 1.5: continued from previous page
Publication Description
[Dio+18] M. Dionísio, P. Bala, V. Nisi, and S. Câmara. “Bringing Locative Me-dia Indoors: Strategies For RemeMe-diation”.
In: Looking Forward, Looking Back: Inter-active Digital Storytelling and Hybrid Art Approaches. Carnegie Mellon University:
ETC Press, Pittsburgh, PA, Jan. 1, 2018, pp. 71–92. u r l: https : / / press . etc . cmu.edu/index.php/product/looking-forward-looking-back/
Book chapter on the remediation of trans-media artifacts for physical exhibitions, in-cluding remediating the locative tour of Fragments of Laura asVR. This remedia-tion is described in secremedia-tion5.1.3.
[Bal+17] P.Bala, M. Dionísio, R. Trindade, S. Olim, V. Nisi, and N. Nunes. “Evaluat-ing the influence of location and medium applied to mobile VR storytelling”. In:
Proceedings of the 16th International Confer-ence on Mobile and Ubiquitous Multimedia, MUM 2017, Stuttgart, Germany, Novem-ber 26 - 29, 2017. Ed. by N. Henze, P.
Wozniak, K. Väänänen, J. Williamson, and S. Schneegass. ACM, 2017, pp. 371–378.
d o i: 10 . 1145 / 3152832 . 3156617. u r l: https://doi.org/10.1145/3152832.31 56617
Poster at MUM (SIGCHI in-cooperation conference) describing the pilot study of a first iteration of The Old Pharmacy, ex-perienced in physical locations with and without links to the narrative location and with different platforms (tablet andHMD).
This pilot study is described in appendix sectionB.1.
Finally, the following contributions were also accomplished during the research pe-riod but are not connected to the main research theme. Since these explore alternative research threads, they are not described in this document.
[Nis+19] V. Nisi, M. S. Dionisio, P.Bala, T. Gross, T. Up, and N. J. Nunes. “Lucid Peninsula, a Physical Narrative Art Installation Comprising Interactive 360º Virtual Reality Components”. In:IJCICG10.1 (2019), pp. 1–15.d o i: 10.40 18/IJCICG.2019010101.
[Bal+20a] P.Bala, M. Dionísio, T. Andrade, and V. Nisi. “Tell a Tail 360º: Immersive Storytelling on Animal Welfare”. en. In:Interactive Storytelling. Ed. by A.-G.
Bosser, D. E. Millard, and C. Hargood. Vol. 12497. Series Title: Lecture Notes in Computer Science. Cham: Springer International Publishing, 2020, pp. 357–360. i s b n: 978-3-030-62515-3 978-3-030-62516-0. d o i: 10.1007 /978-3-030-62516-0\_35.
[Bal+20b] P.Bala, M. Dionísio, S. Oliveira, T. Andrade, and V. Nisi. “Tell a Tail: Lever-aging XR for a Transmedia on Animal Welfare”. en. In: Entertainment Computing – ICEC 2020. Ed. by N. J. Nunes, L. Ma, M. Wang, N. Correia, and Z. Pan. Vol. 12523. Series Title: Lecture Notes in Computer Science.
Cham: Springer International Publishing, 2020, pp. 223–239. i s b n: 978-3-030-65735-2 978-3-030-65736-9. d o i:10.1007/978-3-030-65736-9\_19.
[Dio+20] M. Dionísio, P.Bala, S. Oliveira, and V. Nisi. “Tale of T(r)ails: The Design of an AR Comic Book for an Animal Welfare Transmedia”. en. In: Interactive Storytelling. Ed. by A.-G. Bosser, D. E. Millard, and C. Hargood. Vol. 12497.
Series Title: Lecture Notes in Computer Science. Cham: Springer Interna-tional Publishing, 2020, pp. 281–284.i s b n: 62515-3 978-3-030-62516-0.d o i: 10.1007/978-3-030-62516-0\_25.
[Bal+21] P.Bala, V. Nisi, M. Dionisio, N. J. Nunes, and S. James. “Square Peg, Round Hole: A Case Study on Using Visual Question & Answering in Games”.
In: Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play. CHI PLAY ’21. Virtual Event, Austria: Association for Computing Machinery, 2021, pp. 133–139. i s b n: 9781450383561. d o i: 10.1145/3450337.3483466.
[Dio+21] M. Dionísio, V. Nisi, J. Xin, P.Bala, S. James, and N. J. Nunes. “Amnesia in the Atlantic: An AI Driven Serious Game on Marine Biodiversity”. en.
In: Entertainment Computing – ICEC 2021. Ed. by J. Baalsrud Hauge, J.
C. S. Cardoso, L. Roque, and P. A. Gonzalez-Calero. Vol. 13056. Series Title: Lecture Notes in Computer Science. Cham: Springer International Publishing, 2021, pp. 427–432.i s b n: 978-3-030-89393-4 978-3-030-89394-1. d o i: 10.1007/978-3-030-89394-1_35.