The field ofHuman-Computer Interaction (HCI)gained prominence in the 1980s and be-came a burgeoning field of study. Fueled by fast computing technology,HCIas a field has been described as a "moving target" [Gru12], expanding its original concern of interfacing humans and machines to addressing all aspects of human life through computing [Rog12, p. xi]. Alan Dix has also singled out this expansive nature, describing "HCIas an aca-demic discipline has always been positioned, sometimes uneasily, sometimes creatively, in the tension between solid intellectual rigor and the excitement in new technology"
[Dix17]. This can also be perceived in Figure2.1where the different disciplines ofUser Experience Design (UXD)andInteraction Design (IxD)are mapped [pre21;Saf10]. The core tenet of usability (easy to use, easy to learn [Ben19b, p. xv]) inHCIis central toUXD andIxDas both fields were exports from traditionalHCIinto the world of Design. IxD was prompted by moving away from the concept of single-user and computer [Rog12, p. 3], and applying design thinking approaches [Ben19b, p. xv].UXDwas prompted by new interfaces (e.g., web, tangible, spatial computing, etc.) and the need to make the ex-perience of using these interfaces enjoyable and engaging [Ben19b, p. xv]. This expansion of focus has ledHCIto outgrow its initial concerns, become an interdisciplinary field and reflect onto itself on how it is studied, designed, and evaluated. Several researchers have
characterized this evolution throughparadigms[HTS07],waves[Bød15], circles [Gru17]
and periods [Rog12].
User Experience Design
Interaction Design Industrial
Design
Human Computer Interaction
Usability Engineering Ubiquitous
Computing
Interactive Controls Mechanical
Engineering
Electrical Engineering
Psychology Cognitive
Science
Sociology Philosophy
Human Factors
& Ergonomics Architecture
Information Architecture
Communication Design
Motion Design
Spatial Experience
Contextual Requirements
Data & Info Visualization Functional
Requirements
Generative Design Marketing
Audio Engineering Sound
Design
Writing
Computer Science
Interactive Environments
User Interface Design Application
Design
Software Development
Navigation Design
Guidance Systems
User Interface Scenography
Scenario Design Digital
Signage Media
Installations
Figure 2.1: Infographic on User Experience disciplines fromenvis precisely[pre21] (Cre-ative Commons Attribution-ShareAlike 3.0 Unported), based on original concept from Saffer [Saf10, p. 21]
2.1.1 Computing andHCI
Grudin [Gru17] adopted a historical perspective to the evolution ofHCI, by mapping landmark research and technological advances that are fundamental to modern HCI.
Through reflection on the advances in artifacts (e.g., vacuum tubes, mainframes, com-puters, mobile/hanheld, embedded devices) [Gru17, p. 55], and visionary visions (e.g., Sutherland’s concept of Computer Graphics that lead toGraphical User Interface (GUI)
2 . 1 . H U M A N - C O M P U T E R I N T E R AC T I O N
Table 2.1: Evolution of technology from invention to maturity, and links toInteraction Design (IxD)and key research approaches (adapted from Grudin [Gru17, p. 122])
Technology evolution IxDpriority Key research approaches
1) Hobby & Research None None
2) Routine Use in Business Efficient skilled use Controlled experiments 3) Consumers Adopt Basic
Technology Initial & casual interaction Usability studies, metrics, an-alytics
4) Differentiation &
Self-expression Visual Design & marketing
Inspiration, design, etnogra-phy, sociology, brand integra-tion
[Gru17, p. 39]) , Grudin analyzed four core disciplines as the basis ofHCI:Information Systems (IS),Computer Systems (CS),Library and Information Science (LIS), andHuman Factors (HF)[Gru17, p. 22]. Ultimately, Grudin [Gru17, p. 114] defends that these core fields diverge due to differences in targeted users (specialists versus non-discretionary users versus discretionary users), publication venues and communities, the influence of humanities and psychology (cognitive, social, industrial, engineering), and cultural differences of researchers.
Grudin identifies paths "that extend from the past to the present" to anticipate future change [Gru17, p. 21]. Grudin’s view for future of HCIis heavily influenced by Lick-lider’s blueprint for human-computer symbiosis as a three-phase process: (1) Human-Computer Interaction, where machines are considered extensions of the human; (2) Human-computer symbiosis, where the machine and human are enmeshed in a part-nership; and (3) ultra-intelligent machines, where machines are superior to human intel-ligence [Gru17, p. 19]. Most of the work done currently inHCIis focused on the first two stages, with Grudin accentuating that the advances inArtificial Intelligence (AI), Internet of Things, and "big data" may help usher in a new Era [Gru17, p. 117].
Grudin’s focus on technology allows for reflection and identification of patterns in how technology matures. Firstly, technology following a four-step path from invention to maturity (see table2.1) with different design priorities and research methods. Technology starts as a hobby or research, is applied to business (used by specialists and concerned with efficiency), becomes available to consumers (used by non-discretionary users and concerned with initial/casual interaction), becomes a mature technology (where users expect differentiation and self-expression), before finding new challenges [Gru17, p. 122 ]. Secondly, technology follows a cyclic pattern where influences are cumulative [Gru12], meaning that new technology supports the same activities (e.g., email, text messaging, and social networks all support communication).
2.1.2 HCIasParadigmsandWaves
Kuhn depicts science as happening within paradigms[Kuh96]. Normal science is the result of a prevailing framework orparadigm; aparadigmshift is the result of a scientific
Table 2.2:HCI Paradigms(adapted from [HTS07])
First Paradigm
Interaction as... Man-machine coupling
Goal Optimizing fit between man and machine
Questions How can we fix specific problems that arise in interaction?
Disciplines Engineering, programming, ergonomics
Second Paradigm
Interaction as... Information communication
Goal Optimizing accuracy and efficiency of information transfer Questions
What mismatches come up in communication between computers and people? How can we accurately model what people do? How can we improve the efficiency of computer use?
Disciplines Laboratory and theoretical behavioral science
Third Paradigm
Interaction as... Phenomenologically situated
Goal Support for situated action in the world
Questions
What existing situated activities in the world should we support?
How do users appropriate technologies, and how can we support those appropriations? How can we support interaction without con-straining it too strongly by what a computer can do or understand?
What are the politics and values at the site of interaction, and how can we support those in design?
Disciplines Ethnography, action research, practice-based research, interaction analysis
revolution, where the dominant paradigm is deemed incompatible, a newparadigmis adopted and normal science proceeds [Kuh96].
Inspired by Kuhn and applying it toHCI, Harrison et al. [HST11; HTS07] identify three paradigms based on the metaphor of interaction (see fig. 2.2): (1) a paradigm centered on engineering and human factors, largely atheoretical and problem-oriented;
(2) aparadigmcentered on cognitive science disciplines and oriented towards efficient communication; and (3) aparadigmcentered on humanistic and social science [FT18b].
Theseparadigmsdo not representparadigmshifts as Kuhn described [Kuh96] since they are able to coexist [FT18b;HTS07;Rog12]. They do however advocate that the inclusion of alternative approaches is essential toHCI[HST11].
In parallel, Bødker’s [Bød06;Bød15] characterization ofHCIaswaveshas been popu-larized and adopted inHCIdiscourse [Tay11]. Thesewavesdo not representparadigm shifts as Kuhn described [Kuh96], with awavereplacing another; rather, thewaves coex-ist and overlap, and are representative of how new research questions emerged, how new methods were used, and how new challenges were addressed [Ban11]:
• Firstwave HCI, based on model-driven cognitive science and human factors, studied the interaction of human beings with machines based on strict guidelines, formal methods and systematic testing [Bød15]. Therefore, thiswaveby focusing on human beings is concerned with the design of input and output affordances to facilitate human/machine relations [FT18b], namely on desktops primarily in office settings [Rog12, p. 12].
2 . 1 . H U M A N - C O M P U T E R I N T E R AC T I O N
• Second wave HCI broadened the scope by focusing on groups, work settings and well-established communities of practice [Bød15]. This expansion into real-world context, shifted the focus from the efficiency of interaction to how machines were in reality used by humans [FT18b].
• Thirdwave HCIbroadened the scope as technology spread to homes and everyday life [Bød15]. This thirdwave’s central metaphor of interaction as phenomenologi-cally situated in the world, expanded the values ofHCIto include human values, meaning-making, situated knowledge, and experience [FT18b]. AResearch through Design (RtD)[ZFE07] approach became popular during thiswaveas articulating knowledge throughartifactswas inline with Design and Humanities practices.
While details of characterization ofHCIasparadigmorwavesmay differ, both char-acterizations have in common a gradual expansion ofHCI’s concerns, methodologies and applications [FT18b]. The possible existence of a fourthwavehas also become a popular speculative tool to ponder the future ofHCI[Ash+19;BB17;Ble+14;Fra20;HKV20].
2.1.3 HCIas Periods
Parallel to Grudin’s historical overview ofHCI, Rogers undertook a theoretical approach toHCIby conceptualizing it through itsparadigms,theories,models,frameworksand approaches[Rog12, p. 4]. Rogers’s focus ontheoryinHCIis based on the initial role that the scientific method, useful in constructing theory, had in the advancement of the body of knowledge in earlyHCI[Rog12, p. 15]. The inclusion of different interpretations of theory, emerging from social sciences, arts and philosophy, while might not be directly applicable to quantifiably measure phenomena, introduced a set of lenses to help discuss HCI[Rog12, p. 15].
However, the inclusion of a large set of new disciplines, and subsequently new epis-temologies, introduces difficulties in (1) understanding how theory connects toHCIand (2) how it is used [VBR17]. Concerning the former, table2.3presents some of the roles that theory can adopt inHCI. The roles identified by Berdeson and Shneiderman [BS03]
are referent to a more classical use of theory in the scientific method, while the roles identified by Rogers [Rog05;Rog12] are referent to the expansion ofHCIdue to transdis-ciplinarity and characterization asparadigm/waves[Bød06;Bød15;HTS07]. Concerning the latter, due to the limited application of scientific epistemologies to design new expe-riences and technologies [VBR17],HCIhas approached this by for example focusing on design-oriented theory [Gav12; ZFE07], creatingartifactsthat bridge the gap between theory and practice [HL12] or translational research [BE18;Col+19;GSS14].
Based on the increasing use of theory in HCI, and paralleling art history, Rogers highlights three periods of HCI theory: Classical HCI (referring to the 1980s, when classical theories were imported from cognitive psychology), ModernHCI(referring to 1990s and early 2000s, with the inclusion of theory from social, phenomenological and
Table 2.3: Roles of theory applied to HCI [BS03;Rog05;Rog12]
Role Description Source
Descriptive Offers concepts, clarifying terminology and guiding further inquiry
Bederson and Shneiderman [BS03]
Explanatory Explains relationships and processes Predictive Predicts user performance
Prescriptive Provides guidance for design
Generative Enables practitioners to create new knowledge
Informative Offers selected knowledge and generalizations from an-other discipline
Rogers [Rog05]
Ethnographic
Grounded on discipline, such as anthropology, cognitive science or sociology, offers rich descriptions of a real-world phenomenon
Conceptual
Offers high-level frameworks and dimensions for inform-ing and communicatinform-ing the design and evaluation of proto-types, user interactions and user studies
Critical Based on cultural and aesthetic concerns, offers critiques and reasonings behind interaction design
Wild Understanding use of technology in-situ Rogers [Rog12]
Table 2.4: Theories and approaches of HCI Periods [Rog12]. See appendixAfor a detailed description
Classical
HCI Cognitive approaches
Body of knowledge Basic Research Cognitive modelling
Modern HCI
Alternative cognitive approaches
External Cognition Distributed Cognition Ecological psychology Social approaches
Situated action
Ethnomethodology and ethnography CSCW Theories
Other imported approaches
Activity Theory Grounded Theory Hybrid Theory Contemporary
HCI
Turn to Design Turn to Culture Turn to the Wild Turn to embodiment
cognitive science), and ContemporaryHCI(from early 2000s onwards, becoming more value-led and incorporating moralistic and societal-based perspectives) [Rog12, p. 7].
Table2.4 represents these main periods, including the different approaches and turns, while appendixApresents a more detailed description of these periods.
The ContemporaryHCIperiod coincides as well with systems moving away from the Windows, icons, menus, and pointing (WIMP)style of interaction. It coincides as well with thirdwave HCI focus on experience, rather than efficiency [FT18b]. Since these systems challenged the applicability of established methods and tools, new frameworks began to appear aimed at emergent technologies (such as wearable computers, ubiquitous
2 . 1 . H U M A N - C O M P U T E R I N T E R AC T I O N
Table 2.5:Reality-Based Interaction(RBI)framework [Jac+08]
Themes
Naïve Physics Common sense knowledge of the physical world Body
Awareness & Skills
Awareness of physical bodies and the skills used to control and coordinate it
Environment Awareness & Skills
Awareness of their surroundings and skills to interact, ma-nipulate and navigate within it
Social
Awareness & Skills Awareness of others and skills to interact with them
Trade-offs
Expressive Power Will a new interaction allow for the performance of more tasks in the application domain?
Efficiency Will a new interaction allow for faster task performance?
Versatility Will a new interaction allow for more types of tasks in the application domain?
Ergonomics Will a new interaction allow for the performance of a task without physical injury, fatigue, or discomfort?
Accessibility Will a new interaction allow users with a variety of abilities to perform a task?
Practicality Will a new interaction allow for more practical applications to be developed and produced?
systems to immersive virtual spaces). These frameworks were particulatly influenced by ContemporaryHCI’sturnssuch asturn to embodiment[Dou01] orturn to design[Rog12, p. 68]. For example, frameworks and concepts from Ishii and Ullmer [IU97], Hornecker and Buur [HB06] and Antle [Ant07] support the analysis, design and critique of Tangible Interaction; Wigdor and Wixon [WW11]’s focus on Natural User Interfaces for touch and gestures; or Milgram et. al. [Mil+95]’s taxonomy of augmented reality displays (further discussed in subsection2.3.1). The latter frames the development of augmented reality applications in line with the extent of world knowledge, reproduction fidelity, and extent of Presence metaphor. ConsideringXRsystems, two other frameworks emerge as of note.
Firstly, the Reality-Based Interaction (RBI) [Jac+08] framework revolves around the central concept of "real world", meaning the physical, non-digital world. This framework (see table2.5) is focused on mimicking four themes emerging from the real world (Naïve Physics, Body Awareness & Skills, Environment Awareness & Skills, and Social Aware-ness & Skills). However, these interaction principles can be replaced ("traded-off") for other qualities (Expressive Power, Efficiency, Versatility, Ergonomics, Accessibility, and Practicality) only when needed. Following up on the use of this framework, Girouard et al. [Gir+19] have noted that, in addition to being used to analyze, design, and critique emergent technologies,RBIhas also been used to update previous user-centered models [Poo+16] and as a basis for an extension, incorporating new concepts (such as previous experience in using of digital technologies, as Jetter et al.’s [Jet+13]Blended Interaction) or new application domains (such as creative group work in Geyer et al. [Gey+11]).
Secondly, the Trajectories conceptual framework is a framework for the design of
mixed reality experiences, that tries to bridge the gap between theory (as identified by [Rog12, p. 82] and practice (a strong concept as identified by [HL12]) by leveraging on the knowledge of artists or designers to identify technology requirements in designing new experiences [Rog12, p. 82]. Trajectoriesevolved in three phases as newartifactsystems were created:
• Temporal trajectories [BG08] - a conceptual framework for narrative drive expe-riences mapping story-time (time in the fictional world) and clock-time (time in real-world) with 3 trajectories used to reason about pacing, synchronization, and fictional time travel:Canonical trajectories(temporal mappings set by the experience’
authors),Participanttrajectories(temporal mappings experienced byparticipants), andHistoric trajectories(retellings of past trajectories).
• Interactional trajectories [Ben+09] - an extension of the previous conceptual frame-work, where a trajectory is now defined as a coherent journey through a user expe-rience, with multiple possible hybrid structures (time, space, roles, and interfaces).
Transitions are critical moments in an experience where the trajectory crosses hy-brid structures (e.g., transversal between virtual and real worlds) that endanger the journey’s coherence. Managing trajectories involves orchestration to align/resolve the tension betweenparticipants’ trajectories and canonical trajectories caused by interaction. Interleaved trajectories are referent to the social and collaborative as-pects of having multipleparticipants.
• TheTrajectoriesframework as described in [BG11], with a more extensive descrip-tion of the previous components and a renewed focus on HCIand Performance Studies.
Similarly to Girouard et al. [Gir+19], the use of Trajectories has also been surveyed [VBR17] by looking at how different systems used the framework to analyze, design or describe (e.g., [Ben+11; Fli+11; Fos+13; Fos+14]) or to discuss and build concepts.
For example, Jaller and Stefania use the concept of Trajectories as applicable to Cine-matic Virtual Reality (CVR), to argue for the "importance of transition design, not only for mixed-reality performance but for commercialCVRexperiences andHCIin general"
[JS20]; this is understandable asTrajectories’ framing onHCIand Performance studies has strong parallels toCVR, where the experience of spectating is orchestrated and spatially designed by an artist.