Human Systems Engineering and Design II

Advances in Intelligent Systems and Computing 1026

Tareq Ahram

Waldemar Karwowski Stefan Pickl

Redha  Taiar    Editors

Human Systems Engineering and Design II

Proceedings of the 2nd International Conference

on Human Systems Engineering and Design

(IHSED2019): Future Trends and Applications,

September 16–18, 2019, Universität der

Bundeswehr München, Munich, Germany

Using Face Recognition for Smart Home Control

Bernardo Marques^(&), Paulo Dias, João Alves, and Beatriz Sousa Santos

DETI - IEETA, University of Aveiro, Aveiro, Portugal bernardo.marques@ua.pt

Abstract. Augmented Reality (AR) offers the possibility to present information depending on the context/location: pointing at a given appliance will automat- ically present relevant interfaces. In this work, we explore Pervasive AR in the context of the Smart Home: users may select the control of a given appliance by pointing at it, simplifying the interaction process. This idea is then extended to consider user profile in the loop: face recognition is used to recognize a given user of and adapt the interface not only to location but also to specific users needs. A preliminary study was conducted to evaluate and validate the concept.

Keywords: Pervasive Augmented Reality

!

!

Adaptive user interfaces

!

!

1 Introduction

With the evolution of technology, control and monitor of Smart Homes though digital devices is becoming a reality. As people’s expectations of what technology can do for them are changing, new interaction paradigms are fundamental, as the vision of what a Smart Home entails is continuously evolving [1]. Augmented Reality (AR) appears as an opportunity to provide powerful and flexible user-interface for these intelligent environments [2].

Despite recent success of AR, most solutions present information with limited interaction rather than considering an integrated human-centered approach [3]. AR is starting to be extended to Pervasive AR: a continuous and pervasive user interface that augments the physical world with digital information registered in 3D, while being aware of and responsive to the user’s context [4]. Continuous AR experiences raise challenges about how to display information (which type of information is distracting or relevant) and how to properly interact with that information. Clearly, this requires personalization and adaptation of the content.

Literature suggests user adaptive interfaces provide excellent opportunities for improving user experience based on context [5]. Hence, it is viable to extend this to AR, delivering content according to user profile, surrounding environment and device being used.

©Springer Nature Switzerland AG 2020

T. Ahram et al. (Eds.): IHSED 2019, AISC 1026, pp. 15–19, 2020.

https://doi.org/10.1007/978-3-030-27928-8_3

2 Uninterrupted Augmented Reality Experiences

The work presented is based on an initial application to create and customize unin- terrupted AR experiences in a Smart Home. The application enables remote control of appliances by pointing at them through location-based interfaces. Depending on the available device, 1 of 3 different methods of interaction are presented: 2D standard interfaces, marker-based AR interfaces and continuous AR interfaces. The appliances controlled in this setup were a Philips Hue set, enabling control of several aspects of lights (Fig.1 – (Left)), such as state (on/off) of specific rooms, brightness, color selection. A boiler was also used (Fig.1- (Right)), to control hot water, state (on/off), temperature regulation, and scheduling along the week period (enabling to program automatic turn on and off the boiler). The application was developed using the Unity Game engine, C# scripts and the Vuforia Library.

3 Adaptive UI Using Face Recognition

The application was then evolved to mirror the conceptual framework of Fig.3, composed of four interconnected modules, enabling the use of adaptive interfaces, based on face recognition authentication (Fig.2).

The main modules are as follow:

The Authentication Moduleauthenticates a user through credentials (traditional login/password) or face recognition (uses a photo taken with the camera of the device being used). The OpenCV asset for Unity was adapted to allow the identification of multiple users from a set of existing profiles with previously taken photos, based on the Eigen-Faces algorithm. Different user profiles with specific permissions exist to man- age and control the Smart Home. New user profiles can also be added, edited or removed. Only a user with top-level permission on the hierarchy can perform such task.

Fig. 1. Augmented Reality interfaces able to control lights (Left) and a boiler (Right).

16 B. Marques et al.

ThePose Estimation Moduleanalyses the surrounding environment (6DOF Data Collection), which is then used by the content filter to generate specific adaptive interfaces.

TheAdaptive Moduleuses a contentfilter and selects which interfaces to display according to user profile and location in the Smart Home. Then, the interface generator groups all models (text, images, 3D objects) necessary to create each interface. Based on the device being used, the application automatically displays the suitable interface.

Fig. 2. Adaptive user interfaces: authentication process (Left), available options after authen- tication is conducted with success (Right).

Fig. 3. Conceptual framework for context adaptive interfaces based on User Recognition and Identification.

Finally, theInteraction Moduleenables the user to manage and interact with the appliances in the Smart Home. The user is also able to select other type of interface.

4 Preliminary Results Using Face Recognition

A preliminary study was conducted with 5 participants (1 female and 4 males, ranging in age between 23 to 33 years old, simulating a family composed by two parents, plus three children) to evaluate the robustness of the face recognition authentication process.

Participants were inserted in the system as new users, to create the training model. Each user was asked to log using face recognition ten times. Results show that 74% of the times, participants were successfully found and the remaining 26% of the times the application detected the wrong participant (Fig.4).

The developed application opens up new possibilities for the way users interact and control the Smart Home concept moving forward, by pointing at specific localizations and appliances, while being confronted with adaptive interfaces based on the recogni- tion of the user face. Adaptive interfaces can be useful for this, since they are able to provide the required information in the most suitable way, by taking into account current context of use, such as user related aspects (preferences and customization), aspects related to the technology (available interaction mechanisms) and environment context.

Virtual-generated objects and interfaces will continue to increasingly become more interactive through several modalities responding to voice, gestures and even allowing user interaction through touch, leading to an enhanced experience. Moreover, AR applications will become more persistent over time, enabling users to place virtual objects on top or next to physical objects, enabling sharing mechanisms with other users or allowing later interaction in another moment. AR applications will obtain a greater understanding of the surrounding physical space, through the combination with concepts like machine learning, context awareness or/and big data, allowing to immediately react and adapt to changes in the environment or to the presence of different users.

Fig. 4. Results associated to the use of the face recognition process to authenticate users in the application.

18 B. Marques et al.

5 Concluding Remarks and Future Work

Through the use of Pervasive Augmented Reality, it is possible to upgrade from an application with a unique objective to a multi-purpose continuous experience based on the user face recognition, changing the way they interact with information and the surrounding environment. In the context of the Smart Home, considering a family, different members can have different levels or permissions. The interfaces can adjust its content, ranging from limited control over their rooms and shared-spaces, to full control over all the features of the application. Furthermore, based on the proposed framework, the interfaces can be adapted to other particular contexts, purposes and usages. Although at an early stage, the proposed framework seems promising, opening new possibilities for interaction and control of smart environments. By pointing at specific localizations and appliances, user tailored interfaces are presented in a specific device with minimum interaction. Future development includes the improvement of the application recognition robustness and a further validation. Additionally, we also want to explore emotion recognition, aiming to adapt the Smart Home environment (light- ning settings, room temperature) to the user preferences and improve comfort and performance.

Acknowledgments. To all participants involved in the case studies, thanks for collaborating in providing essential data and relevant feedback. The present study was developed in the scope of the Smart Green Homes Project [POCI-01-0247-FEDER-007678], a co-promotion between Bosch Termotecnologia S.A. and the University of Aveiro. It isfinanced by Portugal 2020 under the Competitiveness and Internationalization Operational Program, and by the European Regional Development Fund. This study was also supported by IEETA - Institute of Electronics and Informatics Engineering of Aveiro, funded by National Funds through the FCT - Foundation for Science and Technology, in the context of the project UID/CEC/00127/2019.

References

1. Wilson, C., Hargreaves, T., Hauxwell-Baldwin, R.: Smart homes and their users: a systematic analysis and key challenges. Pers. Ubiquit. Comput.19(2), 463–476 (2015)

2. Kim, K., Member, S., Billinghurst, M., Member, S., Bruder, G., Duh, H.B., Member, S.:

Revisiting trends in augmented reality research: a review of the 2nd decade of ISMAR (2008– 2017). IEEE Trans. Vis. Comput. Graph. (2018)

3. Pedersen, I.: Radiating centers: augmented reality and human-centric design. In: Arts, Media and Humanities Proceedings - IEEE International Symposium on Mixed and Augmented Reality, ISMAR-AMH 2009, pp. 11–16 (2009)

4. Grubert, J., Zollmann, S.: Towards pervasive augmented reality: context awareness in augmented reality. IEEE Trans. Visual Comput. Graphics23(6), 1706–1724 (2017) 5. Abidin, R.Z., Arshad, H., Shukri, S.A.A.: Adaptive multimodal interaction in mobile

augmented reality: a conceptual framework. In: ICAST 2017 (2017)