Web presence for things through physical mobile interaction techniques

(1)

EDWIN CARLO RIBEIRO MARINHO

WEB PRESENCE FOR THINGS THROUGH PHYSICAL MOBILE INTERACTION TECHNIQUES

Federal University of Pernambuco [email protected] www.cin.ufpe.br/~posgraduacao

RECIFE

2017

(2)

Edwin Carlo Ribeiro Marinho

WEB PRESENCE FOR THINGS THROUGH PHYSICAL MOBILE

INTERACTION TECHNIQUES

A M.Sc. Dissertation presented to the Center for Informatics of Federal University of Pernambuco in partial fulfillment of the requirements for the degree of Master of Science in Computer Science.

Advisor: Kiev Santos da Gama

RECIFE 2017

(3)

Catalogação na fonte

Bibliotecária Monick Raquel Silvestre da S. Portes, CRB4-1217

M338w Marinho, Edwin Carlo Ribeiro

Web presence for things through physical mobile interaction techniques /

Edwin Carlo Ribeiro Marinho. – 2017. 90 f.: il., fig., tab.

Orientador: Kiev Santos da Gama.

Dissertação (Mestrado) – Universidade Federal de Pernambuco. CIn, Ciência da Computação, Recife, 2017.

Inclui referências e apêndices.

1. Engenharia de software. 2. Internet das coisas. I. Gama, Kiev Santos da (orientador). II. Título.

005.1 CDD (23. ed.) UFPE- MEI 2018-004

(4)

Edwin_{Carlo Ribeiro Marinho}

Web_{Presence for Things Through Physical Mobile Interaction} Techniques

Dissertação de Mestrado apresentada ao Programa de Pós-Graduação em Ciência da Computação da Universidade Federal de Pernambuco, como requisito parcial para a obtenção do título de Mestre em Ciência da Computação

Aprovado_{em: 29/08/2017.}

BANCA_EXAMINADORA

______________________________________________ Profa._{Dra. Veronica Teichrieb}

Centro_{de Informática/UFPE}

__________________________________________ Prof._{Dr. Windson Viana de Carvalho}

Instituto_{Universidade Virtual / UFC}

__________________________________________ Prof._Dr.Kiev_{Santos da Gama}

Centro_{de Informática / UFPE} (Orientador)

(5)

To my family, friends and professors. Each one of them gave me support to get here.

(6)

Acknowledgements

No man is an island, entire of itself; every man is a piece of the continent, a part of the main. If a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as if a manor of thy friend’s or of thine own were; any man’s death diminishes me, because I am involved in mankind,

and therefore never send to know for whom the bells tolls; it tolls for thee. —JOHN DONNE

My acknowledgments begin with a thanks to my advisor, Kiev Gama, who allowed a good learning environment, removed obstacles when necessary and was present to give support in times of difficulty. I would also like to thank the Informatics Center of UFPE (CIn), which provides an incredible environment for knowledge exchange.

I would like to thank my friends Rafael Roballo, Daniel Maranhão, and Vinicius Garcia. Along with Kiev they not only started a project that opened doors to the ideas developed in this work, but also gave the necessary support for the beginning of this journey. Among them, a special acknowledgment goes to Roballo due to the strong incentive he gave to start this research.

I am also grateful to all the friends who shared knowledge and gave strength during the research. In particular, I would like to thank Victor Rattis and Tássio Silva, who not measured efforts to help on experiments execution. I would also like to thanks Bruno Cartaxo for being my methodological guru sometimes and Antônio Sá Barreto for being supportive during my direction definition. My gratitude goes to many other friends that are not listed here, but were important during this phase.

Finally, I would like to express my thanks to my family. My wife, father, mother, sister, mother-in-law, father-in-law, sister-in-law, grandmas, uncles and cousins. Due to the huge number of people in this area of my life, I will not list each one name here. But their support was crucial so that I could get here and do this work.

(7)

I do not believe a man can ever leave his business. He ought to think of it by day and dream of it by night. —HENRY FORD

(8)

Abstract

Internet of Things (IoT) is considered as another step in the evolution of the Internet we have today and it has gained significant attention from academia and industry. Researches related to RFID, NFC (Near Field Communication) and other pervasive and IoT technologies is increasing, providing ways to make real world connections to the virtual world. Related to IoT, Web of Things (WoT) aims to create things representation on web through HTTP. Real world objects are being tagged with QR Code, NFC tags, and other technologies to provide a web presence of that object with specific URLs. This work raise scenarios where it is also possible to add web presence for non computational things through the usage of general purpose devices (like smartphones or wearables), improving the way users interact with their surroundings. With the proposed mechanism is possible, for example, to tag things virtually on a non intrusive way, without the need to change real environment, recognizing images (like paintings) and locations to associate an URL with detailed information about the identified thing, providing virtual experiences from physical world.

Keywords: Software Engineering. Internet of Things. Web of Things. Web Presence for Things. Mobile Physical Interactions.

(9)

Resumo

Internet das Coisas (do inglês, Internet of Things - IoT) é considerada como um outro passo na evolução da Internet e tem ganho atenção significante da academia e industria. Pesquisas relacionadas a RFID, NFC (Near Field Communication) e outras tecnologias pervasivas e de IoT estão expandindo, provendo meios de realizar conexões entre o mundo real e o mundo virtual. Relacionada com a IoT, a Web das Coisas (WoT) visa criar representação para as coisas na web utilizando HTTP. Objetos do mundo real estão sendo marcados com QR Code, tags NFC e outras tecnologias para prover presença na web por meio de URLs para um determinado objeto. O presente trabalho aponta cenários nos quais é possível adicionar presença na web para coisas não computacionais com uso de dispositivos de propósito geral (como smartphones ou wearables), melhorando a forma que usuários interagem com as coisas ao seu redor. Com o mecanismo proposto é possível, por exemplo, marcar coisas virtualmente de forma não intrusiva, sem necessidade de alterações no ambiente real, convertendo imagens (como pinturas) e localizações em URLs relacionadas. Tais URLs direcionam para informações detalhadas da coisa identificada, provendo experiência virtual a partir do mundo físico.

Palavras-chave: Engenharia de Software. Internet das Coisas. Web das Coisas. Web Presence para Coisas. Interações Mobile Physical.

(10)

List of Figures

2.1 Results of RUKZIO et al. (2006) on-line questionnaire regarding the

use-fulness of some application areas . . . 23

2.2 Results of RUKZIO et al. (2006) on-line questionnaire regarding user location preference . . . 24

2.3 Results of RUKZIO et al. (2006) on-line questionnaire regarding partici-pants preferences among interaction techniques . . . 24

2.4 Scans of the paper prototype created by RUKZIO et al. (2006) . . . 25

2.5 Evolution of the Internet . . . 26

2.6 Web presence for things . . . 27

2.7 People, places and things infrastructure . . . 27

2.8 Tag resolver . . . 28

3.1 Quantity of questionnaire respondents per age . . . 32

3.2 Educational level distribution of questionnaire respondents . . . 33

3.3 Usefulness of related web site acquisition responses distribution . . . 33

3.4 Responses related to the usefulness of object status information retrieval . 34 3.5 Answer related to the possibility to operate objects or some environment characteristics through mobile phone . . . 34

3.6 Answers to the question related to user position of reference from object . 35 3.7 Comparison of responses regarding usefulness of related web site acquisition 35 3.8 Comparison of responses regarding usefulness of object status retrieval . 36 3.9 Comparison of responses regarding usefulness of object control . . . 37

3.10 Comparison of user position of reference from object . . . 37

3.11 Comparison between interaction type preferences . . . 38

4.1 TAG resolution overview . . . 40

4.2 Mobile application design overview . . . 43

4.3 Source code of pointing abstract class . . . 44

4.4 Source code of touching interface . . . 45

4.5 Source code of scanning interface . . . 46

5.1 Features detected using Oriented FAST and Rotated BRIEF (ORB) . . . . 49

5.2 Features matching example . . . 50

5.3 Multiple Index Tables . . . 53

5.4 Dataset images example . . . 57

5.5 Image matching results . . . 58

(11)

6.1 Application screenshots . . . 64

6.2 Pointing improvements on application . . . 65

6.3 Scanning and touching improvements on application . . . 66

6.4 Experiments with users . . . 66

6.5 User satisfaction concerning image matching usage . . . 68

6.6 User satisfaction concerning QR Code usage . . . 68

6.7 User satisfaction concerning GPS usage . . . 70

6.8 User satisfaction concerning Bluetooth usage . . . 70

6.9 User satisfaction concerning NFC usage . . . 71

6.10 User satisfaction general scores . . . 73

(12)

List of Tables

5.1 Image dataset composition. . . 56 5.2 Image matching results categories. . . 56 5.3 Precision and Recall comparison of image search methods without false

positives reduction. . . 58 5.4 Precision and Recall comparison of image search methods with false

(13)

List of Acronyms

IoT Internet of Things . . . 16

WoT Web of Things . . . 17

URL Unified Resource Locator . . . 16

NFC Near Field Communication . . . 16

RFID Radio-Frequency Identification . . . 16

REST Representational State Transfer . . . 17

HTTP Hypertext Transfer Protocol . . . 17

HTML Hypertext Markup Language . . . 17

BLE Bluetooth Low Energy . . . 17

SIFT Scale Invariant Feature Transform SURF Speeded-Up Robust Features ORB Oriented FAST and Rotated BRIEF . . . 9

FAST Features from Accelerated Segment Test BRIEF Binary Robust Independent Elementary Features LSH Locality-Sensitive Hashing . . . 52

FLANN Fast Library for Approximate Nearest Neighbors . . . 52

SOAP Simple Object Access Protocol . . . 28

Atom Atom Publishing Protocol . . . 28

GPS Global Positioning System . . . 20

(14)

Contents

1 INTRODUCTION 16

1.1 Motivation . . . 18

1.2 Research Goals . . . 19

1.3 Summary of Contributions . . . 20

1.4 Structure of the Dissertation . . . 20

2 BACKGROUND 21 2.1 Physical Mobile Interactions . . . 21

2.1.1 Touching . . . 21

2.1.2 Pointing . . . 22

2.1.3 Scanning. . . 22

2.1.4 Rukzio’s Interactions Analysis . . . 23

2.2 Web Presence for Things . . . 25

2.3 Web of Things . . . 28

2.4 Contextual Data . . . 29

2.5 Mixed Reality Applications . . . 29

3 EVALUATION AND COMPARISON OF INTERACTIONS 31 3.1 Goal . . . 31

3.2 Sample . . . 31

3.3 Instruments and procedure . . . 32

3.4 Results . . . 32

3.5 Discussion . . . 34

3.6 Threats to validity . . . 36

3.7 Conclusion . . . 38

4 URLS FOR EVERYTHING 39 4.1 Tagged Things Infrastructure . . . 39

4.1.1 TAG . . . 39

4.1.2 TAG creation . . . 40

4.1.3 TAG mapping . . . 41

4.1.4 The Path From Physical to URL . . . 41

4.1.5 Contextual Data Transfer . . . 41

4.2 Mobile Application . . . 42

4.3 URLs for Things (NFC and QR Code) . . . 47

(15)

5 URLS FOR IMAGES 48

5.1 Feature Detection and Feature Description . . . 48

5.2 Image Matching Methods . . . 50

5.2.1 Brute Force . . . 51

5.2.2 FLANN . . . 52

5.2.3 Multiple Index Tables . . . 53

5.3 False Positives Reduction . . . 54

5.4 Image Matching Methods Evaluation . . . 55

5.4.1 Dataset . . . 55

5.4.2 Image Matching Experiments . . . 56

5.5 URLs for Images Remarks and Use Case . . . 60

5.6 Threats to validity . . . 60

5.7 Conclusion . . . 60

6 EVALUATION 62 6.1 An External Developer Perspective of the Proposal . . . 62

6.1.1 URLs for Places . . . 62

6.1.2 External Developer Perspective Overview . . . 63

6.2 Physical Mobile Interaction Application . . . 64

6.3 Application Improvements . . . 65 6.4 Experiment . . . 65 6.5 Analysis . . . 67 6.5.1 Pointing . . . 67 6.5.2 Scanning. . . 69 6.5.3 Touching . . . 71 6.5.4 General comments . . . 72 6.6 Threats to validity . . . 73 6.7 Conclusion . . . 74 7 CONCLUSION 75 7.1 Related Work . . . 75 7.2 Future Work . . . 76 7.2.1 Infrastructure evolution . . . 76

7.2.2 Web application experiment . . . 77

7.2.3 Experiments with more technologies . . . 78

7.2.4 Smart things discovery experiment . . . 78

7.2.5 Object control and status retrieval . . . 78

(16)

APPENDIX 86

APPENDIX A - ON-LINE QUESTIONNAIRE 87

(17)

16 16 16

1

INTRODUCTION

For years radios had been operated by means of pressing buttons and turning dials; then as the technology became more sophisticated the controls were made touch-sensitive – you merely had to brush the panels with your fingers;

now all you had to do was wave your hand in the general direction of the components and hope. —DOUGLAS ADAMS (1979)

Internet of Things (IoT) is considered as another step in the evolution of the Internet we have today and it has gained significant attention from academia and industry (PERERA et al., 2014). One of the biggest reasons for the increased attention to this subject is related to the capabilities that the IoT aims to provide. It brings the promise to create a world where all objects will be connected to the Internet and communicating among themselves, requiring less human intervention.

The IoT term was initially introduced by ASHTON (2009) on a presentation which focused in the usage of Radio-Frequency Identification (RFID). According to him, RFID and sensor technologies could be used to enable computers to observe, identify and understand the world. But the Internet of Things concept currently includes more possibilities, referring to several things or objects capable to interact with each other (ATZORI; IERA; MORABITO, 2010).

QR Code, Near Field Communication (NFC) and several other pervasive technologies are now closely related to the Internet of Things, in addition to RFID. These technologies are being used to provide different paths to interact with the physical environment. Interest on researches related to them is increasing (MICHAHELLES et al., 2007). NFC tags are being used, for instance, to make real world connections with the virtual world, attaching digital information to physical things. NFC is a key technology on Internet of Things scenario and these tags may work as triggers to provide digital experience to users through many smartphones that support this technology or portable NFC readers. QR Codes and NFC tags are commonly used to store Unified Resource Locators (URLs), making it possible to access related content on the Web.

(18)

17

Some of these pervasive technologies are widely used in our daily life. But some others are emerging and providing different paths for interaction. As an example, it would be possible to provide data of an olfactory sensor to mobile device and capture smells from real world (DOBBELSTEIN; HERRDUM; RUKZIO, 2017). It would also be possible to use the inherent radiometric properties of different objects to identify them without being intrusive in the environment (ZOU et al., 2017).

DILLON et al. (2011) pointed that researchers are encouraged by the success of the web to interact with real world objects using web standard technologies, directing to a Web of Things (WoT) solution. It is an IoT related concept where things start to get a representation on the Web. Following the WoT approach, GUINARD; MUELLER; PASQUIER-ROCHA (2010) also mention that the Representational State Transfer (REST) architectural model defined by FIELDING; TAYLOR (2002) has been used to integrate data coming from real-world sensors, pointing REST as a dominant standard for web services, well known as RESTful services.

REST uses and inherits advantages of Hypertext Transfer Protocol (HTTP). It uses the web as platform for applications, taking some benefits from HTTP such as authentication, cryptography, scalability and caching. According to TURCHI et al. (2013), an effective IoT solution must use an architecture that considers data transmission and creation of services for applications, which is possible to perform using REST on top of HTTP. With REST, a developer can use HTTP not only to provide Hypertext Markup Language (HTML) web pages, but also to use it as a path to applications data transfer.

Not considering only objects, KINDBERG et al. (2002) describe a scenario on which people, places and things could have presence on the Web. It is exposed by them through some application examples on a project called "Cooltown". They presented cases such as a museum with interactive pieces of art, a conference room enhanced by web pages related to its equipments and also an augmented bus that provides information related to its position. According to the authors, it would be possible to use direct sensing technologies (such as Bluetooth beacons) or indirect sensing (like QR Codes) in order to provide URLs that represent environment related contents on web. The direct sensing concept was explored, for example, by Google on the Physical Web project (PHYSICALWEB, 2017), which proposes the usage of Bluetooth Low Energy (BLE) beacons in order to emit URLs to other devices that are close to the coverage area. On the other hand, the TagHunt platform (GAMA et al., 2015a) provides a web-based mechanism that applies some indirect sensing on scavenger hunt games through the usage of NFC tags or QR Codes, which are used to describe and solve game tasks.

Even that it is already being used some indirect sensing technologies for web presence, providing specific URLs for tagged things through NFC or QR Code (KINDBERG et al., 2002; GAMA et al., 2015a), it is still valid to execute researches involving other technologies, like image matching. It is also attractive the idea to perform GPS coordinates conversion into URLs, correlating a given position on world with a digital content. The "Cooltown" project provides paintings information through tags positioned close to these pieces of art, but content based

(19)

1.1. MOTIVATION 18

image retrieval techniques could be used without the need to change environment with a NFC tag or QR Code. Within this scenario, paintings could be virtually tagged using some computer vision techniques. This possibility was mentioned by KINDBERG et al. (2002), who also additionally pointed, for example, that a device could obtain latitude and longitude coordinates from a GPS sensor, translate into a zip code and then into an URL of a place’s web portal.

This work aims to contribute with another step on web presence for things through the usage of general purpose mobile devices (like smartphones or wearables). It aims to raise an approach which could create this kind of experience through the usage of some mobile physical interaction techniques. It also present use cases on which would be possible to tag things virtually with a non intrusive way, without the need to change real environment. The first use case approach, for example, uses some computer vision techniques to recognize images and provide an URL with detailed informations about it. The second detailed application approach raises a scenario on which geographical locations could be tagged. Then user may obtain URLs related with places close to user device position on earth and also provide this kind of information through the usage of some BLE beacons.

This research focused on how to unify the mobile physical interaction on an approach that could create a web browser for the physical world, retrieving links from everything. This work combines the idea of web presence reported by KINDBERG et al. (2002) with pointing, touching and scanning interaction approaches raised by RUKZIO et al. (2006). It also revisits the experiments performed by RUKZIO et al. (2006) in order to evaluate if the passed time changed user’s perception about the usefulness of some application areas. It was also evaluated about the interaction types itself, retrieving user toughs about each one of them. Some use cases are presented with these interactions and looking to the application of technologies that are not broadly applied nowadays.

Initially, we conducted an on-line questionnaire with an structure very close to the one presented by RUKZIO et al. (2006), raising findings that reinforce this work motivation. After that, a proposal was raised in order to create an infrastructure, which could support mobile physical interactions to provide web presence for everything. Along with that, some researches were performed in order to include technologies different of the ones that are currently being used to perform this kind of task. At the end, it was evaluated the infrastructure with an external developer and also application usage with some potential end users.

1.1 Motivation

According to PERERA et al. (2014), it had an estimation of 1.5 billions computers and 1 billion of mobile devices that were capable to connect to the Internet in 2004. The authors reported that these two categories will join other devices (objects, sensors, etc.) on future, moving to a prediction of about 50 to 100 billions of devices capable to be connected to the Internet until 2020. Another mentioned point is that sensors are becoming more powerful, smaller in size and cheaper, allowing a forecast of huge increase in the number of sensors for the upcoming decades.

(20)

1.2. RESEARCH GOALS 19

Many technologies are closely related to the IoT concept. MICHAHELLES et al. (2007) illustrated the possibility to use RFID to identify ownership of an item, pedigree verification, pharmaceutical products tagging or even to estimate the position of some objects using the information retrieved from some RFID readers.

IoT is intended to create a better world for humans, where objects understand feelings and needs of individuals and act without requiring explicit instructions (PERERA et al., 2014). Internet of Things is considered as another step in the Internet we have today, where pervasive devices and related services are broadly becoming parts of humans daily lives (CONTI et al., 2012).

RUKZIO et al. (2006) presented some application areas where humans could interact with real world using their mobile phones. As an example, it is possible to obtain information of an object current status. In this scenario, user would be allowed to know how much time is remaining to finish wash-machine process, identify if coffee machine is active or turned off, know how much water is left on cooler, see how many tables are empty on a restaurant, achieve cardiology monitor information, identify pet current location, and other information on different situations.

With mobile device, performing object controlling, interaction or environment config-uration is also possible. Scenarios related to this possibilities are: turn on or turn off some equipment, change luminosity intensity, control irrigation flow on garden, open a door, use mobile device as a ticket on public transport or use it as any payment method, lock a vehicle, etc.

Another application area presented by RUKZIO et al. (2006) is to access a website related to something on real world. It would be possible to obtain an URL with more information about a bike, a site related to an art piece on a museum, a nearby restaurant website, recipes possible to be made using a microwave, instructions of a wash-machine, and other pages on the Web.

This work is motivated by the heterogeneity of pervasive technologies to allow different WoT experiences, providing web presence for things through some different approaches and using physical mobile interaction techniques. The main idea is to create a web browser for real world, using the Web as a bridge to perform human digital interaction with things, through users’ smartphones.

1.2 Research Goals

This work has the objective to create an infrastructure on which would be possible to create web presence for things and access it using mobile physical interactions. Based on the previously mentioned motivation, the main research question investigated in this work is: Research question How to design an infrastructure to allow web presence for things using

(21)

1.3. SUMMARY OF CONTRIBUTIONS 20

To achieve the general objective of this work, some specific goals were defined. These minor objectives aim to provide the necessary information to answer main research question. The specific objectives are:

1. Evaluate user’s interest over mobile physical interactions and related application areas.

2. Propose a general infrastructure on which would be possible to include some perva-sive technologies on a mobile physical interaction scenario.

3. Propose new technologies to be included into a mobile physical interaction solution. 4. Evaluate user’s perception of such technologies through some applications using the

proposed infrastructure.

1.3 Summary of Contributions

The following list enumerates the contributions of this work:

1. New evaluation of user’s perception about mobile physical interactions based on the questionnaire presented by RUKZIO et al. (2006).

2. Proposal of an infrastructure to support mobile physical interactions whose focus is to provide web presence for everything.

3. Identification of image matching technologies to be included on this application scenario.

4. Evaluation of the proposed infrastructure over a developer’s perspective.

5. Usage of TAG map to convert location into URL through the usage of Global Posi-tioning System (GPS) and BLE beacon data.

6. Evaluation of an application with the proposed infrastructure over user’s experience perspective.

1.4 Structure of the Dissertation

This work is structured as follows: Chapter 2 present a background context for this work. Chapter 3 covers the reconstruction of a study previously performed by RUKZIO et al. (2006). Chapter 4 presents the proposed infrastructure. The end of Chapter 4 and Chapter 5 show the inclusion of different technologies into the proposed infrastructure. Chapter 6 presents some experiments and studies that were performed to validate the proposal. Finally, Chapter 7 presents the conclusions and possibilities of some future works.

(22)

21 21 21

2

BACKGROUND

This chapter provides concepts and applications related to the current work. Initially some physical mobile interactions approaches are presented: touching, pointing and scanning. The subsequent section provides information about web presence for things. And the final section of this chapter will cover some necessary information about context, concept that will be explored in a piece of this work.

2.1 Physical Mobile Interactions

The paper ”An experimental comparison of physical mobile interaction techniques: Touching, pointing and scanning”presented by RUKZIO et al. (2006) raised, evaluated and compared some physical mobile interaction approaches. At that point, the authors presented three types of interaction: touching, pointing and scanning.

Later, the same main author also included an user mediated interaction experience (RUKZIO et al., 2007). This user-mediated interaction technique considered that user would type in some information provided by the object on mobile device, establishing a link between them. This interaction technique will not be covered in this work and focus will be centered on the three previously mentioned approaches.

2.1.1 Touching

According to RUKZIO et al. (2006), touch is the interaction that involves an action on which user’s mobile device is placed into contact with an smart object the user wishes to interact with. In this case, user must somehow be previously aware about the possibility to perform this interaction with the object, like adding instructions close to the interaction point. This kind of interaction may be performed using some Near Field Communication tags.

Some years before the emergence of NFC tags, WANT et al. (1999) raised some examples of RFID usage to tag objects. One of the presented examples was a business card that could be related with the web page of card owner. The authors also presented the augmentation of some other things with RFID like documents, photo cubes, watches and books. In this same line, RIEKKI (2007) showed a scenario on which NFC tags could provide URLs, SMSs or

(23)

2.1. PHYSICAL MOBILE INTERACTIONS 22

phone numbers to user’s mobile device.

The touching interaction is also being used, for example, to interact with displays through a grid of tags (BROLL et al., 2011). This interaction approach was also used to estimate user location. RASHID et al. (2006) raises a location and object enhanced mixed-reality version of Pacman game through the usage of Near Field Communication. In this game, the authors avoided the usage of GPS and estimate user location through NFC tags readings.

2.1.2 Pointing

The pointing interaction is broadly used within augmented reality applications through the usage of mobile device camera (MILGRAM; KISHINO, 1994). In this kind of application, user points to an augmented content and a projection related to that content is overlapped on camera content into display. Augmented Reality applications perform, for example, the recognition of contents such as fiducial markers or perform some template matching procedure (LIMA et al., 2010).

According to RUKZIO et al. (2006), the pointing interaction approach allows the user to interact with a smart object pointing to it with a mobile device. The authors mention that this kind of interaction is as natural as our everyday physical interaction to point at objects with our finger. By the end, the authors presented the usage of this kind of interaction with the help of a laser bean on mobile device and light sensors on smart objects. But one possibility raised by RUKZIO et al. (2006) is to perform pointing interaction through the usage of different technologies and one of them is image recognition.

A technology that already has a widespread usage is QR Code. This kind of technology is present in many application areas and can be used to tag objects, giving them a virtual representation. As an example, it is being used to provide web links related to a given thing or even to solve virtual games tasks (GAMA et al., 2015a).

2.1.3 Scanning

A scanning interaction approach may return a list of nearby smart objects. It is mentioned by RUKZIO et al. (2006) that one advantage of this interaction technique is that user may not be aware of thing’s augmentation to perform the desired task. The user may, for example, press a button on mobile application that will search for every nearby object capable to be recognized for interaction.

KINDBERG et al. (2002) presented the usage of this kind of interaction to provide information related to places. Some well known applications also provide scanning experience to retrieve nearby specific establishments information (FOURSQUARE, 2017; TRIPADVISOR, 2017).

A platform on which would be possible to tag things with technologies such as GPS or BLE beacons was presented by GAMA et al. (2015b). This kind of technology could support

(24)

scanning interaction, providing content related to a geographical space or even to a Bluetooth beacon close to user’s device.

2.1.4 Rukzio’s Interactions Analysis

Questions were answered on an on-line questionnaire applied by RUKZIO et al. (2006) containing some specific points regarding each interaction type (touching, pointing and scanning). This questionnaire was performed in order to identify the needs of potential users. It also aimed to verify which services could be useful when performing an interaction between an smartphone and a smart physical device. The authors also wanted to identify user’s preferences of interaction type and the places and context on which users would interact with smart objects.

A total of 134 people answered the questionnaire. They were between 17 and 59 years old with an average age of 28 years. One section of the questionnaire was related to three application areas to perform interaction between mobile phones and smart objects. The first one was about information acquisition which is related to an object, directing the user to an web page. The second application area was about retrieving status information about physical entities, such as the time a washing machine needs to complete its process. The last application was about remotely control a device, like turn on or turn off an specific machine. The results of usefulness regarding these three application areas are displayed on Figure 2.1.

Figure 2.1: Results of RUKZIO et al. (2006) on-line questionnaire regarding the usefulness of some application areas.

Source: RUKZIO et al. (2006).

It is possible to note that 37% of the respondents answered information retrieval on related websites as a useful task, 36% said that they were neutral about the usefulness of this application and 27% reported it as not useful. This was the question regarding application areas most closed in score, showing that the opinions about this point were very balanced. Related to the status information application, 64% of the participants related such service as useful. The task to control a device was also well scored, with 73% of the respondents considering this application area as useful.

Another section of the questionnaire was related to when they would use the mobile phone for interactions with objects in smart environments, regarding user position in relation

(25)

with the target object. The Figure 2.2 displays that most of the people would use this interaction independently of their location. Close to it, other users would use this interaction while distant to the target object. A small remainder part of the participants would use this interaction when close to the object (13%) or would never use this kind of application (10%).

Figure 2.2: Results of RUKZIO et al. (2006) on-line questionnaire regarding user location preference.

The last point of evaluation mentioned by RUKZIO et al. (2006) was about user preference among touching, pointing and scanning experiences. Figure 2.3 summarize the results, showing that most of the users did not reported to like the experience of a touching interaction approach. The reason of that most of the users did not like the touching interaction is related to the lack of value added by this technique and also the unnecessary effort to perform the task. But respondents also mentioned that touch would be helpful to avoid ambiguity, increasing accuracy and uniqueness of the identification process especially when devices are close together. The authors reported that participants saw pointing as an intuitive interaction technique with little physical effort. Also they mentioned that scanning was preferred in situations in which exists a physical distance between the user and the target object.

Figure 2.3: Results of RUKZIO et al. (2006) on-line questionnaire regarding participants preferences among interaction techniques.

RUKZIO et al. (2006) raise that the benefits of pointing are seen as being natural, easy to use and quick for addressing the target object directly. The respondents of the questionnaire

(26)

2.2. WEB PRESENCE FOR THINGS 25

also mentioned that pointing would avoid a complex user interface. On the other hand, scanning reported benefit was that it operates at a distance and does not require proximity to the device. While using scanning, listing of all devices was also seen as an advantage.

After questionnaire analysis they developed a low-fidelity and a high-fidelity prototype to refine their findings and to evaluate the interaction techniques in a more practical context. Basically they compared user preferences of interaction type when performing some specific tasks, like identify a recipe related to a fridge or to perform some action such as choose microwave process time. The Figure 2.4 displays the paper prototype that was created to evaluate user interaction with the application. The point (a) illustrate the selection of interaction approach, (b) displays an scanning result, (c) is the screen intended to provide pointing functionality, (d) show the user that the touching interaction was chosen, (e) displays the selected object and (f) illustrate the usage of a service provided by the smart object.

Figure 2.4: Scans of the paper prototype created by RUKZIO et al. (2006).

2.2 Web Presence for Things

As it was previously mentioned, many technologies are related to the Internet of Things, a term introduced by ASHTON (2009) that now refers to the presence of several things or objects capable to interact with each other (ATZORI; IERA; MORABITO, 2010). Over the last decade IoT has gained significant attention from academia and industry because of the capabilities it aims to provide. Internet of Things has the promise to create a world where all objects will be connected to the Internet and communicating among themselves. PERERA et al. (2014) places IoT as another step in the evolution of the Internet, illustrated on Figure 2.5, where devices and other things are all interconnected.

Technologies such as NFC tags are being used to make real-world connections to the virtual world (MICHAHELLES et al., 2007). These tags may provide digital experience on smartphones and portable NFC readers trough their presence on the real world. These

(27)

Figure 2.5: Evolution of the Internet.

Source: PERERA et al. (2014).

technologies are being used, for example, to obtain more information on places like museums (MARTELLA et al., 2016). One example of this usage is exposed by RUDAMETKIN et al. (2010) on a scenario which museum visitors can use mobile devices to scan RFID tags or barcodes of items and receive information about these items. PESONEN; HORSTER (2012) and BORREGO-JARABA; RUIZ; GÓMEZ-NIETO (2011) also reinforced the potential usage of technologies such as NFC in tourism. Another important technology related to IoT is QR Code, which allows non computational things to have a virtual representation (GAMA et al., 2015a).

Along with the evolution of Internet of Things and pervasive technologies, DILLON et al. (2011) pointed that some researchers are motivated by the success of the web to create virtual interactions with real world objects using standard web technologies, directing to a WoT solution, where things start to get a representation on web. GUINARD; MUELLER; PASQUIER-ROCHA (2010), for example, mentioned that the REST architectural model has been used to integrate data coming from real-world sensors.

TURCHI et al. (2013) reported that an effective IoT solution must use an architecture that considers data transmission and creation of services for applications. It would allow possibilities such as the inclusion of real world objects as resources on a REST API, providing a representation of each thing on web.

KINDBERG et al. (2002) raised on paper ”People, Places, Things: Web Presence for the Real World”the concept of web presence for things, people and places. The authors present an infrastructure that aims to provide support for nomadic users. According to them, things not only become present on web when web services are embedded inside them, but also through the creation of services which represent these things on web as displayed on Figure 2.6. This is an idea that has been explored by Google on Physical Web project (PHYSICALWEB, 2017), which proposes the usage of URLs to virtually tag smart objects through the usage of BLE beacons.

Some experiments were performed by KINDBERG et al. (2002) in order to evaluate the proposed infrastructure, which raised aspects like context, URL, URL sensing and ID resolution.

(28)

Figure 2.6: Web presence for things.

Source: KINDBERG et al. (2002).

A general overview of this infrastructure is displayed on Figure 2.7.

Figure 2.7: People, places and things infrastructure.

The authors of ”People, Places, Things: Web Presence for the Real World” presented a concept of Tag resolver, that from an ID would be capable to retrieve an URL related to an specific thing as displayed on Figure 2.8. KINDBERG (2002) reinforced the Tag resolver and also presented on another paper a different idea of tag, which would be a new type of URI. His main objective was to create unique identifiers which could be converted by resolver on different contents, depending of user context.

The possibility to create web presence for things is just a part of WoT. But this concept may take huge advantages from web solutions, specifically from HTTP protocol. This would not only allow objects to have a representation on web, but also to use everything as trigger to digital experiences.

(29)

2.3. WEB OF THINGS 28

Figure 2.8: Tag resolver.

2.3 Web of Things

The action to make it possible to manage and access smart objects through web standards and technologies refers to another concept related to IoT, which is called WoT (PAGANELLI; TURCHI; GIULI, 2016).

Some works bring the idea of things representation on web. Many researches work on the path not only to create web representation for things, but making smart objects functionalities accessible through web standards. DILLON et al. (2011) presents an WoT framework combined with a cyber-physical System perspective. A service-oriented middleware architecture for things is presented by DAT; HYE; CHAE (2011) with the possibility to represent the service as REST, Simple Object Access Protocol (SOAP) or Atom Publishing Protocol (Atom). GUINARD; TRIFA (2009) presents an approach to integrate embedded devices on web using RESTful design principles. GUINARD; TRIFA (2009) also shows how RESTful devices can make it easier to create cyber-physical related pages on web. They point that using REST and devices which are able to connect to the web is possible to prototype all kinds of applications.

On the other hand, KINDBERG et al. (2002) presents the side of WoT where objects are placed on real world, no interaction can be performed directly to them, but they have representation on web. KINDBERG et al. (2002) called this thing representation as web presence. On a similar direction, RUKZIO et al. (2006) presented an architecture to support mobile physical interactions with services on a WoT context. It is created an interaction proxy that considered reasoning, UI generation and service composition to provide virtual functionalities to tagged objects. The authors vision is strongly related to this work, they not only expect to interact with electronic devices but with anything like posters, magazines and other objects. One of the main differences of the current work is that the current work not only uses some of Rukzio’s concepts, but also brings the Tag resolver presented by KINDBERG et al. (2002) to the mobile physical interaction scenario presented by RUKZIO et al. (2007). Another different point of this work is that it simplified on point, reasoning will not be considered as part of the infrastructure, but only contextual data will be transfered to the requested web applications.

(30)

2.4. CONTEXTUAL DATA 29

2.4 Contextual Data

SCHILIT; THEIMER; WELCH (1993) reported that many applications were designed assuming that systems would not change over time. At that time, they proposed a mechanism to dynamically customize applications through the usage of some environment variables. Variables such as LOCATION could be changed with current position. System could use these environment variables information to provide the better functionality according to each value.

Since then, the concept of context-aware systems has being discussed a lot. Most of the examples of context used identity, location and time as relevant contextual information to be handled (FENG; APERS; JONKER, 2004). BETTINI et al. (2010) also include on this list two other information that are object type and user’s activity, calling this group as primary context. According to them, secondary context can be derived from these primary contextual information. BETTINI et al. (2010) mention that context-awareness is being used as a technique to develop applications that are flexible, adaptable, and capable of acting autonomously on behalf of users.

COUTAZ et al. (2005) report context as key in the development of systems which aim to impact social inclusion for information society. One of the most referred definition of context is presented by DEY (2001), that defines context as: ”Context is any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves.”.

COUTAZ et al. (2005) specify that context is defined by entities, a set of roles that entities may satisfy, relations between entities and situations. ZIMMERMANN; LORENZ; OPPERMANN (2007) defined context placing entities as the center of the definition, where these entities can be real or virtual. They categorized contextual information as individuality, activity, location, time, and relations of an entity. Such categorization aimed to provide a definition of a set of contextual data that could be used by context-aware systems. NETO; PIMENTEL (2005) used the W4H questions in order to identify a given context. Their idea would be to identify context based on identity (who), location (where), time (when), activity (what) and devices characteristics (how). These two categorization differ directly on the presence of entity relations on one side and devices characteristics on another. But the definition of ZIMMERMANN; LORENZ; OPPERMANN (2007) cover this scenario since, according to them, an entity could be the device, and their characteristics could be covered by its individuality.

2.5 Mixed Reality Applications

Computer games are popular by the fact it allows that virtual fantasy and imagination may came true (CHEOK et al., 2002). But the digital game experience is usually and widely used through the interaction with a screen on a 2D context. In this scenario, it becomes interesting to search approaches on which the exciting elements of games can be applied to user experiences

(31)

2.5. MIXED REALITY APPLICATIONS 30

with the real world. Within TagHunt (GAMA et al., 2015a), for example, it is possible to interact with public spaces such as museums, shopping malls, squares, etc. To illustrate, a NFC tag can be placed hidden on a museum piece of art which provides the necessary information to solve a specific task of the scavenger hunt game. In general, TagHunt is a Mixed Reality Game which has the objective to provide creation and execution of scavenger hunt games using things from physical world with a digital experience.

The term Mixed Reality refers to the combination of physical world with virtual world (MILGRAM; KISHINO, 1994). This concept has being used on digital games as Mixed Reality Games, combining physical and digital world to provide a novel way of experience on games. This experience may use some technologies such as Bluetooth, RFID, bar codes, QR codes, NFC, among others. Many games engaged in a Mixed Reality experience uses geo location as the way of interaction with the real world. However, the usage of real objects may improve the experience and encourage a huge immersion on game fantasy (COULTON; RASHID; BAMFORD, 2006).

The coverage area of such games is very broad. As an example, GARRIDO et al. (2011) presents a solution that uses common strategy game features integrated to an evaluation system which makes use of Moodle, a platform to create custom learning environments (MOODLE, 2017). The game uses an NFC based experience to combine the rewards of study process with students’ learning motivation.

Games created on TagHunt can be executed on public environment to create ludic experiences with the aid of technologies, expanding the citizen (user) interaction with the urban space (SCHAFFERS et al., 2011). The games are related to physical objects, but the execution works on virtual through some different technologies.

TagHunt is a game which is executed on a web browser. This work is closely related to the TagHunt idea. It could leverage TagHunt possibilities, including technologies different of the ones that were previously used. On the experiment reported on GAMA et al. (2015a), only NFC tags and QR Codes were used to provide task solving functionalities. With the proposal of this work, technologies such as Bluetooth Low Energy or image matching functionalities could also be used to solve game tasks.

(32)

31 31 31

3

EVALUATION AND COMPARISON OF INTERACTIONS

This chapter will present the results of a questionnaire strongly based on the one applied by RUKZIO et al. (2006). This questionnaire is related to the usefulness of some mobile to physical interactions approaches and some application areas that could be applied with these interactions.

3.1 Goal

As it was mentioned before, the paper ”An experimental comparison of physical mobile interaction techniques: Touching, pointing and scanning” (RUKZIO et al., 2006) reported the results of a questionnaire containing some specific points regarding touching, pointing and scanning interaction approaches.

Once it happened a huge change on the number of mobile devices and it became more part of users’ lives since RUKZIO et al. (2006) publication, it is interesting to validate again people perception regarding these physical mobile interaction techniques. This evaluation would allow to identify differences on user feelings related to such approaches. User’s familiarity with more technologies is a key factor to provide some differences on their perception about this subject.

The following sections will present the creation of a questionnaire strongly based on the one presented by RUKZIO et al. (2006), the discussion related to its results and some insights achieved based results analysis.

3.2 Sample

In the study performed by RUKZIO et al. (2006) a total of 134 people answered the on-line questionnaire. Close to this number, a total of 130 people answered the questionnaire presented on Appendix 7.2.5.

The questionnaire was spread to people mainly located in the city of Recife, Pernambuco state, in the northeast of Brazil. It was sent to respondents of different ages in order to achieve opinions from different perspectives.

(33)

3.3. INSTRUMENTS AND PROCEDURE 32

3.3 Instruments and procedure

Since the questionnaire which results were presented by RUKZIO et al. (2006) was not provided, another one was created based on author explanation, including some adjustments. The questionnaire was written in Portuguese and made available through Google Forms (GOOGLE, 2017a) on web. This questionnaire is detailed on Appendix 7.2.5.

The questionnaire was sent to respondents using the Google Forms site. The link was sent using some social medias, e-mail and message systems with a message asking to answer the questionnaire. All the answers were performed on-line by respondents without further assistance.

3.4 Results

The first part of the questionnaire was focusing on some general questions. The first one focused on user age, which presented a result close to the one raised by RUKZIO et al. (2006). The authors reported that the population ranged from 17 to 59 year and an average of 28 years. In the current questionnaire, respondents reported ages that range from 17 to 73 year old. Five of these respondents did not answered their age. The average in this case was about 33 years. The distribution of these answers can be seen on Figure 3.1.

Figure 3.1: Quantity of questionnaire respondents per age.

Source: Made by the author.

Some questions were raised in order to identify the population. On the paper written by RUKZIO et al. (2006) there was 41% of the respondents with an university degree. Summing the numbers of people with university degree completed and post-graduation completed or in course, it will be about 68,4% with an university degree or above. The results presented on Figure 3.2 display that there is a balance in the distribution of the population but a lack of respondents on a section previous to high school.

After the general questions, some others had the objective to identify user perceptions of usefulness regarding some application areas. These questions aimed to identify if the users

(34)

3.4. RESULTS 33

Figure 3.2: Educational level distribution of questionnaire respondents.

believe that applications such as related web site identification, real-time information retrieval or control capabilities would be useful.

On the current questionnaire, the question regarding usefulness of related web site identification was reported as displayed on Figure 3.3. 70% of respondents thought it would be very useful allied with 22,3% that thinks it is only useful. Only 7,7% of them answered that they were neutral to this functionality or believe that it is not useful, but only one respondent (0,8%) reported that it is really not useful.

Figure 3.3: Usefulness of related web site acquisition responses distribution.

The next question of the questionnaire aimed to identify if users believe that obtain current status of an object is a useful application when performing a physical mobile interaction. The Figure 3.4 illustrates the answers of this question. Very useful and useful together represents 82,3% of the population. A little bit more people are neutral to this application when compared to the previous question, represented by 10,8% of the respondents. The remainder part of the group (6.9%) are the ones that this not believe it could be a useful functionality.

(35)

charac-3.5. DISCUSSION 34

Figure 3.4: Responses related to the usefulness of object status information retrieval.

teristics through the mobile phone. As displayed on Figure 3.5, 77,7% of the respondents said it could be a useful application, 11,5% were neutral to such functionality and 10,8% told that it was not useful.

Figure 3.5: Answer related to the possibility to operate objects or some environment characteristics through mobile phone.

The question related to the place of reference between user and the interaction object raised that most part of the users would like to perform this kind of interaction while distant from the object. A second big part of would perform this kind of task independent of position. These results are displayed on Figure 3.6.

3.5 Discussion

This section aims to present a discussion of the data acquired through questionnaire responses. The results here will be compared with the results reported by RUKZIO et al. (2006) in order to verify if the time distance of questionnaire execution made some difference on people perception about the points which are being investigated.

(36)

3.5. DISCUSSION 35

Figure 3.6: Answers to the question related to user position of reference from object.

The first comparison is related to the question about usefulness of website acquisition using a mobile physical interaction approach. In order to compare these results with the one provided by RUKZIO et al. (2006), some answers were combined. In this question, "useful" and "very useful" responses were grouped into "useful". The answers of "few useful" and "not useful" were combined on "not useful". This grouping allowed these answers to be directly compared between each result. The comparison of these answers is displayed on Figure 3.7, where it is possible to see a huge difference on usefulness perception of the respondents.

Figure 3.7: Comparison of responses regarding usefulness of related web site acquisition.

Another comparison was performed to the question regarding object status retrieval. Like on the question related to web site identification, this one was also converted to a format

(37)

3.6. THREATS TO VALIDITY 36

that could be compared with the previous touching, pointing and scanning interaction types evaluation. Despise the fact that there is no such difference as the previous question, there is another positive change on results. In general, more people believe on the usefulness of this kind of application. Basically, part of the percentage that was previously designed to "neutral" and "not useful" responses changed to a useful perception of the subject. These results can be better compared on Figure 3.8.

Figure 3.8: Comparison of responses regarding usefulness of object status retrieval.

A more close result between researches can be seen on Figure 3.9. This result is related to users’ perception about the usefulness of mobile interactions to perform object controlling operations. Basically it is possible to see that there is no relevant difference on users’ perception about the usefulness of this kind of activity.

Comparing the results of when users would perform an interaction between an object or environment and a smartphone, some minor changes where perceived in benefit of a remote user placement. More users got interested about the idea to be positioned far from the object, as can be seen on Figure 3.10.

The last comparison is related to the preferences regarding interaction approaches. Even that pointing interaction type got less scored in this question, in general users are more favorable to physical mobile interactions. Probably this is allied to the evolution of related technologies and the rise of applications that make use of these approaches. The Figure 3.11 show that a bigger percentage of respondents would interact through touching or scanning approaches.

3.6 Threats to validity

Some limitations and threats to validity are related to the current research. The first limitation is related to need to recreate the questionnaire applied by RUKZIO et al. (2006).

(38)

3.6. THREATS TO VALIDITY 37

Figure 3.9: Comparison of responses regarding usefulness of object control.

Figure 3.10: Comparison of user position of reference from object.

(39)

3.7. CONCLUSION 38

Figure 3.11: Comparison between interaction type preferences.

Since it was not possible to use the original questionnaire, it was recreated based on authors explanation about the questions. To minimize the threats to validity, the questionnaire was sent to a population close to the one described by then in therms of age and number of people. But even that the number of participants and the age were close to the original study, differences of context and culture (since it was applied on a different country) may provide divergences on result.

Another threat to validity is related to the perception about usefulness of interactions related to real usage. Respondents answered to questions without have a prototype or application to effectively experiment what was being asked.

3.7 Conclusion

It is important to notice that some user perceptions changed since the initial study was applied by RUKZIO et al. (2006). One of the main changes is related to the usefulness of retrieve content related to an object through a website. This particular opinion give this work a strength point regarding its usefulness to users, since one of the main objectives of this work is to provide a mechanism to extract URLs from anything that surrounds our world.

Besides that, there is a most favorable environment to the adoption of different interaction types. Touching, pointing and scanning are more present nowadays and users are getting used to perform tasks in this way.

(40)

39 39 39

4

URLS FOR EVERYTHING

This research groups some findings presented by RUKZIO et al. (2006) with the idea of web presence raised by KINDBERG et al. (2002). This combination is performed through the definition of an infrastructure which aims to create digital links that can be retrieved from physical world, giving for each thing a representation on web.

The first section of this chapter presents the proposed infrastructure. This proposal uses the touching, pointing and scanning interaction techniques with the final goal to extract URLs from real world.

At the end of this chapter, an use case scenario of such infrastructure is presented. The idea of this use case is to start the infrastructure ability to provide URLs from things based on some technologies.

4.1 Tagged Things Infrastructure

Empowered by the fact that identify web sites related to physical entities is a relevant application area for physical mobile interactions, this section brings a solution which aims to provide a common path to the URL retrieval from things.

The main idea of this infrastructure is to create a web browser to the physical world. This browser focus on URLs extraction from things that belongs to the real world through the usage of mobile devices. This interaction would allow physical entities to work as trigger to digital experiences. The Figure 4.1 shows a general overview of the platform and will be detailed on the next sections.

4.1.1 TAG

The core of this infrastructure is the TAG, an abstraction which is related to the ability to create web presence for some entities. In this case an entity may be physical such as an NFC tag attached on an object or abstract like GPS coordinates representing one place.

At the end, a TAG will represent and be related to something from real world. At this point, a tagged thing may be represented by objects, places or something even more abstract like sounds or smells. If it would be possible to provide data of an olfactory sensor to mobile device

(41)

4.1. TAGGED THINGS INFRASTRUCTURE 40

Figure 4.1: TAG resolution overview.

and capture smells from real world (DOBBELSTEIN; HERRDUM; RUKZIO, 2017), it would also be possible to create a TAG for the identified smell. Techniques of audio watermarking (ARNOLD, 2000) and audio fingerprinting (HAITSMA; KALKER, 2002) may also be applied to capture sounds from real world and provide related URLs.

4.1.2 TAG creation

The TAG creator is an entity which aims to make an specific thing accessible on virtual world through a mobile device. This process is intended to be performed acquiring this virtual representation from some information on physical world. This process may be performed changing the environment like including a NFC tag, QR Code or BLE beacon. But this could also be performed on a non-intrusive way, using some information that is already present on real world through the usage of some technologies such as image matching, audio fingerprinting or GPS location.

The TAG creator must focus on how it will be read through mobile device. This makes it strongly related to the TAG reader and URL resolver entities. As pointed by WANT et al. (1999), a tagging system must consider not only the tag itself, but also keep in mind how the tag detector will be designed. The mobile device must embeds a TAG reader which is able to perform the reading operation for the desired interaction technique. Just to make it more clear, if there is a TAG creator that writes NFC tags, the mobile device must be capable to read these NFC tags through touch approach and call the proper URL resolver for it.

(42)

4.1. TAGGED THINGS INFRASTRUCTURE 41

4.1.3 TAG mapping

Sometimes, it will be necessary the usage of a TAG map. The map is responsible to perform a link between the content creator and URL resolver. The idea of this entity is to fill a gap when the reading technology is not enough to identify URL trough tagged thing. A practical usage of this map will be explained in details later in this work.

Some parts of the system may be performed on-line or off-line. This is the case of TAG creator and TAG map. A service could be created to store all necessary information to perform some image matching process and return the URL related to some image acquisition. The images and related URLs would be stored on a map inside an web service and later a request would be performed to resolve an specific image URL.

4.1.4 The Path From Physical to URL

The mobile device must be capable to read TAGs that were provided by a TAG creator. As an example, if it was created a QR code, the mobile device must have an embedded camera and also have enough technology to read this QR code. After reading process, device should also be capable to extract the URL which is related to the tag. This process is performed in a combination of TAG reader, URL resolver and TAG map (if necessary). The combination of these components in this particular scenario is the path to a web browser to the physical world. As explained before, this work consider the usage of three interaction approaches which are: touching, pointing and scanning. Each approach has a subset of technologies capable to read URLs from things which were provided by their specific TAG creators. These interaction techniques make it possible to retrieve information from real world through the usage of a mobile device. The idea behind the entire process is to identify URLs from everything that surround our world. If the user want to achieve more information about some specific thing, it will exist a way to identify its URL.

This type of interaction is already being used for example with NFC tags or with QR Codes (GAMA et al., 2015a). But the idea of this work is to create an infrastructure on which would be possible to integrate any kind of technology within the help of these interaction techniques.

4.1.5 Contextual Data Transfer

DEY (2001) reported that it is necessary to have architectural support to effectively use context. And according to BETTINI et al. (2010) the planning to use contextual information must be considered from the beginning. This is another interesting point of this infrastructure, the support to provide contextual data transfer among URLs calls.

BETTINI et al. (2010) mentioned that applications can make use of these information at runtime in order to be flexible and adaptable to user needs. Authors also reported that some contextual information modeling techniques used key-value pairs to describe contextual

(43)

4.2. MOBILE APPLICATION 42

information through a list of attributes and their values. A similar approach was presented by SCHILIT; THEIMER; WELCH (1993) storing the values of contextual information on environment variables.

In a similar key-value approach, the contextual data transfer presented in this work is possible using HTTP protocol through specific headers on every request performed by the application. The designed application includes on each request some contextual information related to the device. For example, it is sent on HTTP header the GPS coordinates of the device if available. It also send some device related information such as the list of technologies supported by the device and also device model.

It is useful for context-aware applications that need this kind of information to provide the best experience to the user. The TagHunt presented by GAMA et al. (2015a) could, for example, identify which technologies are available to the user and create quests that only use TAGs capable to be read through user’s device.

The applicability of this kind of data is very extensive. Different content can be rendered through the same URL call in order to better fit user’s or device’s contextual information.

4.2 Mobile Application

More details about how the infrastructure was validated will be reported on evaluation chapter. But one point that was performed during this research project was the creation of a mobile application grouping the presented interaction approaches. This application was created to be executed on Android1devices and uses Java2as its core language.

A general overview of how the application was structured can be seen on Figure 4.2. The ScanningController, PointingController and TouchingController are entities that control the user interface and communication with technologies to perform scan, point and touch interactions respectively. Each of these technologies are provided by specific readers. At the end, the BrowserController entity is responsible to open achieved web pages.

In order to facilitate the inclusion of new technologies inside these mobile physical interaction approaches, interfaces were provided to developers. The pointing interaction of this application basically uses camera. This allowed the creation of an abstract class (Figure 4.3) that holds camera frame size information, receives camera frame data and is able to provide response through a listener.

Implementing this interface allows as an example the analysis to identify a QR Code within camera frame image. Some other computer vision techniques could also be applied using these data and many implementation of this interface can live together inside the solution.

The touching interaction was structured on a path that would allow the usage of NFC capabilities of the device (Figure 4.4). It receives the device context on initialization, that

1_{http://www.android.com} 2_{http://www.java.com}

(44)

Figure 4.2: Mobile application design overview.

(45)

Figure 4.3: Source code of pointing abstract class.

/**

* Abstract class which aims to provide camera content reading

* functionalities. At the end it provides a pointing interaction approach. */

public abstract class ImageContentReader {

protected ImageContentReadListener listener;

protected int width;

protected int height;

/**

* Initialize the content reader abstract class with the necessary * information.

*

* @param listener The listener that will be called when some URL is

* identified.

* @param width The width of the frames that will be sent to

* analysis.

* @param height The height of the frames that will be sent to

* analysis.

*/

public ImageContentReader(ImageContentReadListener listener, int width,

int height) {

this.listener = listener;

this.width = width;

this.height = height; }

/**

* Analyse a given camera frame. *

* @param data The byte array representing camera frame data. */

public abstract void analyze(byte[] data);

}