ThermoDroid: Android based solution for temperature physiological assessment

(1)

Universidade de Aveiro Departamento deElectr ónica, Telecomunicaç ões e Inform ática 2019

Andr ´e Francisco

Pinelas da Silva

ThermoDroid: soluc¸ ˜ao baseada em Android para

avaliac¸ ˜ao fisiol ´

ogica atrav ´es de temperatura

ThermoDroid: Android based solution for

temperature physiological assessment

Dissertaç ão apresentada à Universidade de Aveiro para cumprimento dos requisitos necess ários à obtenç ão do grau de Mestre em En-genharia de Computadores e Telem ática, realizada sob a orientaç ão cient´ıfica do Doutor Jos é Maria Amaral Fernandes, Professor Auxil-iar do Departamento de Electr ónica, Telecomunicaç ões e Inform ática da Universidade de Aveiro, e do Doutor Il´ıdio Castro Oliveira, Profes-sor Auxiliar do Departamento de Electr ónica, Telecomunicaç ões e In-form ática da Universidade de Aveiro.

(2)

(3)

o j ´uri / the jury

presidente / president Professor Doutor Augusto Marques Ferreira da Silva

Professor Auxiliar do Departamento de Electr ónica, Telecomunicaç ões e In-form ática da Universidade de Aveiro

vogais / examiners committee Professor Doutor Sergi Berm ´udez i Badia

Professor Associado da Faculdade de Ci ˆencias Exatas e da Engenharia da Universidade da Madeira

Professor Doutor Jos ´e Maria Amaral Fernandes

Professor Auxiliar do Departamento de Electr ónica, Telecomunicaç ões e In-form ática da Universidade de Aveiro (orientador)

Professor Doutor Il´ıdio Castro Oliveira

Professor Auxiliar do Departamento de Electr ónica, Telecomunicaç ões e In-form ática da Universidade de Aveiro (co-orientador)

(4)

(5)

agradecimentos / acknowledgements

First and foremost I would like to thank my supervisors, and specially Professor Jos ´e Maria Amaral Fernandes, for all the support, ideas, guidance and comprehension. I am very grateful for his constant avail-ability for discussing the dissertation direction and progress. In addi-tion to the time he made available, the suggesaddi-tions and commitment throughout this work facilitated its development. I could not fail to men-tion Francisco Martins, for the setup of the experiment and camaraderie throughout this work.

I would also like to thank my family for their help and unconditional support.

Over the years there have been also many relatives, friends and col-leagues who have backed me up and to whom I am recognized. There-fore, without distinguishing anyone, i leave this reference to all of those who encouraged and prodded me. I am sure they will recognize them-selves as recipients of these words.

(6)

(7)

Keywords Low cost Infrared thermal imaging, psychophysiological assessment, Internet of things, Bluetooth, Android, Kafka message broker

Abstract Currently, the recognition of emotions has assumed an important role in the domain of human computer interaction. However, the devices used are, as a rule, invasive and limit the freedom of movement of the individual.

This dissertation proposes Thermodroid, a non-intrusive, non-contact, portable and inexpensive system for psychophysiological assessment through variations in face skin temperature. A low-cost infrared sensor is used for its measurement through thermal imaging.

Thermodroid has an Android-based interface for controlling, acquiring, storing and analyzing data, communicating via Bluetooth. It can be con-nected to a Kafka message broker for writing and storing the received data and, as proof of concept of ”stream processing”, calculating the maximum and minimum of the thermal values of each captured ther-mal image that it receives.

Thermodroid was tested in a psychophysiology case study addressing the impact that tobacco health warning images in cigarette packaging have on emotions. Thermodroid was able to capture the thermal im-ages, extracting the thermal data of the skin of the face, as well as the physiological signals necessary for the emotional detection, for its later comparison and analysis. Although the observed results suggest that we were able to capture credible reactions to the presented images, such as a discrimination between arousal levels and a correlation with galvanic skin response, a more in-depth analysis should be performed to validate them.

The obtained outcomes point to the feasibility of the use of contactless and low cost thermal imaging in the detection of emotions and do not dismiss the usefulness of the system for supporting a more complex recognition of emotional scenarios.

(8)

(9)

Palavras-chave Imagem t érmica infravermelha de baixo custo, avaliaç ão psico-fisiol ógica, ”Internet of things”, Bluetooth, Android, Kafka message bro-ker

Resumo Atualmente, o reconhecimento de emoç ões tem assumido um papel importante no dom´ınio da interaç ão humano computador. No entanto, os aparelhos utilizados s ão, por norma, invasivos e limitam a liberdade de movimentos do indiv´ıduo.

Nesta dissertaç ão prop õe-se o Thermodroid, um sistema n ão intrusivo, sem contacto, port átil e de baixo custo para avaliaç ão psico-fisiol ógica atrav és das variaç ões de temperatura da pele da face. Recorre-se a um sensor de infravermelho de baixo custo para a sua mediç ão atrav és de imagens t érmicas.

O Thermodroid possui uma interface baseada em Android para con-trolo, aquisiç ão, armazenamento e an álise dos dados, comunicando via Bluetooth. Pode ser ligado a um ”message broker” Kafka para es-crita e armazenamento dos dados recebidos e, como prova de conceito de ”stream processing”, c álculo dos m áximos e m´ınimos dos valores t érmicos de cada imagem t érmica capturada que recebe.

O Thermodroid foi testado num caso de estudo de psicofisiologia abordando o impacto que as imagens de advert ência de sa úde pre-sentes nos maços de tabaco tem sobre as emoç ões. O Thermodroid mostrou-se capaz de capturar as imagens t érmicas, extraindo os dados t érmicos da pele da face, bem como os sinais fisiol ógicos necess ários à detecç ão emocional, para a sua posterior comparaç ão e an álise. Embora os resultados observados sugiram que fomos capazes de capturar reaç ões cred´ıveis às imagens apresentadas, tal como uma discriminaç ão entre os n´ıveis de ”arousal” e uma correlaç ão com a resposta galv ânica da pele, uma an álise mais aprofundada deve ser realizada para os validar.

Os resultados obtidos apontam para a viabilidade da utilizaç ão de imagiologia t érmica sem contacto e de baixo custo na detecç ão de emoç ões, e n ão descartam a utilidade do sistema para apoiar um re-conhecimento de cen ários emocionais mais complexos.

(10)

(11)

List of Figures

2.1 Two-dimensional Valence-Arousal model proposed by Russel [1] . . . 7

2.2 Autonomic Nervous System [2] . . . 9

2.3 QRS complex [3] . . . 10

2.4 Infrared and adjacent spectral regions and expanded view of so-called ther-mal IR [4] . . . 13

2.5 Classification of IR thermography-related factors in humans [5] . . . 14

2.6 ”Emotional sweating and sudomotor response. The delivery of emotional pressure or stress stimulation (b) changes the rest of the (a) temperature distribution.The spotted dark signature is associated with the activity of the sweating glands.”[6] . . . 15

2.7 Pictures of thermography to detect breast cancer . . . 16

3.1 The current Thermodroid component diagram. . . 26

3.2 The Thermodroid physiological signal capture system. . . 27

3.3 An e-Health sensor platform with the various sensor ports. . . 28

3.4 A Raspberry Pi 3B. General Purpose Input/Output (GPIO) pins in first plan. . 28

3.5 First version of the thermal sensor . . . 29

3.6 The FLIR Lepton thermal camera core assembled in its breakout board. . . . 30

3.7 Application interface - An example of the device scan user interface. . . 35

3.8 The various colour palettes implemented. The facial landmark detection li-brary identified the nose. . . 36

3.9 Interface for the first version of the thermal sensor . . . 37

3.10 Interface for the second version of the thermal sensor . . . 37

3.11 Example of an event data stored in the topic ”eHealthLeptonFinal”, with the processed maximum and minimum temperatures for that event. . . 39

4.1 Security scenario - Fire detection at 4 meters . . . 41

4.2 Security scenario - Intrusion Detection . . . 42

4.3 Electrode terminal placement. The GSR ones on the left hand. . . 44

4.4 Normalized Lepton output averages calculated for each image category. . . . 47

4.5 Normalized Lepton output averages calculated for each valence level. . . 48

(14)

4.7 Combinations of valence-arousal levels. The arousal has a larger contribution

to thermal variations than the valence. . . 50

4.8 Normalized Lepton output as a function of the arousal and respective linear regression. . . 51

4.9 Normalized Lepton output as a function of normalized GSR. Only stimuli that had a response from both signals were considered. . . 51

4.10 Measurements of a typical physiological response to an unpleasant stimulus. The amplitude and latency are also shown. . . 52

4.11 Captured thermal images of breathing events. A different pattern for each state can be seen below the nose. . . 53

4.12 Large signal variations triggered by a stressful event. . . 54

A.1 Informed consent signed by the experiment participants. . . 66

B.1 Participant characterization questionnaire. . . 67

B.2 Image evaluation instructions. . . 67

B.3 Image evaluation after being displayed. . . 68

C.1 Typical thermal signal for a participant at rest. . . 69

C.2 Collected measurements from participant 03. . . 70

(15)

List of Figures v

C.25 Collected measurements from participant 32. . . 81 D.1 Arousal classification levels post hoc test. . . 83 D.2 Valence and arousal averages calculated for each image and image category. 84

(16)

(17)

List of Tables

2.1 Relationship between emotional components and their function according to

Scherer . . . 6

2.2 Review of previous research in emotion recognition using IR thermal imaging (legend: (M)men (W)women) . . . 17

2.3 Comparing Message Brokers solutions . . . 21

3.1 FLIR Lepton and its main specifications [7, 8, 9] . . . 30

3.2 Asus Nexus 7 Tablet relevant specifications . . . 31

3.3 The e-Health Bluetooth Low Energy (BLE) identifications . . . 33

4.1 Normalized Lepton output averages and standard deviation calculated for each image category. . . 47

4.2 Normalized Lepton output averages and standard deviation calculated for each valence level. . . 48

4.3 Normalized Lepton output averages and standard deviation calculated for each arousal level. . . 49

(18)

(19)

Acronyms

AMQP Advanced Message Queuing Pro-tocol

ANOVA Analysis of Variance

ANS Autonomic Nervous System BLE Bluetooth Low Energy BVP Blood Volume Pressure CSV Comma Separated Values ECG Electrocardiogram

EEG Electroencephalogram EMG Electromyogram

GPIO General Purpose Input/Output GSR Galvanic Skin Response HAT Hardware Attached on Top HR Heart Rate

HRV Heart Rate Variability

HTTP Hyper Text Transfer Protocol I2C Inter-Integrated Circuit JMS Java Message Service LWIR Long-Wavelength Infrared MOM Message Oriented Middleware MQTT Message Queuing Telemetry

Transport

NITZ Network Identity and Time Zone

NTP Network Time Protocol

PNS Parasympathetic Nervous Sys-tem

PPG Photoplethysmogram

PTSD Post Traumatic Stress Disorder RFCOMM Radio Frequency Communication ROI Regions of Interest

RR Respiratory Rate

SCR Skin Conductance Response SKT Skin Temperature

SNS Sympathetic Servous System SPI Serial Peripheral Interface

STOMP Simple/Streaming Text Oriented Messaging Protocol

TCP Transmission Control Protocol TLS Transport Layer Security

UART Universal Asynchronous Receiver Transmitter

UUID Universally Unique Identifier WLAN Wireless Local Area Network XML eXtensible Markup Language XMPP eXtensible Messaging and

(20)

(21)

Chapter 1

Introduction

1.1 Context and Motivation

Blushing as a sign of embarrassment or shame, and becoming pale in one’s face from shock or fear has been for a long time a indication that our face can convey some emo-tions. In the last couple of decades, the topic of recognition of emotions has assumed an increasingly relevant position in the domains of security, robotics and particularly in human computer interaction [10, 11].

Rosalind Picard stated that ”if we want computers to be genuinely intelligent and to inter-act naturally with us, we must give computers the ability to recognize, understand, even to have and express emotions”[10], on describing the first researches on the use of emotional states in human computer interaction. Besides being one of the ways humans use to com-municate by providing a context [12], an individual’s emotional state has a lot of influence on decision making, learning and reasoning capacity [10, 13, 14], accentuating the interest in taking into account the emotions in the interactions with humans and making its recognition and identification pertinent in the domains described previously.

Of the various ways of recognizing the emotional state of individuals, we have, for exam-ple, the analysis of speech, text, levels of voice, way of moving, gestures, facial expressions, physiological signals [15]. This work focuses on the latter, with emphasis on thermal imag-ing.

Various physiological signals, including galvanic skin response, respiratory rate, elec-trocardiogram, muscle activity and skin temperature, are considered when recognizing and identifying emotional states [16]. However, direct contact with the body is required to acquire most of the physiological signals, and the measurement of some of them is uncomfortable and even intrusive. These facts can cause unrest, anxiety or distraction and then provoke an influence on the measurements. This results in the need to use physiological sensors without contact with the individual and to carry out the measurements on an remote basis, while on the go.

(22)

The advent and democratization of mobile devices and contactless infrared tempera-ture sensors, such as the FLIR One and the Caterpillar S61, will allow for the captempera-ture and processing of the physiological signals on the go, filling in the described flaws in the physio-logical evaluation.

This dissertation will try to assert the usefulness of low cost thermal cameras coupled to mobiles as a possible and viable solution to identify physiological changes, namely those associated with emotions (e.g. blushing).

1.2 Objectives

The main objective is to devise and deploy a non-intrusive, non-contact, portable, low cost system, using a thermal camera, for assessment of temperature variations of the skin of the face and related physiological signals and to test it feasibility under a psychophysiology scenario to assert its potential to be used in emotional evaluation. This is divided in:

• integrating thermal and physiology sensors in a IT solution. • providing basic UI for monitoring and review acquisition.

• allowing extensions to other IT systems, namely for online and offline processing. • testing the feasibility under a realist psychophysiology scenario i.e. assert if it is

pos-sible to acquire and provide information consistent with the case study.

Thus, in order to perform the analysis and approve the solution found, a large amount of data must be collected, if possible in situations outside the laboratory.

The interoperability, maintenance and diversity of scenarios in which the platform can be used impose a generic architecture, using industry standards and off the shelf components. In order to perform an off-line analysis, the captured signals, including Galvanic Skin Re-sponse (GSR), at least, must be stored and processed in order to be reviewed. This recorded data must also be stored online, in a storage system that can be accessed ex-ternally for other studies.

Taking into account the possible outside-the-laboratory environment, the chosen tech-nologies must have a low energy consumption. There are also some restrictions on the choice of devices and sensors to capture the physiological signals. One of which is that the individuals we want to monitor need some freedom of movement.

Another factor that has to be considered is that the sensors have to be non-invasive, and with the least contact points possible, so as not to cause discomfort and not modify the physiological signals that we want to measure.

(23)

Contributions 3

Overall, the cost of the platform components should also be considered, due to the low budget available for this project.

1.3 Contributions

A system for management and visualization of thermal data from the camera sensor including:

• A low level component for interfacing with sensors – thermal data from the camera sensor and the

– galvanic skin response and heart rate from the e-Health Sensor platform.

– Bluetooth Low Energy and classic Bluetooth services for communication between the physiological sensor platform and a local client.

• Android application for the management and visualization of the transmitted data. • Integration with Kafka based messaging bus.

– integration with the Kafka broker through the Producer API. Some features are extracted from the thermal data using the Kafka platform streams processing. – persistence for the gathered data, locally in the application and externally on the

Kafka platform.

• A feasible system for psychophysiology assessment using contactless and low cost thermal imaging as well as the physiological signals necessary to detect emotions.

– psychophysiology study case study addressing the impact of tobacco health warn-ing images in cigarette packagwarn-ing.

1.4 Dissertation Structure

This dissertation, excluding this chapter in which a brief introduction is given, presenting the motivations and indicating its objectives, is divided in the following chapters:

Chapter 2 presents an short review of the background information about the themes dis-cussed in the dissertation, namely emotions, the various physiological signals and thermal imaging. In the topic of emotions we also present ways to recognize them, focusing mostly on physiological signals, for which several recent studies are indicated.

In Chapter 3, a system is proposed and explained, detailing its architecture and choices made. Both the systems’ software components and devices are described. In this chapter

(24)

a detailed description on the communication interfaces and protocol is also provided as well as how the gathered data is stored.

Chapter 4 presents the system evaluation step, describing the experimental scenario, how the experience is set up and the methods used to gather the data generated by the phys-iological signals. After presenting the obtained results, their analysis and a discussion are made.

The final outcomes and conclusions of this dissertation are summed up in Chapter 5. Some suggestions for improvements and future work are also given.

(25)

Chapter 2

Background and related researches

Lately the research about emotions and their detection via computing has seen a great investment in the technological area [16, 17].

Several works have been developed in many areas/fields, with the use of various devices, and the purpose of increasingly using non-intrusive and non-invasive resources, so that their interference in the human body and in the conscience of the individual tested is lower.

Thus, the emotions, their recognition and the best way to do it depending on the objective, target subjects, and the measured signals/devices used, are factors and concepts to deepen in this dissertation.

2.1 Notions about emotions

2.1.1 Definition

Emotion is ”a complex experience of consciousness, bodily sensation, and behavior that reflects the personal significance of a thing, an event, or a state of affairs” [18]. So, emotion is present in the people’s daily lives and influences every reaction that a person has to a significant occurrence, good or bad. Scherer [19] has recently proposed a definition from a more functional point of view as we can see in table 2.1.

Thus, an emotion is a process that comprises five components, cognitive, subjective feeling, motivational, motor expression and neurophysiological. We will focus the emotional recognition mostly on this last component.

(26)

Emotion component Emotion function

Cognitive Evaluation of objects and events

Subjective feeling Monitoring of internal state and organism-environment interaction

Motivational Preparation and direction of action, behavior tendency Motor expression Communication, facial and

vocal, of reaction and behav-ioral intention

Neurophysiological System regulation, bodily symptoms

Table 2.1: Relationship between emotional components and their function according to Scherer

2.1.2 Classification of emotions

Has we have seen, there is no single approach to define or describe emotions. On how to classify them the same problem arises. In reality, there are several theories and models developed for the areas in which they are applied or for the purpose of the selected research. However, the two most used models are the ones proposed by the psychologist Ekman, who describes a ”discrete emotional model” [20] and the ”two-dimensional Valence-Arousal” model proposed by Russell [21].

The discrete emotional model, based on universal emotions that are present in all cul-tures, defines the six core emotions: happiness, sadness, surprise, anger, disgust, and fear. These core emotions serve as a base to construct all the other secondary emotions.

Russell defines the classification of emotions according to the definition of ”weight” in two components: Valence and Arousal. This definition in two dimensions allows for the repre-sentation of the emotions in a cartesian coordinate system as shown in figure 2.1.

This classification characterizes the emotions according to the valence variation (abscissa axis), which represents pleasant / unpleasant and ranges from negative to positive, and arousal (ordinate axis), which represents activation (calm / excited) and ranges from low to high [22, 16].

Given that when the emotions are brought forth there is an autonomic nervous system activation (see chapter 2.2), psychophysiology verifies that there is an association between physiological signals and valence/arousal [23].

(27)

Notions about emotions 7

Figure 2.1: Two-dimensional Valence-Arousal model proposed by Russel [1]

2.1.3 Why to recognize emotions

As already mentioned above in chapter 1.1, the recognition of emotions is a very well researched theme and, it is now known that it is possible to recognize emotions through facial expressions, body posture, gestures, speech, tone of voice, among others [16]. However, all those listed above are totally or partially controllable by the individuals, which often makes it possible to deceive the mechanism / process of recognition of emotions, for example through trained and forged facial expressions, or having a control of the gestures made [16, 17].

However there are cues that are not consciously controllable namely physiological sig-nals like Skin Temperature (SKT), Heart Rate (HR) or at least difficult to modulate or sup-press [17].

There are other areas/problems in which the recognition by physiological signals is very useful. Since inferring emotions through the descriptions of people is very subjective, and there are even individuals who can not move and do not have the capacity to express them-selves, such as disabled people, the elderly, babies, and even people with autism spectrum disorders, the means and studies that recognize their emotions are very relevant, so that we can better understand what they feel and how they react in the different situations they experience [24, 25]. Thus, these cases are some examples where the recognition of emo-tional states via physiological signals is very useful.

Also, in the thematic of security it may be relevant to know someone’s state of mind, for example, in biometric identification. No doubt that with the biometric data it is possible

(28)

to identify people with reliability, however, if they are under threat it may not be enough to identify them and let them access a facility, for example. Thus, the remote recognition and identification of emotions, such as distress or fear, can help address security issues [26].

2.2 Physiological signals

The Autonomic Nervous System (ANS) is a set of nerve cells and nerves that act pri-marily on smooth muscle activity (but also on cardiac and glandular cells) in order to keep the organism in balance or to readjust the internal environment to react to changes in the internal or external circumstances, (e.g. heat, cold, hunger, thirst, anxiety, suffocation) in an attempt to keep it constant [27, 28]. This regulation of the internal balance of the organism is denominated homeostasis [29].

The physiological reactions associated with the ANS relate to functions of the organism such as heart beat, salivation, sweating, breathing, among others [28].

This way, the ANS is responsible for involuntary physiological responses, that is, causes autonomous actions / reactions that the individual can not control. This being the main dis-tinction from the somatic nervous system, which for the most part, is under voluntary control and accessible to the individual’s consciousness [28].

The ANS is divided in Sympathetic Servous System (SNS) and Parasympathetic Ner-vous System (PNS) (Figure 2.2).

The SNS is responsible for the reactions under distress and fear, while the PNS reacts under conditions of stability and calm. Hence it is said that the SNS stimulates the ”fight or flight” response and that the PNS is responsible for the activities of ”rest and digest” [28].

The PNS acts to restore the state of relaxation and stability of the organism, in the hu-man body it is located in the sacral area of the spinal cord and in several cranial nerves. Some of the reactions that the system controls are: decreased heart rate, decreased blood pressure, contraction of the airways, stimulation of digestion, secretions at the intestinal and salivary glands, contraction of the pupil, among others [27, 28].

The SNS is located in the thoracic and lumbar spinal cord, and is responsible for the re-actions to stressful and emergency situations, which are depicted, for example, by dilation of the pupils, increase of sweat production, increase in blood pressure and beat heart rate, relaxation of the airways, increase in body temperature, among others [28].

(29)

Physiological signals 9

Figure 2.2: Autonomic Nervous System [2]

Stress and fear are good examples that show that by monitoring through several physio-logical signals one may find relevant patterns and cues on ANS changes.

Next we show the physiological signals involved in the detection of emotions used by the different approaches.

• GSR Galvanic Skin Response [26, 29, 30, 31, 32, 33, 34, 35] • HR/ECG Heart Rate/Electrocardiogram [26, 31, 32, 33, 34, 35] • EMG Electromyogram [32, 34, 35] • SKT Skin Temperature [30, 32] • RR Respiratory Rate [34, 35] • PPG Photoplethysmogram [30] • EEG Electroencephalogram [33] • PD Pupil Diameter [30]

We found that in the majority of studies, multiple physiological signals were measured and that of all, the GSR and Electrocardiogram (ECG) were the most used.

(30)

Other physiological signals are often regarded as having too intrusive measurement meth-ods to be of practical use, so we will not discuss them here. These and other signals are widely discussed in [28]. We will focus in the most used ones.

Galvanic skin response

The GSR signal, known also as electrodermal activity, is an indicator of the skin con-ductance. When passing a small electric current between two electrodes, the conductance between them can be measured [36]. In certain circumstances, the skin glands produce ionic sweat, which alters the electrical resistance and consequently conductivity [26]. As the sweat glands are controlled by the SNS, the conductance of the skin is a good indicator of the activation of said system [37]. Conductance of the skin increases linearly with the level of general agitation, arousal, of a person. According to Bakker et al. [29] we must bear in mind that this increase can be caused by anticipation of the event causing distress. It is also important to mention that physical activity causes an increase in sweat secretion.

Electrocardiography

Figure 2.3: QRS complex [3]

The ECG measures the variations in electrical current produced during the contractions of the heart. The HR measures the heart rate, in beats per minute. According to [28], in-creased heart rate is the result of a stressful event. Of note in the same book, in chapter 13, the observation that, in advance, the heart rate decreases in the brief seconds before the event.

The analysis of an ECG involves the analysis and detection of the so-called QRS complex, which exhibits a graphic representation such as that of figure 2.3. In it we can see a more ac-centuated/main peak of the signal (R), preceded (Q) and succeeded (S) by the other waves.

(31)

Physiological signals 11

This represents the ventricular depolarization of the heart and the contraction of the heart muscles. Concretely, the Q wave represents the septal depolarization, the R represents the left ventricular depolarization and the S wave the high lateral wall depolarization.

The ordinary QRS duration varies from 0.06 to 0.1 seconds. Thus, knowing how to interpret an ECG and through the analysis of the QRS complex, it is possible to extract the informa-tion of the HR [35] and of the inter-beat intervals, therefore it is possible to determine the Heart Rate Variability (HRV) [35] and also the Respiratory Rate (RR) [38].

Heart rate and heart rate variability

HR represents the number of heart beats per unit of time. It is a signal that is easily obtained in any part of the human body.

As already mentioned, when the ANS responds to a stimuli, it can accelerate or slow down the heart beats. In a situation of stress, specifically, the ANS response will cause an accel-eration of the beat. It is possible to measure the emotional valence through the HR [39]. The HRV can be obtained with the HR and through the inter-beat intervals, since HRV is the oscillation between two consecutive heart beats. In a stress scenario the HRV will be raised. These signals can be inferred from the ECG.

Respiratory rate

The RR measures the person’s breathing speed, if it accelerates or slows down. The breathing varies according to the emotional state of the person, in a situation of stress it increases [39].

This signal can be inferred from the ECG, as the HRV is connected with breathing. Thus, the RR can be estimated by the ECG, since it allows us to obtain the HRV power spectrum. When the RR changes, these are reflected in the HRV [38].

Electromyography

The Electromyogram (EMG) signal measures muscle activity by detecting the surface voltage that occurs when the muscle fibers, for example the arm or face, are contracted by electrical impulses [12]. The average muscle activity correlates with negative valence emotions, for example stress or fear [39].

Skin temperature

As more or less blood is allowed to flow from the core of the human body, which has a relatively stable temperature, to the local that we are measuring, also the temperature of that area will tend to approach the core or ambient temperature, respectively. As such, changes in skin temperature are related to the size of the local blood vessel diameter. This vascular caliber is controlled by smooth muscle tone induced by increased or decreased activity of

(32)

the SNS. A. Merla et al. [40] states that the skin temperature depends not only on the blood vessel diameter, but also local tissue metabolism and the sudomotor response, all controlled by the SNS. Also states a negative stimulus leads to a decrease in skin temperature, while a positive one cause an increase in skin temperature. Kataoka et al. [41] asserts that there is a high correlation among emotions and skin temperatures on nose and forehead. Puri et al. [42] also concluded the same, so attention will be given to these regions.

Photoplethysmography

The Photoplethysmogram (PPG) signal, also called blood volume pressure, is generated by a device known as pulse oximeter which measures the reflection of infrared light illumi-nating on a tissue, which is an indicator of blood flow [43]. Since each heart beat forces blood through the vessels, the PPG can also be used to calculate it. As previously stated, an increase in negative valence emotions causes an increase in heart rate.

Electroencephalography

The brain cells, neurons, communicate with each other through tiny electrical signals. An Electroencephalogram (EEG) measures this activity. The machine captures the electrical signals through electrodes placed on the scalp and converts them into waves that can be observed. The different patterns, frequencies and amplitudes of the waves allow to make a clinical diagnosis or, in our case, to associate mental activity with the emotional state of an individual [44].

Pupil diameter

The diameter of the pupil is sensitive to the amount of light that falls on the eye, dilat-ing and contractdilat-ing through the control of the autonomic nervous system exerted on the iris muscle. Thus, if we discount the variation caused by light, the increase in pupil diameter is an indicator of arousal [45].

Some additional signals that may be interesting to capture, namely to reduce noise and wrong readings, include context information such as external temperature and moisture as well as the individual’s movements [29].

2.3 Infrared thermal imaging

The emission of electromagnetic radiation is a characteristic of several objects. This ra-diation can be described by waves.

It is known that there is a direct correlation between the wavelength of the emitted radiation and the temperature of the object. Thus, this radiation is often known as thermal radiation [46, 47, 4].

(33)

Infrared thermal imaging 13

The infrared spectrum can be categorized into active infrared band (Short-Wave Infrared (SWIR) / Near-IR (NIR)) and in passive infrared band (composed of Mid-Wave Infrared (MWIR) and Long-Wavelength Infrared (LWIR)). All of these designations are based on the wavelength, and as the figure 2.4 shows, we have:

• NIR/SWIR characterized by the wavelength between 0.8 and 3 micrometers. • MWIR whose wavelength varies between 3 and 8 micrometers.

• LWIR with wavelengths of 8 to 14 micrometers.

Figure 2.4: Infrared and adjacent spectral regions and expanded view of so-called thermal IR [4]

Research on the human body has found that passive band IR, specifically the LWIR, is the best choice for measuring infrared radiated by the human body, since that humans emit radiation at wavelengths around 10 micrometers [46].

Although the passive band consists of MWIR and LWIR, there are significant differences between them. The MWIR has the emissive and reflective properties, while the LWIR con-sists mainly in emitted radiation [47].

There are many advantages to using the passive infrared band. Images in this band do not require lighting (a light source) in the environment for image acquisition, while the active band requires external illumination [47]. Regarding the study of the human body, anatomical features such as venous patterns, blood perfusion, may not be observable in the active band [46].

When using IR thermal images, it is necessary to take into account many external factors that can influence not only the evaluation but also the interpretation of the images. Thus, it becomes very important to be aware of the factors that exist, since it is impossible to control them all. In figure 2.5 we get an overview of many of the factors.

(34)

Figure 2.5: Classification of IR thermography-related factors in humans [5]

Through the analysis of the images we can see that Fern ´andez-Cuevas et. al categorizes the factors as follows [5]:

• Environmental factors: Those that are related to the place where the evaluation is performed, such as room size, relative humidity, pressure and ambient temperature. • Individual factors: Those that are related to the subject being assessed and his/her

personal characteristics that could influence skin temperature, such as, hair density, physical activity, medical treatment, intake factors.

• Technical factors: Factors that are linked to the equipment used during the thermal infrared evaluation, such as regions of interest selection, camera features, protocol.

2.3.1 Effect of physiological signals on thermal imaging

Thermal signatures of several physiological signals have already been identified, and through bioheat transfer models, it has been demonstrated that it is possible to calculate the cardiac pulse, the breathing rate, the cutaneous blood perfusion rate, and the sudomotor response at a distance [6].

As pointed out in section 2.2, the GSR and the ECG/HR are of the most used phys-iological signals, and since they have special relevance in this dissertation, we will detail

(35)

the relationship between the cardiac pulse and the sudometer response with IR thermal imaging.

Cardiac Pulse Regarding the cardiac pulse, when the contraction of the heart during the ventricular systole occurs, a pressure wave is generated. This is propagated by the arterial tree, allowing the analysis of the arterial pulse, which reflects the activity of the heart, thus allowing the measurement of cardiac interbeat intervals, heart rate and its variability [6]. Studies as [48, 49, 50], confirmed that it is possible to obtain this data through IR thermal imaging. In [48] and [50] it was studied the hypothesis that the temperature modulation due to pulsating blood flow produces the strongest variation in the temperature signal of a super-ficial vessel achieving results with an accuracy between 88.52% e 95.3%.

[49] proposed an automatic method to determine arterial pulse waveforms through the use of thermal imaging, obtaining a validation result in which 8 participants showed perfect match-ing with pulse-oximeter data.

Sudomotor Response One of the most used measures of physiological signals of ANS activation is the electrodermal response [6]. The Skin Conductance Response (SCR) and related measures, such as GSR, can be easily visualized by a facial thermal image by the presence of cold spots in the thermal distribution of the face (figure 2.6).

Figure 2.6: ”Emotional sweating and sudomotor response. The delivery of emotional pres-sure or stress stimulation (b) changes the rest of the (a) temperature distribution.The spotted dark signature is associated with the activity of the sweating glands.”[6]

Several studies suggest that the identification of the active eccrine sweat glands (the ma-jor sweat glands of the human body) through thermal imaging may be useful as a method of psychophysiological measurement of sudomotor activity and may even be the solution when one can not resort to a contact method [6].

Di Giacinto et al [51] studied the conditioning of fear in patients with Post Traumatic Stress Disorder (PTSD) using thermal IR imaging and GSR. The authors concluded that they de-tected responses of the SNS associated with the disease, and that the analysis of the facial thermal response during the experiment performed as well as the GSR.

(36)

2.3.2 Applications of Termography

In fact, thermography is a method that can be found in many different areas. There are many examples in health, industry, security, sports, building and structures, among many others. [4].

In medicine there has been recurrent use, often in an attempt to resort to non-intrusive means for the detection of diseases [4, 52], pain and its monitoring [4], among others. For example, there are some types of breast cancer that are detectable through the use of this technology [4, 52]. In the case of the study of Figure 2.7 it was possible to detect the tumor, since the area around it, due to the greater vascularization, presents an higher temperature.

It is also possible to use infrared thermal imaging in the detection of circulation problems that diabetics have in the periphery of the body (fingers), dermatology, dentistry, fever screening, diagnosis of dry eye syndrome and other ocular diseases [52].

(a) ”Image shows a signifi-cant amount of heat and vas-cularity (angiogenesis) in the right breast, especially over the lump in the upper outer quadrant.”

(b) ”In this image, the left breast is shown to be cool with a normal limited vascular pat-tern.”

(c) ”This image shows the right breast with the area of the lump directly facing the in-frared detector, showing the increased heat and vascular-ity.”

Figure 2.7: Pictures of thermography to detect breast cancer

In the field of security, it is well known for being used in intrusion detection [4] but can also be used as a complement to detecting someone’s emotional state when trying to iden-tify via biometric data reading [26].

Regarding the industry sphere, there is a focus in the detection of gases, but it is also widely used for predictive maintenance e quality control [4, 53], and from the areas of high and low voltage, polymer molding, among many others [4].

In terms of buildings and construction, an example of the use of infrared thermal imaging is when it is intended to prevent the unwanted leakage of water or energy or locate and determine the cause of the loss. Thus, many times, infrared measurements in the building attempt to find the atypical temperature on the surface of the building, so that it is possible to identify the cause, which could be, for example, problems with insulation of windows or doors, condensation of humidity, among others [53].

(37)

purposes and uses, thus showing the potential and versatility of applications and utility of this technology.

2.3.3 Applications in emotion detection

Although relatively sensitive to changes in the external environment (e.g. fog, wind, sweat), infrared thermography seems an effective method of remotely recognize an individ-ual’s emotional state.

A number of studies have been carried out in this scope (see Table 2.2), many of them showing that it is possible to find correlations between the temperature of the skin detected by thermal imaging and the emotions.

Table 2.2: Review of previous research in emotion recognition using IR thermal imaging (legend: (M)men (W)women)

Ref. IR Device Body area Emotion Stimuli Subjects Results [54] FLIR A310; 320x240px Nose, forehead, maxillary, cheeks Anger, disgust, fear, joy, sadness Video 44, (8W,36M) average age 26.5 Accuracy: W:89.5% M:90.3% [55] Avio tvs8000; 160x120px; Thermistors: Nihon Koden we5000 Fingers; Thermal sensors on nose, temporalis, finger Fear Video 18, (9W,9M) Thermography results were similar to those of the thermistors and corrobo-rated by the decrease in HR and RR [56] FLIR Ther-moVision A320G; 320x240px Forehead, nose, eye region, mouth, cheeks Valence and arousal

Images Study 1a: 80, (40W, 40M) Study 1b: 40, (20W, 20M) Thermal facial changes cor-relate more with arousal than valence. Valence effect only under low arousal [57] FLIR SC655; 640x480px Face, throat Stress Poker game 2 Facial SKT variations, but card predic-tion impossi-ble. Outliers, which need investigation Continued on next page

(38)

[58] FLIR SC7600; 640x480px Forehead, philtrum Stress Mental arithmetic; memory test; public speaking 41, age range 20-65 Accuracy: 90% [59] Testo (881-1); 160x120px Face, forehead, periorbital, nose, mouth, cheeks Arousal Social contact; Tactile contact Study 1: 16W, age range: 19-24; Study 2: 23W, age range: 18-25 Evidence of immediate physiological reactions to social contact [40] AEG 256 PtSi; 256x256px

Face Arousal Emotional pressure and other people’ judgment; pain and fear to feel pain; sexual excitement 10 Thermography is a valid mea-sure of the sympathetic activity and correlated to the other physiological signals [60] FLIR SC640; 640x480px Supra and peri-orbital, nasal Valence and arousal Visual stimuli 12, (9W,3M); average age 24 Accuracy: 80% [61] Optris PI 640; 640x480px Nasal, perinasal Stress Humanoid Robot interaction 16, (2W,14M); average age 27.5 Correlations between: Dis-tance and SKT; Dis-tance/Gaze and SKT [62] Micro-bolometer camera; 320x240px Face(major muscle regions) Happy, sad Images, Video 19, (7W,12M); average age 20.25 Temperature along the fa-cial muscles is influenced by arousal levels [63] FLIR Ther-moVision A40; 320x240px Face, neck, shoulders Affective state (positive, negative, neutral) Video 14, (3W,11M); age range: 25-45 Accuracy: Af-fective vs Neu-tral 90%;can-not distinguish between posi-tive and nega-tive states Continued on next page

(39)

Message Oriented Middleware 19 [64] FLIR A325sc; 320x240px Forehead, eyes, nose, mouth Neutral, moti-vated, over-stressed Robotic therapy inte-grated with games 8 performance 92.6% [42] Indigo Phoenix Mid-Wave; 320x256px

Forehead Stress Stroop color word conflict test 12 (5W,7M) Correlation between ther-mography and energy expenditure is over 0.9

In most studies presented, the thermal imaging is validated by the physiological signals whose changes are known to be directly related to the ANS namely HR, RR and Blood Vol-ume Pressure (BVP) (see section 2.3.1).

However some refer that resorting only to infrared thermal imaging may have shortcomings, namely being able to detect arousal but not valence [63, 56]. The consequence is that it may be possible to detect the triggering of an emotion and to prove its existence at the level of impact, but its valence can not be easily distinguished.

Although not resorting to temperature as an absolute method, some studies such as [65] also use the recognition of facial expressions to estimate emotions. The authors conclude that the use of an integrated method (in this case, voice, visible light images and infrared images) for the recognition of the emotional state obtains a better performance than any isolated method.

2.4 Message Oriented Middleware

One major concern on this work was the necessity of connecting several devices and analysis programs namely to relay and process data. From the start the option to use a Message Oriented Middleware (MOM) system was a natural solution. Using a MOM allows the abstraction of the complexity of the technical part of the connections and services, further decoupling the data producers from their clients. That way, the systems will not only be dissociated from each other by the message interface but also by the indirect communication between them.

The MOM systems provide a communication that is distributed on the basis of an asyn-chronous interaction model [66], i.e. the physiological sensor platform data gatherer may be momentarily disconnected from the broker and the external analyzing software that our system will still work.

A MOM is a bus structured platform that deals with sending / receiving messages, of any type of data, providing the necessary APIs, allowing the clients not to worry about han-dling the message transfer [67]. Messages are sent by the MOM to the ”consumers” that

(40)

subscribed to a destination, originated by one or many processes called ”producers” that published those messages to that destination, the MOM behaving as a broker, a middleman between the two.

The message payloads, typically, are not processed by the MOM, only their header, that provides the necessary information to forward them to the respective destinations, queues or topics [68].

Queues keep the messages in reserve until they are delivered to a service that sub-scribes it. If there are multiple ones that subscribe the queue, at maximum a single one is selected to receive the message. This message reservation allows for asynchronous communication, and even temporary unavailability, between those who send or receive the messages, since a direct communication is not necessary. Usually used in a point-to-point model, the messages are delivered in the order they were sent [68].

Contrary to a queue, topics deliver the messages to every subscribed consumer when it receives them, without guarantee that they are delivered in the order in which they were sent and even that a consumer received it. Ordinarily, when topic subscribers are off-line they lose all messages that are sent, unless the subscription was a durable one, that is, one that, like a queue, retains the messages. Topics are usually used in the Publish/Subscribe model [68].

2.4.1 Messaging Protocols

In regard to the most used protocols when referring to MOM implementations, these are: AMQP Advanced Message Queuing Protocol (AMQP) is an open standard inter-operable messaging protocol for MOM. It is defined by the features of message orientation, queuing, point-to-point and publish-and-subscribe connection types, reliability and se-curity [69]. Mostly used to connect servers to each other, it is defined in layers: a type system; an asynchronous protocol for the transfer of messages; a standard but extensible message format and messaging capabilities [69].

MQTT Message Queuing Telemetry Transport (MQTT) is a very simple and low resource usage, publish/subscribe protocol used for machine-to-machine applications, that is, for transmitting device data to servers. As security comes with a processor and com-munication overhead, only a small number of security standards are available. It is up to the MQTT implementations to supplement them with standards like Transport Layer Security (TLS) if required [69, 68].

XMPP eXtensible Messaging and Presence Protocol (XMPP) it is a eXtensible Markup Lan-guage (XML)-based protocol, originally designed for chat systems, but extended to include publish-subscribe messaging. A device-to-server protocol it is mostly used to connect people to messaging servers. As it is extensible it is also used as a publish-subscribe system [69, 68].

(41)

Message Oriented Middleware 21

STOMP Simple/Streaming Text Oriented Messaging Protocol (STOMP) an easy to imple-ment and simple protocol, similar to Hyper Text Transfer Protocol (HTTP) with many client implementations in several languages [68].

Kafka It is a non standardized protocol, which communicates using binary data over Trans-mission Control Protocol (TCP), and which allows high scalability and great fault toler-ance as it runs on distributed systems. A destination, topic in this case, can be divided into several partitions of different brokers, succeeding in handling a large amount of data and allowing parallelization [70, 71]. Only allows the publish/subscribe pattern [70, 68].

2.4.2 Message Broker Solutions

There are several open source Message Brokers available, some of the most notable are Mosquitto, RabbitMQ and ActiveMQ and Kafka. The differences between them reside, among others, in the supported protocols, the provided clients developed for the various programming languages, the scalability support and the main goal for their development. Thus, we will give an overview of the main characteristics of these solutions.

Message Brokers Overview

In order to better understand the diversity of the available broker solutions, a comparative summary of the Messages Brokers referred to above is presented in the table 2.3.

Table 2.3: Comparing Message Brokers solutions

Mosquitto RabbitMQ ActiveMQ Kafka

Protocol MQTT AMQP STOMP AMQP STOMP

MQTT HTTP Kafka Transport Proto-col TCP TCP TCP, SSL, NIO, UDP TCP Highlighted Features persistence, lightweight clustering, high availability, multi protocol, persis-tence persistence, clustering, RESTful API clustering, storage, processing records Security SASL PLAIN;

TLS; ACL SASL PLAIN or challenge -response; TLS; ACL JAAS; SSL/TLS; ACL SASL PLAIN; Kerberos; SSL; ACL Clients lan-guages C Java .NET/C# Earlang Java C Python C++ C# Ruby Perl PHP

Java and many others Messaging pat-terns Publish /Sub-cribe Publish /Sub-cribe; Re-quest/Reply; Point-to-Point Publish /Sub-cribe; Point-to-Point Publish /Sub-cribe

(42)

Mosquitto

Eclipse Mosquitto is an open source message broker that implements the MQTT proto-col and, as such, is lightweight and is suitable for use on all devices from low power single board computers to full servers [72].

As mentioned previously, the MQTT protocol provides a lightweight method of carrying out messaging using a publish/subscribe model, this leads to Mosquitto being suitable for Inter-net of Things messaging like low power sensors or mobile devices such as phones, embed-ded computers or microcontrollers [72].

RabbitMQ

RabbitMQ is written in Erlang and open source. Allows the management of multiprotocol messages, the most relevant being STOMP and AMQP. RabbitMQ has as main design goals reliability, scalability and to quickly handle messages in-memory, if persistence is not a requirement. It has message queues optimized for almost empty queues, the throughput diminished if the number of messages start to grow in the queue [73].

ActiveMQ

ActiveMQ is open source, written in Java, and one of the most widely used message bro-kers, being a feature-complete messaging solution, which can be used to implement many communication and integration patterns [74]. It supports several protocols, such as AMQP, STOMP, MQTT, HTTP. It is fully compatible with the Java Message Service (JMS) and also provides many clients to several programming languages. ActiveMQ offers a number of ad-vanced features such as rich deliver semantic, e.g. virtual queues or composite destination, advanced clustering configuration, such as slave master or broker networks, and message storage that allows for persistence. [74].

The broker networks, and the many topologies imaginable, allow an increased scalability and performance [68]. ActiveMQ has a simple configuration that works for a variety of mes-sage delivery cases, handling the problems and difficulties of messaging that its clients might have [68].

Apache Kafka

Apache Kafka is a solution that differs from the rest since it allows records processing. This streaming platform is normally used in two situations: when one intends to construct real-time streaming applications in which the data streams are processed, or when one wants to build real-time streaming data pipelines that get data between systems with relia-bility [68, 73].

In Kafka, communication between clients and servers is done through the TCP protocol with a agnostic, simple and high performance language [70].

(43)

Message Oriented Middleware 23

Kafka only allows publish and subscribe [70] of streams of records, in a similar way to a message queue or enterprise messaging system, to a topic specific disk based append log [73]. The platform also allows one to store streams of records with fault tolerance and their processing as they arrive.

Kafka is designed in a clustered architecture that will serve as the backbone of the entire product [73]. This architecture allows the system to provide a high availability of its services, since when a cluster machine fails, the resources are redistributed by the remaining ones [73, 70].

All the compared solutions are different from each other, with different features and results in different comparisons depending on what is needed or is tested / implemented.

Given the diversity of the offer and the distinct functionalities, through the purpose for which the need for the implementation of a MOM arises, the chosen solution will have to be weighed. There is no perfect solution or a better one than the others. There are several solutions that are better adapted to some problems than others, hence the choice is case-by-case.

In our case, the stream processing capabilities of the Kafka platform are a great advan-tage to the signal feature extraction, allowing for the potentially different external consumers that would analyze the signal data to have those features already extracted when they re-ceive the message.

(44)

(45)

Chapter 3

Thermodroid architecture and

implementation

In this chapter ThermoDroid is presented, a platform that captures and gathers various physiological signals, with focus on thermal imaging, and allows the classification of an in-dividual’s emotional state. The system is explained and proposed, detailing its architecture, hardware and the choices taken during implementation. In the first section the global ra-tionale behind a similar platform is presented. The architecture of the proposed platform is depicted, detailing both the systems’ software and hardware components. The last section of this chapter addresses the platform implementation. A description of the applications, communication interfaces and protocol is also provided as well as how the gathered data is persisted.

3.1 Thermodroid Architecture

3.1.1 General Architecture

Thermodroid, which in essence resembles a remote monitoring system, consists of mod-ules of: acquisition of signals from sensors (1), communication (2), processing of signals (3), user interface (4) and data storage (5).

• (1) The sensors component, which includes the thermal camera we want to evalu-ate, collects the following physiological signals: electrical potentials produced by the cardiac tissue, ECG, and the resistance of the skin influenced by the state of sweat glands, GSR, for reference and control. To be compared with these signals, we also collect the facial skin temperature.

It consists of three parts. The electrodes and other components of the ECG placed in the arms, those of the GSR in the index and middle fingers and the external tempera-ture sensor, in this case a thermal infrared camera, contactless.

(46)

physiological signals compiled by the sensors component to the systems responsible for processing and analysis. Since the individual, from an outside the lab perspective, potentially needs freedom of movement we will need to communicate wirelessly. The signals of the individual being tested need to be monitored locally, by the Android application, e.g. to verify the connections, the signals themselves or perform a prelim-inary analysis. Bluetooth technology is cheap, uses little power, is fast enough and is democratized, so it is a good choice to perform that task. If working in a laboratory, with plenty of electricity, the collected data is also transferred over Wireless Local Area Network (WLAN), using the Kafka messaging protocol, to an online instance of the Kafka message broker system, allowing future analysis by a external remote user. • (3) The signal processing component consists of elements, not only of hardware (in

this case an Android device) but also software, that collect and pre-process all phys-iological signals received from the sensor component, sending them to the storage component and performing signal filtering and feature extraction.

• (4) The user interface component interacts with the user, controlling the Bluetooth connections to the devices, displaying the data in real time, allowing its assessment. Later on, in order to draw conclusions about the feasibility of the infrared temperature sensor and the platform in the physiological evaluation of emotions, it is possible to navigate the stored data of each individual to visualize it for analysis.

• (5) The data storage component stores the signals, characteristics, and metrics ex-tracted from them, either by taking advantage of the storage features of the Kafka platform, or in case of the Android application, locally, in a Comma Separated Values (CSV) file, to take advantage of the ease of manipulation and movement these files allow, since we do not need to make random updates or queries to the data in them. Figure 3.1 conveys the essence of the system’s features and components.

(47)

Thermodroid Architecture 27

3.1.2 System Hardware

Thermodroid is a tangible system based on several hardware components that we will depict:

• Physiological sensor platform – e-Health Sensor Platform. – Raspberry Pi 3.

– Thermal camera. – Battery.

• Android device. • Data storage server.

Physiological sensor platform

Depicted in figure 3.2 is the physiological sensor platform with its elements, the e-Health Sensor Platform, with the GSR sensor leads and the pulsioximeter, the Raspberry Pi 3 and the battery, as well as the thermal camera in its breakout board which we will describe in more detail in the coming paragraphs.

Figure 3.2: The Thermodroid physiological signal capture system.

e-Health Sensor Platform To capture the physiological signals needed, the Cooking Hacks e-Health Sensor Platform V2.0 was used, as in figure 3.3. It allows the use of ten sensors, such as blood pressure, glucometer, pulsioximeter, EMG, airflow, ECG or GSR. Some sen-sors use the same communication paths and can not work simultaneously, like the EMG and ECG ones, but it was not the case with those we used. At first we were going to use the pulsioximeter to get the HR, but after a few weeks we discovered that it was not compati-ble with the RaspberryPi3 [75], so a new device, a BLE HR monitor for fitness applications,

(48)

was incorporated in the system. The integration was simple, as the device implemented the default BLE Heart Rate Profile [76] and we already had the BLE service working. However the device was not used, as it implies a close contact with the skin above the heart and its attachment to the body was problematic, for hygienic reasons, as it is not disposable. Thus, we employed the ECG and GSR sensors.

Figure 3.3: An e-Health sensor platform with the various sensor ports.

It is compatible with an Arduino, but an Hardware Attached on Top (HAT) adapter can be bought in order to connect it to a RaspberryPi. A C++ Arduino library is provided to read the sensor’s electric signals and extract features from them, e.g. skin resistance and conductance voltage, letting the developer read the sensor data easily. Again, an adapter is necessary to use it in a RaspberryPi, in this case the ArduPi middleware library [77].

Raspberry Pi 3 To power the thermal camera and capture the thermal data, a Raspberry Pi 3 is used. A Raspberry Pi is a series of low cost single board computers, usually running a Linux operating system, developed and sold by the Raspberry Pi Foundation with the goal of improving the knowledge about computer science. There are several products available, but the one that we had at hand was a third generation Raspberry Pi model B, as seen on figure 3.4.

Figure 3.4: A Raspberry Pi 3B. GPIO pins in first plan.

It comes equipped with a 1.2GHz 64-bit quad-core Arm Cortex-A53 CPU, has 1GB RAM, integrated 802.11n WLAN and Bluetooth 4.1+BLE connectivity [78]. A set of 40 GPIO pins provides a way to connect external devices or HAT boards. GPIO pins allow control over

(49)

Thermodroid Architecture 29

alternative functions such as Universal Asynchronous Receiver Transmitter (UART), Inter-Integrated Circuit (I2C) as well as Serial Peripheral Interface (SPI) [79].

It also has a microSD card slot for inserting a microSD card with the chosen operating system and applications, functioning as data storage. It is powered by a 5V micro USB port allowing a mobile phone charger to be used as a power supply. The backward compatibility of the system [80] allows the developers to use most of the projects, libraries and devices already developed for older models.

The cheap price, a lot of connectivity options and popularity, with the vast range of libraries already implemented, makes the Raspberry Pi series the most suitable for our project.

Wireless communication Providing the e-Health Sensor Platform with wireless commu-nications is necessary to comply with the remote monitoring. Although there are modules and boards for sale that supply these capabilities to the e-Health platform, as a Raspber-ryPi3 is used to capture the thermal images, a better solution is to assemble both of them and use the combined boards as one, taking advantage of a single point of aggregation and communication from the sensor component to the visualization and storage ones. Bluetooth is used to communicate with the Android device and WLAN with the Kafka server.

Battery The battery that powers the sensor platform is a low cost powerbank with a Li-Ion 4000mAh and a 5V*2A exit socket. The recommended power source of the RaspberryPi3 is 5V*2.5A, but we had no problems powering the unit.

Thermal camera In the first version of the system, the thermal imaging device was one de-veloped internally by a fellow student. It was based on a Melexis MLX90621 LWIR thermopile infrared sensor array, as seen on figure 3.5a, external and contactless. For integrating the Melexis thermopile sensor we used an in house integration based on a Arduino (figure 3.5b) with a BLE interface, a nRF8001 BluetoothLE board.

(a) ”The Melexis MLX90621 thermal infrared sensor array.”

(b) ”Prototype of the thermal imaging device. The sensor itself can be seen in the lower left quadrant and the BLE chip in the upper right.”

(50)

Although the temperature was perfectly calibrated from factory, with a range between -50o_{C and 300}o_{C, due to its low resolution, 16x4 pixels, and consequent difficulty in}

identi-fying facial features it was replaced by a more expensive FLIR Lepton.

Currently, in the second version, the temperature sensor used is a FLIR Lepton 1.5 camera core, a passive thermal imaging module developed specifically for consumer mobile equipment (figure 3.6). It is sensitive to LWIR range radiation (wavelengths between 8000nm and 14000nm), the so called thermal infrared radiation [7].

Figure 3.6: The FLIR Lepton thermal camera core assembled in its breakout board.

The sensor inside the FLIR Lepton is a microbolometer array with a resolution of 80x60 pixels. A bolometer pixel has a thermally sensitive layer that changes its resistance accord-ing to the change of its temperature. The pixel temperature increases and decreases as the input radiant power goes up and down. Thus, by measuring this resistance, the infrared radiation of the object that issued it and, consequently, the temperature of that object can be determined [81].

The main characteristics of the Lepton camera are listed in table 3.1.

Specification Description

Sensor technology Uncooled microbolometer

Sensor material Vanadium Oxid

Spectral range LWIR, 8000nm-14000nm

Scene Dynamic Range 0-120oC

Sensor format 80x60px, progressive scan

Pixel size 17000nm

Frame rate 9Hz

Horizontal field of view 50o

Control interface CCI (I2C-like)

Video interface SPI

Operating power 150mW

Standby power 4mW

Package dimensions – socket version 10.8x10.6x5.9 mm (wxlxh)

Weight 0.55grams

ThermoDroid: Android based solution for temperature physiological assessment

Andr ´e Francisco

Pinelas da Silva

ThermoDroid: soluc¸ ˜ao baseada em Android para

avaliac¸ ˜ao fisiol ´

ogica atrav ´es de temperatura

ThermoDroid: Android based solution for

temperature physiological assessment

Contents

List of Figures

List of Tables

Acronyms

Chapter 1

Introduction

1.1

Context and Motivation

1.2

Objectives

1.3

Contributions

1.4

Dissertation Structure

Chapter 2

Background and related researches

2.1

Notions about emotions

2.2

Physiological signals

2.3

Infrared thermal imaging

2.4

Message Oriented Middleware

Chapter 3

Thermodroid architecture and

implementation

3.1

Thermodroid Architecture