Performance measurements and analysis of bluetooth low energy

(1)

André Guedes Linhares

PERFORMANCE MEASUREMENTS AND ANALYSIS OF

BLUETOOTH LOW ENERGY

M.Sc. Dissertation

Federal University of Pernambuco [email protected] <www.cin.ufpe.br/~posgraduacao>

RECIFE 2016

(2)

(3)

André Guedes Linhares

PERFORMANCE MEASUREMENTS AND ANALYSIS OF

BLUETOOTH LOW ENERGY

A M.Sc. Dissertation presented to the Center for Informatics of Federal University of Pernambuco in partial fulfillment of the requirements for the degree of Master of Science in Computer Science.

Advisor: Stênio Flávio de Lacerda Fernandes

RECIFE 2016

(4)

Catalogação na fonte

Bibliotecária Monick Raquel Silvestre da S. Portes, CRB4-1217

L755p Linhares, André Guedes

Performance measurements and analysis of bluetooth low energy / André Guedes Linhares. – 2016.

107 f.: il., fig., tab.

Orientador: Stênio Flávio de Lacerda Fernandes.

Dissertação (Mestrado) – Universidade Federal de Pernambuco. CIn, Ciência da Computação, Recife, 2016.

Inclui referências e apêndice.

1. Ciência da computação. 2. Tecnologia bluetooth. I. Fernandes, Stênio Flávio de Lacerda (orientador). II. Título.

004 CDD (23. ed.) UFPE- MEI 2016-103

(5)

André Guedes Linhares PERFORMANCE MEASUREMENTS AND ANALYSIS OF BLUETOOTH LOW ENERGY

Dissertação de Mestrado apresentada ao Programa de PósGraduação em Ciência da Computação da Universidade Federal de Pernambuco, como requisito parcial para a obtenção do título de Mestre em Ciência da Computação Aprovado em: 17/05/2016. BANCA EXAMINADORA __________________________________________ Prof. Dr. Kelvin Lopes Dias Centro de Informática / UFPE __________________________________________ Profa. Dra. Jeísa Pereira de Oliveira Domingues Departamento de Estatística e Informática / UFRPE __________________________________________ Prof. Dr. Stênio Flávio de Lacerda Fernandes Centro de Informática / UFPE (Orientador)

(6)

(7)

Acknowledgements

I’d like to express my gratitude to my supervisor, Stênio Fernandes, for the useful comments, remarks and engagement through the learning process of this master thesis. Also, I’d like to thank my wife, family and friends who supported me throughout this journey.

(8)

(9)

Resumo

O paradigma da Internet das Coisas (Internet of Things - IoT) tem recebido grande destaque tanto das comunidades científicas como da indústria nos últimos anos. Algumas tecnologias de comunicação sem fio como Bluetooth Low Energy (BLE) tem sido apontadas como tecnologias que terão um papel fundamental na concepção de aplicativos de IoT. Um entendimento profundo de algumas características da tecnologia, como vazão e atraso na trans-missão de dados, latência no estabelecimento de conexão e consumo de energia, é essencial para explorar ao máximo a tecnologia em cenários de IoT e identificar o nicho de aplicação onde esta tecnologia é mais aplicável. Este trabalho apresenta uma avaliação de desempenho da tecnologia BLE em cenários de transmissão de dados e estabelecimento de conexão. As métricas vazão máxima, atraso fim-a-fim, atraso ida-e-volta (Round-Trip Time - RTT) e latência de estabelecimento de conexão são avaliadas através de experimentos. Este trabalho também investiga como alguns fatores, como intervalo de conexão, encriptação de link e tamanho de pacote, impactam nas métricas avaliadas. Por fim, este trabalho apresenta modelos analíticos para as métricas investigadas.

(10)

(11)

Abstract

The Internet of Things (IoT) paradigm has been rapidly gaining ground in the academic and industry communities as one of the most important emerging technologies. Some wireless communication technologies such as Bluetooth Low Energy (BLE) have been pointed as key technologies that will drive IoT applications. In order to leverage the BLE technology in IoT applications and identify the niche of applications this technology is more suitable, it is essential we have a deep understanding regarding some features such as link capacity, data transfer delay, connection establishment latency, and power consumption. This work evaluates the networking capabilities from BLE in scenarios of data transfer and connection establishment. The metrics maximum throughput, one-way delay, round-trip time, and connection establishment latency are evaluated through experimental measurements and we investigate how some factors (e.g. connection interval, pairing, packet size) impact on these metrics. Finally, this work proposes analytical models for the metrics investigated.

(12)

(13)

List of Figures

2.1 BLE protocol stack . . . 24

2.2 Connection establishment message chart . . . 27

2.3 Link encryption message chart . . . 28

2.4 Connection Events . . . 29

3.1 Research aspects versus number of studies . . . 34

3.2 Metrics explored . . . 35

3.3 Performance evaluation methods . . . 36

4.1 Experimental setup . . . 42

4.2 User and kernel space architecture . . . 43

4.3 Maximum Throughput: Bihistogram split by pairing levels . . . 50

4.4 RTT: Histograms grouped by packet size levels . . . 52

4.5 RTT: Bihistogram split by pairing levels . . . 53

4.6 One Way Delay: Bihistogram split by pairing levels . . . 54

4.7 One Way Delay: Histograms for each connection interval level . . . 55

4.8 One Way Delay: Histograms for each packet size . . . 56

5.1 Connection establishment latency: Histogram for all pairing disabled samples . 62 5.2 Maximum Throughput: Regression curve . . . 63

5.3 Maximum Throughput: Regression residuals . . . 64

5.4 One-way Delay: Scatter plot . . . 65

5.5 One-way Delay: Regression curve . . . 66

5.6 One-way Delay: Regression residuals . . . 67

5.7 Connection establishment latency: Regression curve . . . 68

5.8 Connection establishment latency: Regression residuals . . . 69

A.1 Maximum Throughput (kbit/s): Run Sequence plot for samples from 7.5 to 100 (ms) . . . 82

A.2 Maximum Throughput (kbit/s): Run Sequence plot for samples from 125 to 1000 (ms) . . . 84

A.3 Maximum Throughput (kbit/s): Histograms for samples from 7.5 to 100 (ms) . 86 A.4 Maximum Throughput (kbit/s): Histograms for samples from 125 to 1000 (ms) 88 A.5 RTT (ms): Run Sequence plots for all samples . . . 92

A.6 RTT (ms): Histograms for each connInterval level . . . 94

A.7 One Way Delay (ms): Run Sequence plot for all samples . . . 100

(14)

A.9 Connection Latency (ms): Histogram for pairing disabled samples . . . 108 A.10 Connection Latency (ms): Run Sequence plot for pairing enabled samples . . . 110 A.11 Connection Latency (ms): Run Sequence plot for pairing enabled samples

without outliers . . . 112 A.12 Connection Latency (ms): Histogram for pairing enabled samples . . . 114

(15)

List of Tables

3.1 Research questions . . . 31

3.2 Number of selected studies per source . . . 33

3.3 BLE link evaluation studies . . . 38

3.4 Research gaps . . . 40

4.1 Factors and levels from maximum throughput experiment . . . 45

4.2 Constants from maximum throughput experiment . . . 45

4.3 Factors and levels from RTT experiment . . . 46

4.4 Constants from RTT experiment . . . 46

4.5 Factors and levels from connection latency experiment . . . 48

4.6 Constants from connection latency experiment . . . 48

5.1 Mean values for one-way delay metric . . . 61

A.1 Maximum Throughput (kbit/s): Central tendency statistics . . . 90

A.2 Maximum Throughput (kbit/s): Dispersion statistics . . . 91

A.3 RTT (ms): Central Tendency Statistics . . . 96

A.4 RTT (ms): Dispersion Statistics . . . 98

A.5 One-way delay (ms): Central Tendency Statistics . . . 102

A.6 One-way delay (ms): Dispersion Statistics . . . 104

A.7 Connection Latency (ms): Central tendency statistics . . . 116

(16)

(17)

List of Acronyms

6LoWPAN IPv6 over Low power Wireless Personal Area Networks . . . 23

ATT Attribute Protocol . . . 23

BER Bit Error Rate . . . 34

BLE Bluetooth Low Energy . . . 19

CE Connection Event . . . 20

CoAP Constrained Application Protocol . . . 26

CRC Cyclic Redundancy Check . . . 28

EDA Exploratory Data Analysis . . . 49

GAP Generic Access Profile . . . 23

GATT Generic Attribute Protocol . . . 23

IETF Internet Engineering Task Force . . . 23

IoT Internet of Things . . . 19

ISM Industrial Scientific Medical . . . 24

L2CAP Logical Link Control and Adaptation Protocol . . . 23

LL Link Layer . . . 23

PDU Protocol Data Unit . . . 24

RTT Round-Trip Time . . . 41

SIG Special Interest Group . . . 23

(18)

(19)

1

Introduction

The Internet of Things (IoT) paradigm has been rapidly gaining ground in the academic and industry communities as one of the most important emerging technologies. According to that paradigm, many resource constrained objects (e.g. sensors) will be able to interact to each other and services on a global scale through the Internet (GUBBI et al., 2013).

Some wireless communication technologies, such as Bluetooth Low Energy (BLE), have been pointed as key technologies that will drive IoT applications (BORGIA, 2014). Certainly, BLE won’t be the best technology for every IoT application that will come up. We believe a deep understanding of some features such as link capacity, latency, and power consumption is essential to determine when BLE technology meets the communication requirements of a given IoT application.

Actually, a deep understanding of those features is very important in order to leverage the BLE technology in IoT application scenarios and to identify the niche of application this technology is more suitable. Besides that, it also reveals which technologies, indeed, compete in a given niche. Therefore, such technology evaluation is applicable to others emerging wireless technologies such as IEEE 802.15.4, Z-Wave, and DECT ULE.

1.1 Research Questions and Problem Statement

The purpose of this work is to evaluate the networking capabilities of BLE technology in scenarios of data transfer and connection establishment. Below we list the research questions elaborated in order to guide this work:

What is the maximum data rate?

What is the expected latency during data transfer?

What is the expected latency during connection establishment? What are the factors that influence the throughput?

(22)

20 CHAPTER 1. INTRODUCTION

What are the factors that influence the data transfer latency? How does the data transfer latency is affect by those factors?

What are the factors that influence the connection establishment latency? How does the connection establishment latency is affect by those factors?

The following paragraphs formulate some hypotheses regarding the research question presented above. The hypothesis rely on BLE technical aspects which are discussed in details on the Background Chapter (Chapter 2).

In BLE, data transfer occurs in very specific times called Connection Event (CE). The interval between two consecutive CE is defined by the connection interval parameter. If an error occurs during a CE (e.g. two consecutive packets are received with invalid CRC), that CE is terminated prematurely and the data is sent in the next CE. Since in real application scenarios it is expected to have error during transmission, some CE are terminated prematurely therefore the connection interval parameter should impact considerably on the data throughput. Bigger values of connection interval imply more time in "radio silent" when a transmission error occurs thus the throughput value should decrease. Due to the same reason, the connection interval should also influence the latency in both data transfer and connection establishment scenarios.

The packet size from upper layers (above L2CAP layer) should impact on the data transfer latency since the bigger is the packet size more LE Link Layer Data PDUs are required to carry out the data transfer. Moreover, the maximum length of LE Link Layer Data PDU is 27 octets. The L2CAP header is 4 octets. Therefore, upper layer packets with more than 23 octets are broken into smaller packets in order to be transmitted over the LE link. At the receiving side, these packets are reassembled and delivered to the L2CAP layer. This packet fragmentation and reassembly might also cause some impact on data transfer latency metric.

Finally, the BLE pairing feature should have some impact in both data transfer and connection establishment scenarios since it enables BLE Link Layer services such as link encryption which might add some overhead.

1.2 Organization

The remainder of this work is organized as follows: Chapter 2 presents the BLE technol-ogy background which are relevant for this work. All technical aspects regarding connection establishment and data transfer are discussed in details. Chapter 3 presents a systematic review conducted in order to evaluate the state-of-the-art of BLE technology in terms of performance evaluation. It also discusses the main studies highlighted by the systematic review and identify the research gaps which were used to formulate the research questions from this work. Chapter 4 describes the methodology applied to evaluate the BLE networking capabilities. This chapter provides a formal definition for each metric under evaluation, describes the experimental setup,

(23)

1.2. ORGANIZATION 21 and presents an exploratory data analysis from the experiments outcome. Chapter 5 elaborates and discusses the main results and contributions from this work. It evaluates the hypothesis raised by the exploratory data analysis, proposes analytical models for the metrics evaluation and discusses the results from this work in relation with the results presented in other studies listed in Chapter 3. Finally, Chapter 6 draws the final considerations.

(24)

(25)

23 23 23

2

Bluetooth Low Energy Background

The Bluetooh Low Energy technology is a wireless communication technology de-veloped by the Bluetooth Special Interest Group (SIG). It was introduced in Bluetooth 4.0 specification, released in 2010 (BLUETOOTH, 2010), and it has been improved in Bluetooth 4.1 (BLUETOOTH SIG, 2013) and Bluetooth 4.2 (BLUETOOTH SIG, 2015) specifications.

BLE main features are fast connection, low power, low data rate, and short range. In contrast with classic Bluetooth, the BLE technology is much more power-efficient since it was designed with low power consumption in mind. BLE achieves such lower power because it has a lower duty cycle, lower power peak, and lower memory footprint.

In the academic field, several works have been exploring the BLE technology in ap-plications such as smart homes (COLLOTTA; PAU, 2015; CHOI; PARK; LEE, 2015; PAPP et al., 2015), healthcare systems (ZHANG et al., 2014; STREY et al., 2013), medical devices (PARK; CHO, 2013; YU; XU; LI, 2012; LIN et al., 2014; ZHOU et al., 2013), indoor location (SHARHAN; ZICKAU, 2015; CABARKAPA; GRUJIC; PAVLOVIC, 2015), industrial automa-tion (GROVER et al., 2015) and vehicular networks (KANDHALU; XHAFA; HOSUR, 2013; FRANK et al., 2014; BRONZI et al., 2014; LIN; TALTY; TONGUZ, 2015). In the industry, the BLE technology has been deployed in several devices such as Laptops, mobile phones, HID devices, watches, wristlets, heart rate devices (fitness), keyfobs, and medical devices. It is also the wireless communication technology used by Apple’s iBeacon solution which is widely deployed.

2.1 Protocol Stack Overview

The traditional BLE protocol stack comprises the following layers: Physical Layer, Link Layer (LL), Generic Access Profile (GAP), Logical Link Control and Adaptation Protocol (L2CAP), Attribute Protocol (ATT), Generic Attribute Protocol (GATT), and BLE profiles (e.g. proximity and HID over GATT profiles). Recently, the Internet Engineering Task Force (IETF) introduced the IPv6 over Low power Wireless Personal Area Networks (6LoWPAN) layer on top of L2CAP to enable transmissions of IPv6 packets over BLE link. Figure 2.1 illustrates the

(26)

24 CHAPTER 2. BLUETOOTH LOW ENERGY BACKGROUND traditional BLE and the IETF protocol stacks.

Figure 2.1: BLE protocol stack

Source: Made by the author

The Physical Layer specifies features from the BLE technology such as frequency, number of RF channels, and modulation. BLE operates in the 2.4 GHz Industrial Scientific Medical (ISM) band at 2400-2483.5 MHz which is divided in 40 RF channels, 2 MHz each. The RF channels use Gaussian Frequency Shift Keying (GFSK) modulation. This layer is implemented by the BLE controller.

The LL defines how the physical channels are used (37 channels for data transfer and 3 channels for advertising), the packet format (data, control, and adverting Protocol Data Units (PDUs)) as well as the air interface protocol which consists multiple access scheme, device discovery, connection establishment and maintenance, and data transfer. Also, the LL defines a state machine with five states which is implemented by the BLE controller. The states are: standby state (no packets are sent or received), advertising state (send advertising packets and possibly respond to requests), scanning state (listen for advertising packets and possibly send requests to advertisers), initiating state (listen for advertising packets from a specific device and initiate a new connection), and connection state (connection is established). Within the connection state, two roles are defined: Master and Slave. When entered in the connection state from the initiating state, the controller is in Master role. When entered in connection state

(27)

2.1. PROTOCOL STACK OVERVIEW 25 from the advertising state, the controller is in Slave role. The LL is implemented by the BLE controller.

The HCI layer provides an uniform interface method for the host accesses the Bluetooth controller capabilities. This interface is based on commands and events. The specification defines several HCI commands which are used by the host to instruct the controller to carry out operations such as scan nearby devices, connect to devices, transfer data, terminate connection, etc. The controller sends HCI events to the host to inform the status of those commands (e.g. success, fail, busy, etc.) and to general notification such as connection is established and link is encrypted. This layer is implemented by both host and controller.

The GAP layer specifies some operational modes and procedures, such as device discov-ery and connection establishment, the data format from advertising packets and some security aspects. It also defines four roles for BLE devices: Observer, Broadcaster, Peripheral, and Central. The Broadcaster role is optimized for transmitter only applications. Devices supporting the broadcaster role use advertising events to broadcast data. The Observer role is optimized for receiver only applications. Devices supporting the observer role are the complementary device for a broadcaster and receives broadcast data contained in advertisements events. Neither broadcaster nor observer roles support connections. The Peripheral role is optimized for devices that support a single connection and are less complex than central devices (i.e. devices with memory, processing and/or energy constraints). The Central role supports multiple connections and is the initiator for all connections with devices in the peripheral role.

The L2CAP layer provides connection-oriented and connectionless data services to upper layer protocols such as multiplexing capabilities, segmentation and reassembly operations, per-channel flow control, error control and retransmission, and quality of service. However, BLE uses a simplified version of L2CAP which provides multiplexing service and basic mode. This protocol can be implemented in the Bluetooth controller or in the host. In the Linux use case, the L2CAP is implemented in the host at the kernel level.

ATT, GATT and BLE profile layers define a server/client communication protocol on top of L2CAP layer. The communication is based on attributes which are structures to represent data (e.g. temperature value or luminosity level from a sensor). The client application can discover the attributes from the server and read or write attributes by executing the procedures defined by GATT. The BLE profiles standardize the application specific attributes. For instance, any application which implements the temperature profile should define the same set of attributes. The client and server roles are determined by the BLE profile and they are independent of the slave or master role. These protocols can be implemented in the Bluetooth controller or in the host. In the Linux use case, they are implemented in the host at the user level.

Differently from all other layers discussed in this Section, the 6LoWPAN layer has not been developed by the Bluetooth SIG but by the IETF. 6LoWPAN is an adaption layer, on top of L2CAP, that provides several strategies (e.g. address auto configuration, header compression, neighbour discovery, multicast address mapping) to enable IPv6 packets to be transmitted over

(28)

26 CHAPTER 2. BLUETOOTH LOW ENERGY BACKGROUND BLE. 6LoWPAN enables other IETF protocols such as UDP and Constrained Application Protocol (CoAP) to run on top of BLE link. The 6LoWPAN layer is specified in IETF (2014) and it can be implemented in the Bluetooth controller or in the host. In the Linux use case, the 6LowPAN is implemented in the host at the kernel level.

2.2 Connection Establishment

The Bluetooth specification defines some connection modes and procedures. The con-nection modes are non-connectable, direct connectable, and undirected connectable and they are implemented by the Peripheral device only. Peripherals device at non-connectable mode does not allow a connection to be established while directed connectable mode accepts a connection request from a specific Central device and undirected connectable mode accepts a connection request from any Central device.

The connection procedures are auto connection, general connection, selective connection, and directed connection establishment procedures and they are implemented by the Central device only. The auto connection establishment procedure allows the Host to configure the Controller to autonomously establish a connection with one or more Peripheral devices. The general connection establishment procedure allows the Host to establish a connection with a set of known Peripheral devices. The selective connection establishment procedure is very similar to the general procedure but it takes advantage from the Controller’s white list so only advertising events from the selected Peripherals are considered during the procedure. Finally, the directed connection establishment procedure allows the Host to establish a connection to a specific Peripheral device.

In general, a Peripheral device enters in connectable mode when it starts sending con-nectable advertising events and a Central device starts any connection establishment procedure by scanning advertising events. Figure 2.2 illustrates the messages involved during the a directed connection establishment procedure. The messages between the Host and Controller are HCI commands or events while the messages between the Controllers are Link Layer PDUs (e.g. ADV_IND, CONNECT_REQ, and DATA). In this example A represents the Peripheral and B represents the Central.

In order to enter in undirected connectable mode, Host A sends the LE Set Advertising Parameters and LE Set Advertising Enabled commands. The former is used to configure advertising parameters such as advertising type (in this example, undirected advertising) and interval while the latter instructs the Controller A to start advertising. From this point on, the Controller A sends advertising events.

In order to start the directed connection establishment procedure, Host B sends the LE Create Connectioncommand. This command has several parameters including peer address, scanning window, scanning interval, connection interval and connection latency. The scanning window and interval parameters specify how long and how frequent the controller should scan

(29)

2.2. CONNECTION ESTABLISHMENT 27

Figure 2.2: Connection establishment message chart

for advertising events. If scanning window and interval are set to the same value, the controller scans continuously. The connection interval and latency parameters are related to data transfer and are discussed in details on Section 2.3. Once the first advertising from Controller A is received, Controller B sends the LL CONNECT_REQ PDUs to request a connection. Then, both hosts receive the LE Connection Complete event and they are ready to send data to each other through LL DATA PDUs.

When pairing is enabled, the message flow between hosts and controllers are the same, except that before exchanging LL DATA PDUs, the link is encrypted. Figure 2.3 shows the HCI command/event and Link Layer PDUs involved during the link encryption. Note that the encryption procedure is started by the Central device.

For further details about the HCI commands and events used to connection establishment and link encryption refer to Section 7.8 LE Controller Commands, Volume 2, Part E from Bluetooth Core specification (BLUETOOTH SIG, 2015)). Also, for more details on BLE connection modes and procedures refer to Section 9.3 Connection Modes and Procedures, Volume 3, Part C from Bluetooth Core specification (BLUETOOTH SIG, 2015)).

(30)

28 CHAPTER 2. BLUETOOTH LOW ENERGY BACKGROUND

Figure 2.3: Link encryption message chart

2.3 Data Transfer

Once a connection is established, Central and Peripheral devices are able to exchange data. The data transfer occurs in very specific times which are called connection events. During a connection event, the Central and Peripheral alternate sending and receiving packets. The connection event is considered open while both devices continue to send packets. The connection event is closed in the following situations: i) neither Central nor Peripheral has more data to send; ii) no ack packet received; iii) two consecutive packets received with an invalid Cyclic Redundancy Check (CRC); and iv) the next connection event is about to open. It is worth emphasizing that connection events may vary in length.

Right after the connection is established, the Central send the first packet which sets an anchor point for the timing of all future connection events. At each anchor point, the Central sends packet to the Peripheral. The Peripheral acknowledges that packet and may or not send packets on its own as illustrated in Fig. 2.4. The start of connection events are spaced regularly with an fixed interval and doesn’t overlap. The Peripheral listens for packets sent by the Central at every anchor point.

(31)

2.3. DATA TRANSFER 29

Figure 2.4: Connection Events

requirements. The connection event interval is configured through the connection interval parameter from the HCI command LE Create Connection. The connection interval can assume values from 7.5ms to 4s.

The Peripheral is allowed to miss some connection events deliberately but the connection is not considered terminated. The parameter connection latency from the HCI command LE Create Connectionspecifies how many connection events the Peripheral is allowed to miss. This feature enables the Peripheral to save energy when it doesn’t have any data to send.

(32)

(33)

31 31 31

3

Literature Review

This section presents a systematic review which was conducted in order to evaluate the state-of-the-art of BLE technology in terms of performance evaluation. It was developed following the guidelines present in Kitchenham (2004) and Brereton et al. (2007).

The ultimate motivations to carry out this systematic review were: i) to understand the state-of-the-art in-depth, ii) to identify research challenges and open questions, iii) to evaluate how saturated is this area, and iv) to support the research problem from this work.

3.1 Review Protocol

The primary objective of this systematic review is to identify the most relevant studies in that area and to analyze the state-of-the-art. In order to guide this review towards its goals, some research questions were carefully elaborated. They are presented in Table 3.1.

Table 3.1: Research questions

What are the most relevant works on BLE performance evaluation area? How many works are they?

What is the most frequently evaluation method applied? Which metrics those works evaluate?

How those works are related? Is there any divergence in their outcome?

The most reliable repositories of scientific studies were considered as sources of studies in order to ensure the high quality of this review outcome. The sources consulted were the following:

IEEE Xplore Digital Library ACM Digital Library

(34)

32 CHAPTER 3. LITERATURE REVIEW

Springer Link

Multidisciplinary Digital Publishing Institute (MPDI)

The keywords used in this review were derived from the two main terms: Bluetooth Low Energy(the technology under investigation) and performance evaluation (the aspect of the technology under investigation). The selected keywords were the following: Bluetooth, BLE, BTLE, performance, evaluation, measuring, measurements, analysis, model, modeling, analytical, simulation, experiment, and experimental.

The search string varied since each source has its own search string syntax system. However, all search strings used in this systematic review had the same goal: identify studies that have the keywords presented above in their title, following the given logic expression:

(bluetooth OR ble OR btle) AND (performance OR evaluation OR measuring OR measurements OR analysis OR model OR modeling OR analytical OR simulation OR experiment OR experimental)

The search queries retrieved a great number of studies that do not answer the research questions from this systematic review. Therefore, a selection criteria was established to determine which studies are indeed relevant for this effort.

The inclusion criteria were based in two aspects: publication date and main plot of the study. The publication or submission date must be after 2010 (including). The BLE technology was released in the middle of the year 2010 thus it doesn’t make sense to consider works prior to 2010 since they are most likely related to "classic" Bluetooth technology which is not part of any research questions. The primary or secondary plot of the study must be related to performance evaluation of some aspect of BLE technology (e.g. connection, device discovery, energy consumption, link capabilities), and this should be evidenced by analyzing the title, abstract, and keywords of the study.

3.2 Data Analysis and Mapping

The search queries retrieved 692 studies. After the selection criteria, this number was narrowed down to 23. 629 studies (91%) were not selected due to the publication date. This suggests that "classical" Bluetooth has been widely explored, but BLE might not yet. Some other studies were not selected because they were not focused on performance evaluation. For instance, Tabish et al. (2013) and Georgakakis et al. (2011) present a very trivial comparison between BLE and others Wireless Sensor Network (WSN) technologies based on technology features (e.g. range, radio frequency, network topology) while Joh and Ryoo (2014), Joh, Yang and Ryoo (2015), Galinina et al. (2014) and Mal’kov et al. (2015) present some applications of the BLE technology. Finally, a few more studies were not selected because even if they are about performance evaluation, they are focused on "classical" Bluetooth.

(35)

3.3. STUDIES REVIEW 33 Table 3.2 presents the number of selected studies according to its source. It shows that IEEE Proceedings and Journals have been the main source of studies in the BLE performance evaluation area. Surprisingly, no BLE performance evaluation study, following the selection criteria of this systematic review, was found in ACM Digital Library.

Table 3.2: Number of selected studies per source

Source # of selected studies IEEE Xplore Digital Library 15

Springer Link 4

MPDI 3

ScienceDirect 1 ACM Digital Library 0

All selected studies were screened following the guidelines described in Keshav (2007). The next paragraphs provide a preliminary discussion about the selected studies in order to elaborate answers to the research questions presented in Table 3.1.

The studies explore some aspects from the BLE technology such as device discovery, connection establishment, data transfer, and signal propagation. The majority of the studies explore only one of these aspects while a very few of them such as Siekkinen et al. (2012) and Sommer et al. (2011) explore more than one. Figure 3.1 shows the number of studies that investigates each of these aspects. It shows that device discovery and data transfer have been the most explored aspects with 10 and 12 studies published so far.

These studies evaluate at least one metric. The most recurrent metrics are: energy consumption, latency, throughput, and packet loss. Figure 3.2 shows the number of studies that explores each of these metrics. It reveals that more than 50% of the studies considered in this systematic review evaluate the energy consumption of BLE technology and, therefore, energy consumption has been the most explored metric.

Experiments with real hardware have been the most applied method to evaluate the performance of BLE technology as shown in Figure 3.3. In contrast, the simulation method has been applied by a very few studies due to the lack of BLE support in the main simulation tools (MIKHAYLOV, 2014).

Finally, some studies such as Siekkinen et al. (2012), Dementyev et al. (2013), and Mikhaylov, Plevritakis and Tervonen (2013) provide some comparison between BLE and others WSN technologies (e.g. ZigBee) based on performance evaluation.

3.3 Studies Review

This section presents a review of the studies identified in the systematic mapping which are related this work. All studies that explore networking metrics in data transfer scenarios of

(36)

Figure 3.1: Research aspects versus number of studies

the BLE technology are presented in details and discussed in the following paragraphs.

Gomez, Demirkol and Paradells (2011) presents an analytical model for the maximum throughput of BLE technology in an error-prone link. The maximum throughput is modeled as a function of Bit Error Rate (BER) and the connection interval parameter. To validate the model, the authors performed simulations for various BER and connection interval values and illustrated both model and simulation results in a diagram. The authors claim that their model is accurate in predicting the maximum BLE throughput since the model and simulation curves match. However, the paper doesn’t discuss or provide any information regarding the simulation tool used. Finally, according to the model proposed in this work, the maximum BLE throughput is 236.7 kbit/s which is achieved in absence of bit errors.

Gomez, Oller and Paradells (2012) investigates the impact of the connection interval and connection latency parameters, and Bit Error Rate (BER) on BLE performance in terms of energy consumption, latency and piconet size. It also evaluates the maximum throughput. The experiments considered both one-way and round-trip ATT communication scenarios. Energy consumption has been analyzed both theoretically and empirically, based on CC2540 radio chip. According to the authors, the maximum slave lifetime obtained is 14.1 and 12.4 years for the one-way and round-trip methods, respectively. These values are achieved for both options for a connection interval of 86.25 ms and a connection latency of 370 ms. With these settings, the master obtains one sensor measurement reading every 32 seconds. The authors also present how the lifetime is influenced by the BER. The latency results have been obtained by simulation

(37)

3.3. STUDIES REVIEW 35

Figure 3.2: Metrics explored

however the authors don’t elaborate on the simulation tools used. According to the outcomes, for very low BER values (e.g., 10−6), the average latency of round-trip and one-way ATT message exchanges are smaller than 2 ms and 1 ms, respectively. For greater BER values, the latency increases drastically and it can achieve up to 1 second. The piconet size results presented in this study have been obtained theoretically and considers ideal communication (BER = 0). According to the outcome, considering the round-trip scenario and 7.5 ms connection interval, only 1 slave is supported while considering one-way scenario and 4 seconds connection interval the maximum number of slave is 5917. The authors highlight that the maximum piconet size in real scenarios may be smaller then due to limitations in terms of memory or antenna availability in the master. Finally, experiments with two CC2540 devices were conducted and revealed that 58.48 kbit/s is the maximum throughput achieved.

Mikhaylov, Plevritakis and Tervonen (2013) compares the maximum throughput, the minimum frame turnaround time, and the energy consumption of BLE, IEEE 802.15.4 and SimpliciTI technologies. The authors present analytic models for the maximum throughput and the minimum frame turnaround time metrics, considering each wireless technology. The authors also carried out experiments with real hardware in order to validate the analytic models. The experiments considered only 7.5 ms and 15 ms values for connection interval. According to their model, the maximum throughput is 320 kbit/s however the experimental results revealed that it is 123 kbit/s. Likewise, the model outcomes that the frame turnaround time is less than one millisecond, but the experimental results reveled it is 7.6 ms. The authors discuss

(38)

Figure 3.3: Performance evaluation methods

about those mismatches between their models and the experimental results and conclude that it is caused by some implementation details on the BLE firmware running on CC2540 device. This work also presents an analysis of the maximum throughput of BLE adverting channels. The maximum throughput measured was around 8 kbit/s for ADV_IND and 2.2 kbit/s for ADV_NONCONN_IND and ADV_SCAN_IND events. Regarding the energy consumption, the authors claim that BLE technology requires two to seven times less energy to transfer data than the SimpliciTI and the IEEE 802.15.4.

Nieminen et al. (2014) evaluates the BLE performance from the perspective of IETF and Bluetooth Special Interest Group standardization technologies: IPv6, 6LoWPAN, and CoAP. The main goal of this work is to demonstrate that these technologies are feasible to enable IoT applications. In addition, the authors discuss gateway operation covering global IPv6 connectivity and proxy-cache functionality. The proposed model for energy consumption is based on energy measurements from TI CC2540 chip described in Instruments (2010). The authors co-relate the energy consumption with the IPv6 packet size and the connection interval. The authors also evaluate the IPv6 throughput using both theoretical results (obtained by means of analysis, assuming an ideal link) and experimental results. However, only two values for the connection interval (7.5 and 30 ms) were considered in the experimental results. Finally, the authors evaluate the round-trip latency from the CoAP request-reply exchange.

Mikhaylov (2014) introduces a novel simulation tool which is capable of simulating the communication in the networks of BLE devices. The basis for the tool is the MiXiM framework

(39)

3.3. STUDIES REVIEW 37 based on the popular OMNeT++ engine. The tool implements the PHY and Link layer only. It supports simulation of BLE advertising, scanning, connection establishment, connection upkeep and connection termination. The data collect during simulations include: number of sent and received packets, the used radio channel at each moment of time, and consumed current and energy. In order to validate the proposed simulation tool, the author conducted a series of experiments and compared the results with the ones previously reported in Mikhaylov, Plevritakis and Tervonen (2013). This study also presents some evaluation on the maximum throughput and energy consumption, considering several connection interval values and number of slaves varying from 1 to 64 devices. The tool doesn’t take in consideration the BER which is essential in order to simulate real life scenarios. Finally, the author claims his tool is the only BLE simulation tool available at a time. The tool has not been released.

Bronzi et al. (2015) investigates the potential of BLE technology for inter-vehicular communications. The study analyzes the BLE performance through experiments using BLE-enabled mobile devices, considering both single-hop and multi-hop scenarios. It investigates the impact of distance, speed, and signal interference on the delivery ratio, round-trip time, and packet rate metrics for single-hop scenarios and analyzes the end-to-end delay for the multi-hop scenario. The outcomes demonstrate that up to 50 m the two devices are able to achieve 100% delivery rate. From this point on the communication between the two devices is unreliable as the delivery rate drastically drops to around 40% at 80 m distance. They also show that the RTT remains stable in 100 ms for distances below 60 m and it rapidly increases with longer distances. The authors analysis reveals that the end-to-end delay for two hops is 2.85 seconds. Due to this long delay, they state that BLE is not suitable for some safety applications such as signaling a sudden emergency brake or a danger on the road ahead. However, the authors don’t elaborate on which connection and advertising parameters values were considered in the analysis. These parameters impact directly on the end-to-end delay. Aggressive values could reduce the end-to-end delay, changing that statement. Finally, the authors conclude that BLE can be used to exchange information between vehicles while driving, but it cannot be considered as a complete replacement for on-board communication interfaces such as DSRC/802.11p.

In Giovanelli, Milosevic and Farella (2015), the authors present a performance evaluation of BLE throughput and energy consumption. Their analysis is based on experiments carried out with three different hardware manufactures (Texas Instruments, Nordic Semiconductor, and ST Microelectronics) running as Peripheral devices and a smartphone running as the Central device. According to the outcomes, the maximum throughput achieved is 64 kbit/s for all manufactures. The authors claim that this throughput value is limited by the BLE firmware from the smartphone. The authors also evaluate the current consumption as a function of the throughput. As expected, the outcome shows that the current consumption decreases with the decrease of the data throughput. Finally, the authors evaluate the impact of the number of slave devices in the maximum throughput. They conducted experiments with up 6 slaves. The results show that with two slaves the maximum throughput drops to 21.3 kbit/s while with six slaves

(40)

38 CHAPTER 3. LITERATURE REVIEW it achieves 8 kbit/s. Based on the experimental outcome, the authors purpose a model that calculates the throughput as a function of the connection interval.

3.4 Discussion

The studies presented in Section 3.3 that explore the BLE link features are summarized in Table 3.3. The table represents a matrix which correlates these studies with the metrics they investigate and the evaluation method used in their analysis.

Table 3.3: BLE link evaluation studies

Throughput RTT One-way Delay Theoretical (GOMEZ; DEMIRKOL; PA-RADELLS, 2011), (MIKHAYLOV; PLEVRITAKIS; TERVONEN, 2013) (MIKHAYLOV; PLEVRITAKIS; TERVONEN, 2013) Simulation (MIKHAYLOV, 2014) (GOMEZ; OLLER; PA-RADELLS, 2012) (GOMEZ; OLLER; PA-RADELLS, 2012) Measurement (GOMEZ; OLLER; PA-RADELLS, 2012) (MIKHAYLOV; PLEVRITAKIS; TERVONEN, 2013) (GIO-VANELLI; MILOSEVIC; FARELLA, 2015) (MIKHAYLOV; PLEVRITAKIS; TERVONEN, 2013) (BRONZI et al., 2015)

Table 3.3 shows that throughput, RTT, and one-way delay are the main networking metrics used to evaluate the BLE link. It also evidences that others common networking metrics such as packet loss and delay jitter have not been evaluated by any studies. Probably, the reason why packet loss has not been explored is because the BLE Link Layer guarantees packet delivery. This way, from the upper layers point of view, the packet loss rate is always zero. Regarding the delay jitter, BLE doesn’t seem to be suitable for real-time audio and video applications so this metric is not very important.

Table 3.3 also reveals that, for BLE link evaluation, experimental measurement analysis has been the most common method with four studies while theoretical and simulation methods

(41)

3.5. FINAL CONSIDERATIONS 39 has two studies each. As stated in Gomez, Oller and Paradells (2012) and verified in this work, the reason why simulation method has not been widely explored so far is the lack of BLE support in simulation tools. The simulation tool introduced in Mikhaylov (2014) has not been released publicly.

The review presented in Section 3.3 shows that the studies diverge about the maximum throughput the BLE link can achieve. The theoretical models introduced in Gomez, Demirkol and Paradells (2011) and Mikhaylov, Plevritakis and Tervonen (2013) state that the maximum throughput is 236.7 kbit/s and 320 kbit/s, respectively. The studies that analyze this metric through experimental measurement states a much lower throughput, but they still diverge from each other: Gomez, Oller and Paradells (2012) states it is 58.48 kbit/s while Mikhaylov, Plevri-takis and Tervonen (2013) states 123 kbit/s and Giovanelli, Milosevic and Farella (2015) 64 kbit/s.

The BLE latency analysis from Gomez, Oller and Paradells (2012) and Mikhaylov, Plevritakis and Tervonen (2013) seems to diverge as well. Considering connInterval equals to 7.5 ms for instance, the measurements in Mikhaylov, Plevritakis and Tervonen (2013) indicate that the RTT is slightly above 7.5 ms while the simulation results present in Gomez, Oller and Paradells (2012) show that the RTT is around 4 ms. The RTT measurements from Bronzi et al. (2015) (around 100 ms) diverge drastically from the outcome from both Gomez, Oller and Paradells (2012) and Mikhaylov, Plevritakis and Tervonen (2013). However this is expected since Bronzi et al. (2015) evaluates very specific scenarios (inter-vehicular communication) with different factors such as distance and speed.

Three studies investigate the throughput metric (GOMEZ; OLLER; PARADELLS, 2012; MIKHAYLOV; PLEVRITAKIS; TERVONEN, 2013; GIOVANELLI; MILOSEVIC; FARELLA, 2015) while two studies (MIKHAYLOV; PLEVRITAKIS; TERVONEN, 2013; BRONZI et al., 2015) explores the RTT metric, through experimental analysis. Their analysis consider very few and small values of connInterval (7.5, 15, and 30 ms). According to the BLE spec, the connIntervalcan assume values from 7.5 ms to 4 seconds. Thus, these studies are not exhaustive. Moreover, none of those studies propose an analytical model based on measurements for the metrics under investigation.

3.5 Final Considerations

The previous section also reveals that none of the studies evaluates scenarios where the BLE link security feature is enabled. Likewise, none of them considered different L2CAP packet sizes in their evaluation. Those two parameters may have a significant impact in maximum throughput, RTT, and one-way delay metrics.

Additionally, only Naik, Kuwelkar and Magdum (2015) considers the connection estab-lishment scenario, but it evaluates the energy consumption metric. Therefore, none of the studies explores the BLE connection establishment latency. Factors such connection interval parameter

(42)

40 CHAPTER 3. LITERATURE REVIEW and encryption enabled are expected to impact on connection establishment latency.

Finally, only one study (NIEMINEN et al., 2014) evaluates the BLE throughput from the Internet of Thingsperspective (6LoWPAN over BLE scenarios). However only two connInterval values (7.5 and 30 ms) were considered in the analysis. Further analysis considering more connInterval values and BLE link encryption feature is required. Table 3.4 summarizes all research gaps we detected through this systematic review.

Table 3.4: Research gaps

The impact of connection interval parameter on maximum throughput, RTT, and one-way delay metrics through experimental analysis.

The impact of BLE security feature on maximum throughput, RTT, and one-way delay metrics.

The impact of L2CAP packet size on RTT and one-way delay metrics. Models based on measurement for BLE maximum throughput, RTT, and one-way delay.

Overall analysis of connection establishment latency. How connection in-terval parameter and link encryption impact on connection establishment latency.

(43)

41 41 41

4

Materials and Methods

This Chapter describes the methodology used to explore the BLE networking capabilities. The selected performance metrics, namely maximum throughput, round-trip time, one-way delay, and connection establishment latency, are evaluated through experimentation. These metrics are essential to identify in which niche of applications the BLE technology is more suitable.

In order to present results close to real-life use cases, this work evaluates all performance metrics through measurement using real hardware. Complementary evaluation through simu-lation method was considered however, due to the lack of BLE support in simusimu-lation tools, it was not carried out. As presented in Chapter 3, Mikhaylov (2014) presents the only known simulation tool for BLE, but it hasn’t been released yet.

This Chapter is organized as follows: Section 4.1 defines each metric under investigation, Section 4.2 describes the overall experimental setup (e.g. hardware configuration, software versions, BLE device manufacture), Section 4.3 elaborates on experimental design aspects such as sample size and number of repeated trials, Section 4.4 provides detailed information about each experiment carried out (e.g. experiment application description, performance metrics and how they are measured, factors and levels, constants), and, finally, Section 4.5 presents an Exploratory Data Analysis on the results.

4.1 Performance Metrics

Maximum Throughput is the rate of data successfully delivered over a communication channel and it is usually measured in bits per second (bps). This metric is calculated through the following formula:

C= D/T 4.1

D is the total amount of data captured in the Receiver node. This value can be found by multiplying the total number of packets captured by the size of the packets. T is the time that took to capture the packets.

Round-Trip Time (RTT) is the amount of time between the instant the Sender node sends an ordinary packet and the instant the Sender node receives the corresponding acknowledge

(44)

42 CHAPTER 4. MATERIALS AND METHODS packet.

One-Way Delay is the amount of time the packet takes to travel between Sender and Receiver nodes. This amount of time is the difference between the time the packet arrives at the Receiver node and the time the packet departures at the Sender node. Evidently, the system clock from Sender and Receiver nodes must be synchronized otherwise that time difference won’t reflect the packet delay properly.

Connection Establishment Latency corresponds to the amount of time it take from the instant the user requires a connection until the instant the BLE link is ready for data transfers.

4.2 Experimental Setup

All experiments carried out in order to evaluate the BLE networking metrics were performed following the experimental setup described in this section. The setup consists in two laptops one meter away from each other, running the Linux kernel version 3.17 and the BlueZ software stack version 5.28. Both laptops were equipped with a Bluetooth 4.0 USB adapter from CSR with CSR8510 A10 chipset. Figure 4.1 illustrates the experimental setup.

Figure 4.1: Experimental setup

The btmgmt tool is used to configure the Bluetooth adapters. The Bluetooth adapter is managed by the Management Interface within the Bluetooth kernel subsystem. The Management Interface is a Command & Event interface which provides functionalities such as powering on the adapter, enabling low energy support, enabling connectable mode, and starting advertising. So the btmgmt tool enables the user to configure the Bluetooth adapter by sending commands to the Management Interface.

The l2test tool is used to generate traffic between the two nodes. This tool provides basic networking operations such as listen for connections, establish connections, send data, and receive data. These operations are implemented thought L2CAP socket APIs such as bind(), connect(), listen(), accept(), send(), and receive().

(45)

4.3. EXPERIMENTAL DESIGN 43 The btmon tool is used to capture the HCI command and events as well as the data packets sent and received by the Bluetooth dongle. The metrics under investigation are measured through the analysis of the packets captured during the experiments. The packets are captured in the HCI core layer from the Bluetooth kernel subsystem. More precisely, an outgoing packet is captured right before the Bluetooth subsystem sends it to the device driver, while an ingoing packet is captured at the instant the device driver delivers it to the Bluetooth subsystem. The kernel then sends a copy of the captured packet to the btmon process which in turn dumps it in a file. Figure 4.2 illustrates the overall architecture of both user and kernel space, and shows where exactly the packets are captured.

Figure 4.2: User and kernel space architecture

All tools used in this experimental setup (btmgmt, l2test, and btmon) are part of the BlueZ project.

4.3 Experimental Design

The evaluation of the metrics presented in 4.1 is conducted through experiments based on repeated measurement design. Therefore, the number of trials considered in the evaluation is very important since it might compromise the evaluation outcome.

(46)

44 CHAPTER 4. MATERIALS AND METHODS It is well-known that the bigger is the number of trials, more accurate are some statistics such as sampling mean and standard deviation. However, performing lots of trials can take too long, making unfeasible some experimental evaluations. Besides that, depending on the parameters defined for the statistical model (e.g. confidence level, type II error, and estimation error), increasing the number of trials after a certain threshold would be wasteful.

Preliminary experiments (as well as the outcome presented in Section 4.5) have shown that none of the metrics under investigation have a heavy-tailed distribution. According to Central Limit Theorem, the sample statistic is a good estimator as long as the sample is size greater than 30. Additionally, we want to provide 95% confidence level for all statistical analysis. According to Sematech (2006), a number of trials equals or greater than 100 meets that confidence level. Therefore, we chose the number of trials equals to 100 since this value is enough to achieve our statistical model requirement.

Another important factor that could compromise our evaluation outcome is the number of packets to be transmitted during the experiments. If we choose a small number of packets, the experiments might not reproduce some important conditions from BLE link layer. For instance, small number of packets might not reproduce situation where a connection event is closed because no ACK packet is received or two consecutive packets are received with an invalid CRC. As stated before, closing a connection event prematurely impacts directly on the metrics we are evaluating. In order to identify a more accurate number of packets to all experiments, a preliminary evaluation was carried out. We could verify that, with 500 packets, we always reproduced the situation where connection events were closed. Thus, we believe that transmitting 500 packets are enough to reproduce all BLE link layer conditions as well as provide good statistics accuracy.

4.4 Experiments Description

Four experiments were performed in order to evaluate the four metrics introduced in this Chapter, and they are described in details in the following Sections. As discussed in Chapter 2, BLE is a short-range technology. Since preliminary experiments suggested the distance doesn’t have significant impact on the metrics and adding just a few levels of distance would increase the time to run all experiments considerably, the distance between Sender and Receiver nodes is fixed in one meter for all experiments.

4.4.1 Experiment #1: Maximum Throughput

The Experiment #1 aims at evaluating the maximum throughput from BLE technology. It consists of two nodes: Sender and Receiver. The Sender node sends 500 data packets while the Receiver node simply receives them. The experiment terminates when all packets are received by the Receiver node. Both Sender and Receiver nodes dump the packets. The l2test tool is

(47)

4.4. EXPERIMENTS DESCRIPTION 45 used to send and receive packets while the btmon tool is used to dump packets. Both nodes are configured using the btmgmt tool.

The performance metric from this experiment is the maximum throughput and it is measured by analyzing the captured packets in the Receiver node. The maximum throughput is calculated by the Equation 4.1. The amount of data transmitted is found by multiplying the number of packets captured and the packet size while the elapsed time is calculated by subtracting the time stamp from the last packet and the time stamp from the first packet.

The factors from this experiment are the connection interval and pairing. The levels from these factors considered in this experiment are listed in Table 4.1.

Table 4.1: Factors and levels from maximum throughput experiment

Factor Levels

Connection interval 7.5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 75, 100, 125, 150, 175, 200, 250, 300, 350, 400, 450, 500,

and 1000 milliseconds Pairing true and false

As presented in Chapter 2, the connection interval can assume several values from 7.5 ms to 4 seconds. Preliminary experiments have shown that slightly changes on that interval have a considerable impact on the maximum throughput when connection interval is small. For that reason, the connection interval levels are more dense in the ranger of 7.5 ms and 200 ms. The pairingvariable can assume only two levels: true (Sender and Receiver are paired nodes) and false (Sender and Receiver are not paired nodes).

In this experiment, the connection latency parameter, L2CAP packet size and the number of packets transmitted are constant and their values are present in Table 4.2.

Table 4.2: Constants from maximum throughput experiment

Constant Value Connection latency 0

Packet size 27 octets # of packets 500

The packet size value was chosen based on the maximum amount of data which can be transmitted by one LL Data PDU. The maximum length from the PDU payload is 27 octets, but 4 octets are already used by the L2CAP header. Thus, the payload size is set to the remaining available data (23 octets) in order to use the full payload capacity.

(48)

46 CHAPTER 4. MATERIALS AND METHODS Since this experiment aims to obtain the maximum data rate, the connection latency parameter is set to zero.

The experiment described in this Section provides 46 samples (one for each combination of connection interval and pairing levels). The experiment is repeated 100 times.

4.4.2 Experiment #2: Round-Trip Time

The Experiment #2 aims to evaluate the round-trip time from BLE technology. It consists in two nodes: Sender and Receiver. The Sender node sends a packet and waits for the reply. Once the Receiver node receives a packet, it immediately sends a reply. The experiment terminates when the Sender receives the reply. Both Sender and Receiver nodes dump the packets.

In this experiment, the round-trip time is the performance metric and it is measured by analyzing the captured packets in the Sender node. The RTT is calculated by the difference between the time stamp from the inbound packet and the outbound packet.

The factors from this experiment are connection interval parameter, pairing, and packet size. The levels considered in this experiment are listed in Table 4.3.

Table 4.3: Factors and levels from RTT experiment

Factor Levels

Connection interval 7.5, 15, 30, 40, 50, 75, 100, 150, 250, 500, and 1000 milliseconds

Pairing true and false

Packet size 20, 50, 100, 250, and 500 octets

The levels of the packet size factor were carefully chosen to reproduce scenarios where a different number of LL Data PDUs are required to transmit the packet. As presented in Chapter 2, the maximum lenght of the LL Data PDU payload is 27 octets so to transmit a 20-octets packet only one PDU is required while to transmit a 500-octets packet several PDUs are required.

In this experiment, the connection latency parameter is the only constant and its value are present in Table 4.4.

Table 4.4: Constants from RTT experiment

The experiment described in this Section provides 110 samples (one for each combination of the factors). The experiment is repeated 100 times.

(49)

4.4. EXPERIMENTS DESCRIPTION 47

4.4.3 Experiment #3: One-Way Delay

The Experiment #3 aims to evaluate the one-way delay from BLE technology. It consists in two nodes: Sender and Receiver. The Sender node sends 500 data packets while the Receiver node simply receives them. The experiment terminates when all packets are received by the Receiver node. Both Sender and Receiver nodes dump the packets.

The performance metric from this experiment is the one-way delay and it is measured by analyzing the captured packets in Receiver nodes. The one-way delay is calculated by the difference between the time stamp from two consecutive packets.

As discussed in Section 4.1, the traditional approach to measure the one-way delay is to calculate the difference from the time stamp when the packet arrives at the Receiver node and the time stamp when the packet departures at the Sender node. This approach requires that system clock from both nodes are primarily synchronized, using a clock synchronization protocol. The Network Time Protocol (NTP) the standard clock synchronization protocol from Linux distributions. Preliminary experiments have shown that the one-way delay from BLE can achieve 1 or 2 milliseconds hence the synchronization accuracy required to implement the traditional approach would be in the microseconds magnitude which is not provided by NTP. For this reason, we use an alternative approach based on the time difference from consecutive packets.

The factors, levels, and constants from this experiment are the same from the Round-Trip Time experiment described in Section 4.4.2 and they are described in Tables 4.3 and 4.4, respectively.

Likewise the Round-Trip Time experiment, this experiment provides 110 samples (one for each combination of factors). The experiment is repeated 100 times.

4.4.4 Experiment #4: Connection Establishment Latency

The Experiment #4 aims to evaluate the latency from the connection establishment procedure from the BLE technology. It consists in two nodes: Central and Peripheral. The Peripheral node is put in connectable mode and then the Central node initiates the connection establishment procedure. The experiment terminates when the connection is established.

In this experiment, the latency is the performance metric and it is measured by analyzing the captured HCI commands and events. The connection establishment latency is calculated by the difference between the time stamps from LE Create Connection HCI command and LE Connection CompleteHCI event when pairing is disabled, and by the difference between the time stamps from LE Create Connection HCI command and Encryption Change HCI event when pairingis enabled.

The factors from this experiment are connection interval and pairing and the levels considered in this experiment are listed in Table 4.5.

(50)

48 CHAPTER 4. MATERIALS AND METHODS

Table 4.5: Factors and levels from connection latency experiment

Factor Levels

Connection interval 7.5, 15, 30, 50, 75, 100, 150, 250, 500, 1000 milliseconds

Pairing true and false

In this experiment, the connection latency, scanning window, scanning interval, and advertising interval are constant and their values are present in Table 4.6.

Table 4.6: Constants from connection latency experiment

Scanning window 2.5 ms Scanning interval 2.5 ms Advertising Interval 20 ms

The constant values were carefully chosen in order to minimize the impact of the discovery process in the connection establishment latency. The advertising interval is set to the minimum value so the Peripheral node advertises at the maximum frequency. The scanning window and scanning interval are set to the same value so the Central node scans continuously. The experiment described in this Section provides 20 samples (one for each combination of connection interval and pairing). The experiment is repeated 100 times.

4.4.5 Uncontrolled Factors

The outcome from all experiments described in the previous Sections may be affected by some uncontrolled factors. These factors are enumerated and described below.

Environment Signal Noise: A high noise in the environment may cause packet losses,

increase packet delay as well as degrading the link capacity;

System Load: If system load is high during the experiments, the outcome results may

(51)

4.5. EXPLORATORY DATA ANALYSIS 49

4.5 Exploratory Data Analysis

This section presents an analysis of the outcome from the experiments based on the Exploratory Data Analysis (EDA) methodology introduced in Sematech (2006). EDA is an approach for analyzing data that employs a variety of techniques to maximize insight into a data set. Most EDA techniques are graphical in nature with a few quantitative techniques and they were applied in order to investigate samples features such as central tendency, dispersion, distribution, randomness, and outliers.

4.5.1 Experiment #1: Maximum Throughput

The maximum throughput outcome consists of 4600 measurement. Each pair of pairing and connection interval levels (e.g pairing disabled and 10 ms) has 100 samples.

Figures A.1 and A.2 show the run sequence plot from all samples grouped by the connection interval levels. The circles represent the measurements with pairing factor at level disabledwhile the crosses represent measurement at level enabled. The plots from connection interval at levels 40, 75, 150, 250, 350, 500 and 1000 ms suggest that few samples may be outliers. The dump files from those samples were analyzed and we are confident that those samples represent valid measurements. These measurements represent situations where the connection event was prematurely closed. Therefore they are not considered outliers in the further analysis. We can observe that, according to the way the circles and crosses are displayed in relation to each other in the graphs, the pairing factor doesn’t seem to have much impact on the maximum throughput.

Figure 4.3 shows a bihistogram from all samples, split by the levels of the pairing factor. We can notice that the bihistogram is symmetric. This bihistogram feature indicates that the factor pairing doesn’t have any considerable impact on the sample distribution neither the sample locality and variation. This supports the affirmation from the previous paragraph about the pairingfactor having minimal or no influence on the maximum throughput.

Figures A.3 and A.4 presents the histograms from all samples grouped by the levels of connection interval. The analysis of the histograms shows that for low levels of connection interval (e.g. from 7.5 to 20 ms) the samples have only one mode and seems to have a normal distribution. The histograms from samples from 25 to 50 ms presents one mode and left-skewed distribution while the remaining samples present are bimodal. Therefore, the connection interval factor influences on sample distribution, locality and variation, impacting directly on the maximum throughput metric.

Due to the different distribution model from the maximum throughput samples, it is hard to select only one statistic to summarize the central tendency and dispersion from all samples. For that reason, Table A.1 presents several measurements of central tendency, including the canonicals mean, median, and mode. The mode is computed as the midpoint of the histogram

Performance measurements and analysis of bluetooth low energy

André Guedes Linhares

PERFORMANCE MEASUREMENTS AND ANALYSIS OF

BLUETOOTH LOW ENERGY

M.Sc. Dissertation

André Guedes Linhares

PERFORMANCE MEASUREMENTS AND ANALYSIS OF

BLUETOOTH LOW ENERGY

Acknowledgements

Resumo

Abstract

List of Figures

List of Tables

List of Acronyms

Contents

1

Introduction

1.1

Research Questions and Problem Statement

1.2

Organization

2

Bluetooth Low Energy Background

2.1

Protocol Stack Overview

2.2

Connection Establishment

2.3

Data Transfer

3

Literature Review

3.1

Review Protocol

3.2

Data Analysis and Mapping

3.3

Studies Review

3.4

Discussion

3.5

Final Considerations

4

Materials and Methods

4.1

Performance Metrics

4.2

Experimental Setup

4.3

Experimental Design

4.4

Experiments Description

4.4.1

Experiment #1: Maximum Throughput

4.4.2

Experiment #2: Round-Trip Time

4.4.3

Experiment #3: One-Way Delay

4.4.4

Experiment #4: Connection Establishment Latency

4.4.5

Uncontrolled Factors

4.5

Exploratory Data Analysis

4.5.1

Experiment #1: Maximum Throughput