GaitGate – Towards a multi-scenario
clinical gait characterization system for
neurological diseases
Joana Catarina Moreira Rodrigues
D
ISSERTATIONMestrado Integrado em Bioengenharia
Supervisor: João Paulo Trigueiros da Silva Cunha, PhD
characterization system for neurological diseases
Joana Catarina Moreira Rodrigues
Mestrado Integrado em Bioengenharia
O movimento humano, especificamente a marcha, está diretamente relacionada à cognição, sendo uma interação complexa entre o sistema nervoso central, nervos e músculos. Assim, a quantifi-cação da marcha e da sua variabilidade são de extrema importância, sendo indicadores da saúde geral de um indivíduo.
A prevalência de doenças neurológicas relacionadas com o movimento tem aumentado, prin-cipalmente na população idosa, sendo uma das principais causas de alterações da marcha. Assim, o diagnóstico clínico e o acompanhamento de pacientes com doenças neurológicas são frequente-mente realizados através da inspeção visual dos seus padrões de marcha, o que é uma medição subjetiva e dependente da opinião dos médicos. Um sistema automático que avalia e quantifica a marcha traz melhorias significativas na prática clínica, ajudando os médicos a prescrever trata-mentos farmacológicos mais adequados e permitindo uma análise mais objetiva que detecta até pequenas variações.
O sistema NeuroKinect é a solução proposta pelo Laboratório BRAIN, no INESC TEC, e trata-se de um sistema de análise de marcha baseado na câmara Microsoft Kinect, que adquire dados de pacientes que caminham para frente e para trás, ciclicamente, posteriormente processa os dados, medindo um total de 48 parâmetros de marcha, e, finalmente, fornece feedback imediato, com a geração de um relatório que inclui alertas, que ajudam na interpretação dos resultados. Com a descontinuação do sensor Microsoft Kinect, surgiu o objetivo deste trabalho, de adaptar o atual sistema NeuroKinect para tecnologias mais recentes.
Assim, novas câmaras de profundidade (RGB-D) e ferramentas de rastreamento de movimento humano foram estudadas e uma nova interface foi desenvolvida, o Nui-MTracker. Esta recorre ao Nuitrack SDK e é compatível com vários sensores RGB-D. Esta aplicação de software integra o sistema NeuroKinect, permitindo a visualização e a extração dos dados durante aquisições da marcha.
Estudos comparativos foram realizados entre o sistema NeuroKinect inalterado e três novos sistemas, que utilizam o Nuitrack SDK com três sensores RGB-D diferentes: Microsoft Kinect v2, Orbbec Astra e Intel RealSense D415. Para isso, foram realizadas aquisições com 22 sujeitos, incluindo duas modalidade de movimento distintas: permanecer imóvel e caminhar. Uma análise do ruído, precisão da deteção, amplitude de movimento, taxa de frames extraídos e número de passadas identificadas revelou que uma solução genérica de rastreamento de movimento corporal, como o Nuitrack SDK, afeta negativamente o desempenho do sistema, demonstrando que deve haver um compromisso entre a generalidade de um sistema e o desempenho desejado.
Human motion, specifically human gait, has proven to be directly related to cognition, being a complex interaction between the central nervous system, nerves and muscles. Thus, the quantifi-cation of gait and its variability is of utmost importance since they are indicators of the overall health of an individual.
The prevalence of movement related neurological diseases has raised, mainly in the elderly population, being one of the main causes of gait alterations. Thus, the clinical diagnosis and follow-up of patients suffering from neurological diseases is frequently performed by visually inspecting their gait patterns, which is subjective and dependent on the doctors’ opinion. An automatic system that assesses and quantifies gait would bring significant improvements in clinical practice, aiding doctors to prescribe more adequate pharmacological treatments and enabling an objective analysis that detects even small variations.
NeuroKinect is the current solution proposed by BRAIN Laboratory, at the INESC TEC, which is a Kinect-based gait analysis system that acquires data from patients walking forwards and backwards, cyclically, then processes data, quantifying a total of 48 gait parameters, and, fi-nally, provides immediate feedback, with the generation of a report, which includes alerts to help the interpretation of the results. With the discontinuation of the Microsoft Kinect sensor arose the goal of this work, to adapt the current NeuroKinect system to more recent technologies.
Thus, new depth cameras and body-tracking tools were studied and a new body-tracking inter-face was developed, Nui-MTracker, relying on Nuitrack SDK and compatible with several RGB-D sensors. This software application incorporates the NeuroKinect system, enabling the visualiza-tion and saving of the data during gait acquisivisualiza-tions.
Comparative studies were performed between the NeuroKinect system with the new ones, relying on Nuitrack SDK and three different RGB-D sensors: Microsoft Kinect v2, Orbbec Astra and Intel RealSense D415. For that, data from 22 subject were acquired, including two distinct tasks: standing still and gait. An analysis of the noise, positional tracking accuracy, range of motion, frame rate and number of detected strides, revealed that a generic body-tracking solution like Nuitrack SDK impacts negatively the performance of the system, showing that there has to be a compromise between the generality of the system and its desired performance.
Em primeiro lugar gostaria de agradecer a todos aqueles que me ajudaram no desenvolvimento desta dissertação, com principal ênfase ao meu orientador, o Professor João Paulo Cunha, por me ter permitido dar continuidade a um projeto já iniciado, e por me ter incentivado a continuar, sempre com ideias e discussões que me ajudaram bastante a estruturar o meu trabalho, de forma autónoma e independente. De agradecer também a todos os restantes elementos do BRAIN, sem-pre acolhedores e dispostos a ajudar e colaborar nos testes práticos. Do mesmo modo, a toda a gente que contribuiu nas aquisições de marcha, sejam eles colaboradores do INESC ou da FEUP, um muito obrigado.
Cinco anos após ter ingressado a maior aventura da minha vida, fica um sentimento de gratidão imenso a esta faculdade, um local que foi mais que uma casa para mim, onde para além de todos os conhecimentos técnicos que adquiri, cresci como pessoa e conheci pessoas que tenho a certeza que levarei para a vida. Foram sem dúvida os melhores anos da minha vida e a despedida deixa um misto de sentimentos inexplicável.
Sempre muito importantes no meu percurso, não pode faltar a minha família, pais, irmã e avós, que me proporcionaram sempre tudo e suportaram toda a minha mudança para o Porto. Ao meu namorado que sempre me motivou mesmo quando tudo parecia impossível e aos meus amigos da FEUP, companheiros de todos os momentos, confidentes e conselheiros, um muito obrigada!
Joana
Abstract iii 1 Introduction 1 1.1 Context . . . 1 1.2 Motivation . . . 2 1.3 Main Contributions . . . 3 1.4 Thesis Outline . . . 3
2 Human Motion and Neurological Diseases 5 2.1 Clinical Motion Assessment . . . 5
2.1.1 Gait Analysis . . . 6
2.2 Neurological Disorders . . . 8
2.2.1 Pathological Gait . . . 8
2.2.2 Parkinson’s Disease . . . 11
3 Human Motion Capture 15 3.1 Motion Capture Solutions . . . 15
3.2 Vision-based MoCap Work-flow . . . 17
3.3 Vision-based Mocap Systems . . . 18
3.4 RGB-D Sensors . . . 19
3.4.1 Microsoft Kinect v2 . . . 20
3.4.2 Stereolabs Zed . . . 22
3.4.3 Orbbec . . . 23
3.4.4 Intel RealSense . . . 24
3.4.5 ASUS Xtion Pro . . . 25
3.4.6 Mobile In-depth Cameras . . . 26
3.5 Body Tracking Tools . . . 27
3.5.1 Kinect SDK . . . 28
3.5.2 Nuitrack SDK . . . 30
3.5.3 Orbbec Body Tracking SDK . . . 31
3.5.4 OpenPose . . . 32
3.5.5 Selected Motion Capture Technologies . . . 32
4 Nui-MTracker: a NeuroKinect Evolution 35 4.1 NeuroKinect . . . 35
4.1.1 KinecTracker(KiT) Application . . . 36
4.1.2 Kinect Motion Analyzer(KiMA) Application . . . 39
4.1.3 Gait Parameters Extraction Pipeline . . . 41 vii
4.1.4 Report Generator . . . 47
4.2 Nui-MTracker Software Application . . . 48
5 Multiple RGB-D Sensor Comparative Study with Nui-MTracker 57 5.1 Preliminary Phase Protocol . . . 58
5.1.1 Test 1: Optimal Camera Setup . . . 58
5.1.2 Test 2: Maximum Frame Rate . . . 59
5.1.3 Test 3: Interference between systems . . . 60
5.1.4 Short Verification . . . 62 5.2 Comparative Study . . . 64 5.2.1 Participants . . . 64 5.2.2 Experimental Setup . . . 64 5.2.3 Experimental Protocol . . . 65 5.3 Data Analysis . . . 66
5.3.1 Stand Still Task . . . 66
5.3.2 Gait Task . . . 69
6 Conclusions and Future Work 73
A Interference between systems 75
B Short verification of the systems 81
2.1 Representation of a normal gait cycle, with insights on swing and stance as well
as single and double support phases. . . 7
2.2 Basic terminology describing a gait cycle. . . 8
2.3 Graphical representation of characteristic step sequence in gait disorders: a) nor-mal gait; b) spastic gait (paraparetic); c) cerebellar ataxic gait; d) parkinsonian gait. . . 9
2.4 Exemplification of some deformities that frequently affect PD patients and are caused by muscular rigity: a) striatal hand deformity; b) striatal foot deformity; c) camptocormia, with extreme flexion of the thoracolumbar spine. . . 12
2.5 Representation of PD patients’ posture and associated tremor during gait. . . 13
3.1 General workflow of vision-based motion capture. . . 17
3.2 Methodologies for depth estimation. a) Passive Stereo; b) Active stereo; c) Struc-tured light; d) Time of flight. . . 20
3.3 The Microsoft Kinect v2 camera, developed for Xbox 360. . . 21
3.4 Kinect depth sensing. a) IR dots seen by IR detector; b) Depth map in gray values. 21 3.5 Stereolabs Zed stero camera. . . 22
3.6 Orbbec Persee camera. . . 23
3.7 Orbbec Astra sensor. . . 24
3.8 Intel RealSense Depth Camera D415 (above) and D435 (below). . . 25
3.9 Asus Xtion Pro Camera. . . 25
3.10 IPhone X front bar included sensors. . . 26
3.11 Framework for development of RGB-D sensor based applications. . . 27
3.12 Kinect’s skeletal tracking pipeline. a) Depth image; b) Inferred body parts; c) Hypothesized joints; d) Tracked skeleton. . . 28
3.13 Representation of body joints tracked by the Microsoft Kinect v2. . . 29
3.14 Architecture of the Kinect SDK v2 . . . 29
3.15 Representation of joints tracked by Nuitrack SDK. . . 30
3.16 Nuitrack SDK Architecture. . . 31
3.17 Representation of joints tracked by Orbbec Body Tracking SDK. . . 31
3.18 OpenPose, detecting body, foot, hand, and facial keypoints in real-time, relying on a single RGB camera. . . 32
3.19 Simple interface developed on C#, relying on Astra body tracking SDK and Orbbec Astra. . . 34
4.1 Schematic representation of NeuroKinect system, a Kinect v2 based motion track-ing system. . . 35
4.2 Use Case Diagram of KiT (KinecTracker) application. . . 37 ix
4.3 First graphical user interface of KiT Application. . . 37
4.4 Main graphical user interface of KiT (KinecTracker) Application. . . 38
4.5 Preferences menu graphical user interface of KiT Application . . . 38
4.6 Format of the output files obtained by NeuroKinect. . . 39
4.7 Use case diagram of KiMA (Kinect Motion Analyser application. . . 40
4.8 Main graphical user interface of KiMA application. . . 40
4.9 Representation of the acquiring setup, its range of distance and the orientation of the tracked variables. . . 41
4.10 3D body positions processing, including data filtering, automatic segmentation in gait cycles, segmentation in gait cycles, motion parameters computation and results storage in a CSV file. . . 42
4.11 Graphical representation of a subject’s center of mass distance to the sensor during an acquisition with 9 gait cycles. The blue line represents the median, while the red lines represent the thresholds. . . 43
4.12 Graphical representation of heel strike detection and segmentation in strides . . . 44
4.13 Graphical representation of inflection point detection for swing and stance phases identification. . . 46
4.14 Arm swing representation in the height plan (X-Z axis). . . 46
4.15 Example of a motion quantification report generated by NeuroKinect. . . 47
4.16 Schematic representation of NeuroKinect system, evolved by Nuitrack to a multi-RGB-D adaptable system, including Nui-MTracker, a new developed interface. . 48
4.17 Use case diagram of Nui-MTracker application. . . 49
4.18 Activity diagram of Nui-MTracker application. . . 50
4.19 First graphical user interface of Nui-MTracker application: saving new subject. . 51
4.20 First graphical user interface of Nui-MTracker application: selecting previously saved subject. . . 52
4.21 First graphical user interface of Nui-MTracker application: subject information being displayed after selection. . . 53
4.22 Warning dialog of NUI-MTracker application, displayed when the camera is not detected. . . 53
4.23 Main graphical user interface of Nui-MTracker application: starting an acquisition. 54 4.24 Main graphical user interface of Nui-MTracker application: stopping an acquisition. 54 4.25 Preferences dialog of Nui-MTracker application. . . 55
5.1 Main phases of the experimental protocol . . . 57
5.2 Physical configuration of the four RGB-D based acquiring systems. . . 58
5.3 Comparison between the structure of the skeletons tracked by Kinect SDK (left) and Nuitrack SDK (right). . . 63
5.4 Experimental setup including four portable computers, four tripods and four cam-eras. Three distances from the cameras were marked on the floor, with tape, cor-responding to 1 meter, 3 meters and 5 meters. . . 65
5.5 Graphical representation of the difference between estimated position by new sys-tems K2, OA and RS and the existing system, K1, with tendency lines. . . 69
5.6 Center of mass distance to the sensor of the same subject in three different systems: K1, OA and RS. . . 70
5.7 Distance between ankles of the same subject and same walk in three different systems: K1, OA and RS. . . 70
A.1 Graphical representation of noise measurement by joint for Kinect v2 with Kinect
SDK. . . 76
A.2 Graphical representation of noise measurement by joint for Kinect v2 with Nu-itrack SDK. . . 77
A.3 Graphical representation of noise measurement by joint for Orbbec Astra with Nuitrack SDK. . . 78
A.4 Graphical representation of noise measurement by joint for RSD415 with Nuitrack SDK. . . 79
B.1 Report generated by the system K1, for short validation test. . . 82
B.2 Report generated by the system OA, for short validation test. . . 83
2.1 Characterization of pathological gait patterns and associated medical conditions. . 10
2.2 Non-motor symptoms of Parkinson’s Disease. . . 11
2.3 Studies on straight line walking after withdrawal of medication in patients suffer-ing from Parkinson’s Disease. . . 14
3.1 MoCap approaches description and examples. . . 16
3.2 Compatibility of Nuitrack SDK with different RGB-D sensors for different opper-ating systems. . . 30
3.3 Compilation of the depth sensors described and its characteristics. . . 33
4.1 Gait Features extracted for the motion quantification report generation . . . 45
5.1 Results of range of motion measurements for NeuroKinect system. . . 59
5.2 Results of range of motion measurements for each system relying on Nuitrack SDK. 60 5.3 Results of the frame rate obtained for each system, together with the number of extracted frames. . . 60
5.4 Average noise values measured in each acquisition, for each system (in millimeters). 62 5.5 Demographic information on the experiment participants. . . 64
5.6 Noise measurement results (mean and standard deviation), with the four systems acquiring data while the subject stands still (in millimeters). . . 67
5.7 Difference between the estimated position of joints using K1 and the new systems, K2, OA and RS (in centimeters). . . 68
5.8 Measured parameters during gait acquisitions of 19 subjects, for three systems: K1, OA and RS. . . 72
6.1 Characteristics of the Nuitrack SDK based system with Orbbec Astra (OA) and Intel RealSense D415 (RS), comparing with the NeuroKinect system. . . 74
A.1 Noise measurement by joint for Kinect v2 with Kinect SDK, in millimeters. . . . 75
A.2 Noise measurement by joint for Kinect v2 with Nuitrack SDK, in millimeters. . . 76
A.3 Noise measurement by joint for Orbbec Astra with Nuitrack SDK, in millimeters. 77 A.4 Noise measurement by joint for RSD415 with Nuitrack SDK, in millimeters. . . 78
C.1 Gait parameters extracted from 19 gait acquisitions, using the NeuroKinect system. 86 C.2 Gait parameters extracted from 19 gait acquisitions, using Nuitrack-based system with the Orbbec Astra sensor. . . 87
C.3 Gait parameters extracted from 19 gait acquisitions, using Nuitrack-based system with the Intel RealSense D415 sensor. . . 88
BMI Body Mass Index
COM Centre Of Mass
CSV Comma-Separated Values fps Frames per second
IR Infrared
KiMA Kinect Motion Analyser
KiT KinecTracker
MDS-UPDRS Movement Disorder Society - Unified Parkinson’s Disease Rating Scale MoCap Human Motion Capture
NUI Natural User Interface OS Operating System PD Parkison’s Disease RSD415 Intel RealSense D415 SDK Software Development Kit SL Structured Light
STD Standard Deviation ToF Time of flight
Introduction
1.1
Context
During the past decades, the study of locomotion has shown that the characterization of gait and its variability are indicators of the severity of a disease. As the body movement is a complex interaction between the central nervous system, nerves, and muscles, it becomes important to know what disorders affect gait, how they affect it and how the measurement of kinetic and kinematic features is performed to detect significant gait variations [1].
Walking is a daily, yet complex, activity. The patterns of walking depend on many factors, such as age, personality, mood, and sociocultural factors. As age advances, gait disorders become more frequent. Gait impairment jeopardize both quality of life and personal independence. Thus, its monitoring and quantification are utterly important to perceive the overall health of an individual [2,3,4,5].
Neurological conditions may also be a cause for gait disorders, for instance, Parkinson’s Dis-ease, Huntington’s DisDis-ease, Tourette’s Syndrome, several neuropathies, chorea, tremor and essen-tial tremor, among many others [6]. Quantification of human motion, such as gait analysis, is considered a useful tool in clinical environments for diagnosis, evaluation, and treatment of these conditions [7]. Gait analysis becomes useful for monitoring the patient’s recovery also in reha-bilitation situations, namely in post-operative, consequence of trauma by impact, broken bones, muscle atrophy, and post-stroke, etc.
In spite of the advances in the understanding of how the analysis and objectification of gait may improve patient care, patients’ evaluation is still usually made relying on direct observation, making this a subjective process [8, 9]. Therefore, an user-friendly and practical system that automatically measures gait features, analyses them and gives immediate feedback, gains clinical relevance, making it possible to keep the patient’s registry objective.
1.2
Motivation
As the diagnosis and follow-up of neurological diseases are subjective, the evaluation of a patient in two different moments does not depend on the same factors, raising the clinical need of a system that extracts relevant gait features objectively, allowing a more accurate analysis.
Advancing the way patients are evaluated and the understanding of their progression makes the treatment and the prescription of drugs more adequate, which may lead to significant im-provements in patient care and, consequently, quality of life. Moreover, an automated system that quantitatively assesses human motion enables to keep the evaluation criteria constant, even if the medical doctors are not the same.
The development of a system for this purpose has been proposed previously in our Brain Lab-oratory, at the INESC TEC. This system, named NeuroKinect, has been continuously advanced [10,11,12,13,14], consisting of a vision-based gait analysis system to be used in the assessment of patients suffering from neurological diseases, with emphasis on Parkinson’s Disease (PD), rely-ing on Microsoft Kinect v2, a RGB-D sensor, which tracks body joints. In summary, the existrely-ing system includes four main blocks:
• Kinect motion capture: in a corridor, the patient walks towards the Microsoft Kinect v2 depth sensor and in the opposite direction.
• Human motion tracking: through a body-tracking tool, Kinect SDK, the three-dimensional position of each body joint are identified, tracked and saved;
• Gait parameters extraction: a file containing the information of each body position along time is processed, in order to quantify 48 gait parameters;
• Feedback: an automated report is generated, which includes the numerical average values of each feature, as well as the standard deviation, graphical representations of stepping and asymmetry, and alerts indicating whereas the values are within the standard control population.
Our goal has been updating this system in order to get a robust solution to implement in clinical practice, allowing to keep track of the patient’s disease progression, not only in hospitals but also from home, as a telemedicine solution.
Bearing this in mind, the recent discontinuation of the Kinect sensor is a major limitation for our system, making it obsolete in a few years. Thus, it becomes of utmost relevance to update it for more innovative technologies as other depth sensors and other body-tracking tools. The Kinect sensor is already noisy by nature, becoming also important to study the noise of the new systems and with the expectation of minimizing it.
1.3
Main Contributions
By the end of this dissertation work, we intend to find a solution that replaces the Kinect sensor and integrates the existing system. Thus, a study on RGB-D sensors and body-tracking tools was performed. The selection of the technologies to be implemented is based on the similarities with Kinect, since we want to integrate them into the existing system, and on the combination of good technical specification with an affordable price [14].
The comparative studies between Nuitrack SDK with different RGB-D cameras and the un-changed NeuroKinect system is also an important contribution, which may help researchers in the scientific community to equally evolve their Kinect-based systems or to have a more informed choice when developing a new vision-based system.
For the new systems studied, a new software application was developed, Nui-MTracker, which helps to visualize data during acquisitions and to save the acquisitions’ data in the same format as NeuroKinect does. This interface is now compatible with several RGB-D sensors: Intel RealSense D415, Orbbec Astra, Microsoft Kinect and ASUS Xtion (not tested). However, Nuitrack is still a recent company, continuously updating its software, working on improving its accuracy, using several cameras at the same time and increasing the number of compatible cameras. Bearing this, the developed interface may be very helpful for future tests and for different applications.
Finally, a new data-set was created, acquiring two modalities of movement, gait and standing still, with four different systems at the same time. Data from 22 subjects were acquired with the existing solution, Kinect and Kinect SDK, and three new solutions, relying on Kinect, Orbbec Astra and Intel RealSense D415, with Nuitrack SDK. This data-set may as well be useful for future tests.
1.4
Thesis Outline
After this introductory chapter, five more chapters are presented. Chapter2gives an overview of human motion quantification and its clinical relevance regarding neurological movement condi-tions. Chapter3gives an overview of human motion capture techniques, highlighting vision-based solutions, as well as a study of several cameras and several body-tracking tools. Then, in Chapter 4the existing system is described, together as the new interface developed, with insights of how its integration works. Chapter5includes the protocols of acquisitions with volunteers, the differ-ent tests performed and the analysis of the obtained results. Finally, in Chapter6conclusions are drawn and some suggestions for future work are made.
Human Motion and Neurological
Diseases
This chapter presents an overview of human motion and the relevance of its quantification in the assessment of neurological diseases. A wider study on gait analysis was performed, as it is the most common way to assess patients suffering from neurological diseases in clinical practice. In order to identify relevant motion parameters to be quantified, a general review on neurological diseases and its motor complications is also presented, with highlights on Parkinson’s Disease (PD) and some pathological gait characteristics associated with neurological disorders [15].
2.1
Clinical Motion Assessment
As general population is living longer, assuring health and social care becomes more important, as well as quantifying and continuously assessing people’s health [16]. Although most of the elderly population remains healthy and active, an increasing portion encounter frailty and musculoskeletal mobility disorders [17], whose causes complicate walking, standing and performing daily tasks, leading to a significant reduction in quality of life and the enhancement of the risk of falling. Human mobility is an indicator of health and well-being, therefore its quantification allows the evaluation of a patient, supporting therapeutic diagnosis and treatment. [16,18]
Nowadays, most of the clinical evaluation of a subject’s movement is still made relying on direct observation, through a set of standardized motor tests, such as walking back and forth, sitting and standing successively, balancing in one leg, performing small hand movements, among others. During the evaluation, the clinician checks on the patient’s stability, posture, coordination, ease of movement and classifies the patient, resorting mainly to numeric scales. However, the assessment of that each clinician depends on personal traits, skills and years of experience, being a subjective and qualitative task [16,18,19,20].
Quantifiable tests and standardized rating scales exist to support the medical examination. Regardless of the existing standardized and generalized rating scales not being able to reach all
neuromuscular disorders, there are condition-based ones that facilitate body movement charac-terization. For instance, for patients suffering from Parkinson’s Disease, there is the Movement Disorder Society - Unified Parkinson’s Disease Scale (MDS-UPDRS), a stratified scale used to classify the patient’s stage of disease with 5 levels of classification, being normal, slight, mild, moderate and severe, evaluating not only motor symptoms, but also mental behavior, mood, abil-ity to perform daily activities and others, from a 0 to 5 point scale. This scale is considered semi-quantitative since it is still based on a doctor’s opinion [21,22,19]
As human mobility functions as a health indicator, quantifying motion and detecting anoma-lies, even small ones, may help to predict neuromuscular disorders in early stages, enabling early medication prescription as well as rehabilitation planning, potentiating better outcomes. [21, 23, 16, 24]. These small changes in normal patterns of motion are hard to detect through visual analysis, which justifies the clinical need of an automatic and computerized system that performs a quantitative assessment of a patient, with proper resolution. In the already available systems the captured and processed motion information may be as simple as the position of the body in space or as complex as the deformations of the face and muscles[8].
In the past years, several technological solutions have been developed, aiming to define the normal and pathological patterns of movement and to improve the existing ways of diagnosis and treatment, making it more objective and less variable [25]. Even though computers are more pow-erful, algorithms more complex and the sensing options more precise, small and comfortable to the users, body movement quantification systems are still a challenge, due to the physical complexity of the human body, which has multiple degrees of freedom and numerous segments interconnected by movable joints and other variable factors that affect motion performance as mood and general well-being. Making the process even more challenging, there is the lack of rigor when collecting data, especially in clinical non-controlled environments. Besides, for systems relying on inertial and magnetic sensors, their attachment to the body may be a time-consuming and invasive proce-dure. Bearing this, motion quantification approaches are still relatively rare in hospitals’ current procedures [8,25].
2.1.1 Gait Analysis
Walking is a complex movement that requires two essential abilities: locomotion and equilib-rium. Locomotion includes the initiation and maintenance of rhythmic stepping, and equilibrium includes the capacity to maintain balance and an upright posture [26,4]. During gait, the whole body moves, being a complex interaction that involves the central nervous system, nerves, and muscles. Thus, gait analysis is relevant for motion quantification assessment, enabling the learning of lower and upper limb kinetics and kinematics, such as preferred walking speed, posture, bal-ance, arm swing, ease of movement and stride-to-stride variability, which reflects gait instability and risk of falling [27,28,2,8]. In fact, most of the currently performed clinical tests to evaluate cardiovascular, visual and cognitive impairments are accompanied by gait analysis [19,29].
Furthermore, it has been demonstrated that there is a strong relationship between gait and cognition [19]. Daily walking requires planning, choosing the best route, identifying obstacles
and continuously interacting with the environment. A normal and proper walking relies, not only on sensorimotor systems, but also on the interaction between the decision-making and cognitive dimensions, such as visuospatial perception and orientation, and on affective dimension, such as mood and cautiousness. Bearing this, some mental conditions, such as dementia, depression, and anxiety, for instance, induce gait alterations, which may be consequently identified at early stages with gait analysis [4,2].
Figure2.1graphically represents one cycle of the normal pattern of walking.
Figure 2.1: Representation of a normal gait cycle, with insights on swing and stance as well as single and double support phases. Adapted from [2].
Initiating gait requires an upright body posture and functioning postural reflexes, to maintain balance and stability whilst walking. A stride is considered the interval between two consecutive heel strikes of the same foot. In the beginning, one leg is raised and directed forward, flexing the hips and knees, while the other one remains in the stance phase, supporting on the ground. As the heel of the swinging leg places on the ground, the body’s weight, centered in the stance leg, is gradually shifted to the other leg. This moment, when both feet are placed on the ground, is called double limb support phase. Then, the initial stance leg lifts and moves forward, until a new heel strike occurs, starting the swing phase of the opposite leg again. Meanwhile, the body keeps an upright posture, the shoulders and pelvis remain relatively at the same level and each arm swings in the opposite direction of the correspondent leg [2].
In this kinf of movement, it is possible to measure some spatiotemporal variables, such as walking speed, cadence (steps per minute), stride width, step length, and stride length, as repre-sented in Figure2.2. As some authors consider that the use of stride-to-stride variability provides more reliability for the assessment of the patients’ ability in generating consistent and rhythmic steps than using only average values, it is of utmost importance to measure not only the average values but also the variability [13,30].
Nowadays, gait analysis, as a subjective measurement performed by clinical doctors, or as an objective measurement performed automatically, is not commonly used for medical diag-nosis alone, since usually more metrics are needed to avoid amiss diagdiag-nosis. Nevertheless, it
Figure 2.2: Basic terminology describing a gait cycle. Adapted from [2].
is frequently requested to quantify the mobility state of a medical disorder and determine the neuromuscular-skeletal contributions to that state. In fact, gait analysis is now considered a use-ful clinical tool, providing quantitative information that aids medical prescriptions and assesses patient’s treatment outcomes [1].
2.2
Neurological Disorders
Neurological disorders affecting gait, balance and posture are both debilitating and common [24]. Recognizing gait disorders may help to characterize and diagnose patients on the course of neu-rological diseases and so, a study on the gait pathologies and associated neuneu-rological conditions was made.
The most common neurological movement disorder is Parkinson’s Disease, affecting seven to ten million people worldwide [31]. It is the second most common degenerative disease of the central nervous system, following Alzheimer’s disease, having its incidence and prevalence on the rise along with changing population demographics, which gets increasingly aged [32,33,15].
Taking into account the substantial prevalence of PD, a deeper research on this disorder and its broad spectrum of motor manifestations was made, to better understand which motor varia-tions would be relevant to measure. Furthermore, since PD caused motor alteravaria-tions are spread to the whole body, a system developed with the purpose of evaluating this disease is still applicable to other neurological conditions, such as Huntington’s Disease, Sydenham’s chorea, several neu-ropathies, Tourette syndrome, cerebral palsies, several types of sclerosis, stiff person syndrome and even autism [8].
2.2.1 Pathological Gait
Since most gait abnormalities are the result of neurological diseases [26], it is important to identify which gait disorders are more frequent and how they correlate with neurological diseases.
Abnormal gait has two distinct sources, neurological or non-neurological, the latter mainly due to arthritis, cardiac disease, chronic lung disease, and peripheral vascular disease [34, 35, 26]. Regarding neurological gait disturbances, they have revealed higher prevalence in the elderly population, often causing immobility, risk of falling and consequently augmented mortality [19].
As a consequence of these disturbances, gait may manifest distinct variations, namely in flu-ency and speed of movement. This phenomenon is called dyskinesia [8]. There are three types of dyskinesia:
• Hyperkinesia: Increased muscular activity that results in excessive and involuntary move-ments.
• Bradykinesia: Slowness of movement and impaired ability to perform voluntary move-ments.
• Hypokinesia: Partial or complete absence of voluntary movements.
Dystonia is also a movement abnormality characterized by prolonged and uncontrollable mus-cle contraction, causing involuntary movements, repetitive movements and leg/foot abnormal pos-tures, typically with inversion, plantar flexion and tonic extension of the big toe [8,2].
There are several pathological gait patterns that may be attributed to neurological conditions. An exemplification of some gait disorders categories, both as the associated conditions, are repre-sented in Table2.1. Also, Figure2.3shows examples of step sequence variations of three patholog-ical gait patterns: spastic, cerebellar ataxic and parkinsonian, together as a normal representation of step sequence.
Figure 2.3: Graphical representation of characteristic step sequence in gait disorders: a) normal gait; b) spastic gait (paraparetic); c) cerebellar ataxic gait; d) parkinsonian gait. Adapted from [2].
Table 2.1: Characterization of pathological gait patterns and associated medical conditions. Gait disorder Characterization Medical Conditions Reference Steppage High stepping gait, attempting to
lift the legs high enough
Neuropathy [36] Waddling Drop in pelvis while walking,
unilateral or bilateral,
Myopathy, muscular dystrophy
[36,2] Choreiform Slow walking, excessive
in-voluntary movements affecting knee and hip flexion, irregular, dance-like and broad-based, hy-perkinesia
Sydenham’s chorea, Hungington’s Disease
[36,2]
Spastic Weakness on the affected side, arm flexed, adducted and internally rotated, slow gait, wide-based, asymmetrical, legs slightly bent at the hip and adducted, circumduction of affected legs during walk (Fig. 2.3-b) Hemiparesis, para-paresis [36,2] Cerebellar ataxic
Broad-based, lack of coordi-nation, unsteady and insecure standing, jerky movements asso-ciated with risk of falling, cau-tious walk
Hereditary cerebellar atrophy, chronic alco-holism, multiple scle-rosis (Fig.2.3-c)
[36,2,8,19, 37]
Sensory ataxic Deficits in proprioception, cau-tious, worsening without visual input, uncoordination, fear of falling, short steps, slow walking
B12 deficiency, myelopathy, tabetic neurosyphilis, un-controlled diabetes, polyneuropathy [36, 2, 19, 37]
Parkinsonian Muscular rigidity, rest tremor, impaired postural stability, slow walking, short steps, shuffling gait, bradykinesia
Parkinson’s Disease, drugs adverse events, dementia (Fig.2.3-d)
[36,2,8,37]
Apraxic Difficulty in gait initiaton, short steps, shuffling gait, with ten-dency to fall backwards
Stroke sequelae, front lobe tumours, cere-brovascular disease
[19,37]
Vestibular Asymmetric, chronic imbalance resulting from vestibular loss, insecurity while standing and walking, easier running than slow walking
Exposure to ototoxic drugs
2.2.2 Parkinson’s Disease
Parkinson’s Disease is a progressive neurological disorder characterized by a large number of motor and non-motor consequences, that affect patients’ quality of life. These are caused by the failure of dopamine synthesis, in the brain, which is mainly responsible for motor control, emotional responses and the ability to feel pleasure and pain [15,38,33,32].
Regardless of the absence of a cure, the treatment options include pharmacologist treatment and surgery. Some of the crippling signs of PD are, to some extent, reversible upon the admin-istration of the dopamine percursor, Levodopa [39]. While PD is not a direct cause of death, its complications may be severe and lead to mortality after 7-14 years [32]. Despite the general effec-tiveness of dopaminergic drug therapy, gait deficits in PD patients are typically resistant to phar-macologic treatment. In addition, prolonged drug intake may also be associated with decreased responsiveness to medication [32,38].
Table2.2details the most common non-motor consequences of the disease, such as sensory, cognitive and neuropsychiatric disorders, sleeping disturbances and autonomic dysfunction.
Table 2.2: Non-motor symptoms of Parkinson’s Disease, adapted from [32,38]. Sensory Partial or complete loss of smelling and tasting sensations,
abnormal dermal sensation throughout the body, pain in the limbs, shoulder, back (oral, thoracal, abdominal and genital pain may also occur).
Cognitive Slow thinking and processing of information (bradyphrenia), visuospatial dysfunction, impaired speech fluency and mem-ory impairments.
Neuropsychiatric Dementia, visual hallucinations and illusions, depression, anxiety, apathy, anhedonia (reduced ability to experience plea-sure), fatigue, other behavioural abnormalities.
Sleeping Fractionated sleep, REM behaviour disorder, vivid dreams, daytime drowsiness, restless legs syndrome.
Autonomic Dysfuntion Orthostatic hypotension, gastrointestinal alterations, constipa-tion, urinary and sexual dysfuncconstipa-tion, abnormal sweating, low blood pressure, seborrhoeic keratosis.
As for the motor symptoms, there are four cardinal symptoms required to diagnose and evalu-ate PD: bradykinesia, tremor, muscular rigidity and postural instability [15,38,32,33]. Ordinarily, the diagnosis relies in the detection of the combination of, at least, two symptoms, usually exclud-ing postural instability that does not manifest until later stages of disease. [40]
• Bradykinesia is the most characteristic clinical feature of PD patients, and refers to slow-ness of initiation and executing voluntary movements, with a progressive reduction in their speed and amplitude. It often leads to difficulties performing daily tasks, such as feed-ing, cutting food, hygiene, dressfeed-ing, shavfeed-ing, and others, usually by losing the ability to perform fine motor movements. Other consequences of bradykinesia are hypomimia (ex-pressionless face), handwriting difficulties, loss of spontaneous movements and gesturing,
dysphagia (impaired swallowing), sialorrhoea (excessive drooling) and decreased blinking [38,32,15].
• Tremor at rest occurs in around 80% of the affected population and is the most easily recognized symptom. It usually manifests in the distal part of an extremity, initially occur-ring unilaterally but progressively affecting both sides of the body, frequently in the hands (supination-pronation tremors), lessening during sleep or when the affected body part is ac-tively in use. Tremor at rest may also show in the lips, chin, jaw and legs. Tremor is not that directly related to dopamine reduction as bradykinesia and rigidity, hence, the tremor does not accentuate with disease progression. Some patients may also suffer from action tremor or postural tremor [15,38,32,40,41].
• Muscular rigidity is characterized by the increasing resistance to the movement, resulting in a decreased range of motion. It may also be accompanied by the cogwheel phenomenon, which is very common in patients with PD, particularly when associated with underlying tremor. Rigidity often leads to muscle or joint pain [15, 38, 32, 40, 41]. Moreover, pos-tural deformities are associated with muscle rigidity. For instance, axial rigidity (in the neck or trunk) may lead to abnormal postures, such as enterocolitis and scoliosis. Other consequences may be striatal limb deformity (usually in hands or feet), neck flexion and camptocormia (extreme flexion of the thoracolumbar spine) [38], as shown in Figure2.4.
Figure 2.4: Exemplification of some deformities that frequently affect PD patients and are caused by muscular rigity: a) striatal hand deformity; b) striatal foot deformity; c) camptocormia, with extreme flexion of the thoracolumbar spine. Adapted from [38].
• Postural instability usually occurs in later stages of the disease, due to loss of postural reflexes. Postural instability, together with freezing of gait, is the main cause of falls in PD
patients. Furthermore, 70% of the elderly have impaired postural reflexes, which makes it misleading when used in diagnosis [38,15,40].
Focusing on gait, PD is characterized by a shuffling gait, comprising stooped posture, short steps, and reduced arm swing [38], resulting in considerable difficulties for patients to walk or turn whilst simultaneous performing motor or cognitive tasks [42]. A representation of the stooped posture of PD patients during gait, as well as the tremor associated is illustrated in the Figure2.5.
Figure 2.5: Representation of PD patients posture and associated tremor during gait. Adapted from [43].
In a review by Morris et al [42], the gait features significantly affected by PD were identified. The information was compiled and adapted in Table2.3. In fact, most of the gait alterations iden-tified in the experiments are quantifiable variables which must be implemented when developing a computerized gait analysis system.
Besides, other PD gait disturbances may include blocking, hesitancy and gait festination, where steps become progressively smaller and faster, which leads to the loss of balance and falling. Motor blocking, also called freezing of gait, is a form of akinesia, that only affects around half of the PD population, and commonly happens to the legs during walking, also increasing the risk of falling [32].
Table 2.3: Studies on straight line walking after withdrawal of medication in patients suffering from Parkinson’s Disease. Adapted from [42].
No. of PD patients
Withdrawal time
Findings Reference
44 not stated Reduced arm swing and trunk rotation; forward stooped posture; slow and shuffling gait
[44] 21 (12
fe-male)
12h Decreased stride length and speed; cadence within nor-mal range
[45]
14 12h, 14h,
16h, 18h
Stride length and speed reduced; double support phase duration within normal values
[46] 20 (11
fe-male)
12h Decreased stride length and walking speed; rhythmicity within normal; increased variability in all parameters during "off" phase of medication
[47]
7 12h Stride, step and double limb support time longer [48] 10 (4
fe-male)
24h decreased walking speed and stride length; moderate stride asymmetry; decreased cadence
[49] 16 (7
fe-male)
12h Walking speed and stride length reduced; cadence in-creased
[50] 1 elderly
female
12h Reduced speed, stride length, cadence, angular dis-placement and plantar-flexor power
[51] 11 (3
fe-male)
8h Decreased walking speed, cadence, stride length [52] 12 12h Decreased arm swing; increased arm swing asymmetry [53]
Human Motion Capture
A key point of human motion quantification systems is registering the body movement, by sensing the body, in a process called human motion capture (MoCap). This chapter presents a review of MoCap methods, focusing mainly on vision-based solutions.
3.1
Motion Capture Solutions
MoCap mainly refers to capturing large scale body movements, such as movements of the head, arms, torso and legs. Human motion capture may be as simple as tracking the subject as a single object or as complex as tracking it as an articulated motion of a high degree of freedom skeleton structure [54].
There are many alternatives when sensing the body. A sensor is defined as the on-body or external receiver or detector of a certain input. The source is considered as the origin of a signal or measured characteristic of the analyzed movement [8]. Bearing this, it is possible to categorize MoCap approaches, as described in the Table3.1.
Regarding the sensorization sources considered in the Table3.1, inertial, magnetic and me-chanic solutions rely on sensors attached to the body, which is not practical in clinical practice since the placement of the sensors in the body segments to track is time-consuming and makes them more prone to human errors. Besides, the subjects of interest are impaired and debilitated and may be uncomfortable for them to be under these conditions if it is not strictly necessary. Similarly, acoustic solutions are not indicated for their complexity and slow update rates. Be-sides, their set-up is not wireless, they may not be very accurate and need controlled and quiet environments [8].
With respect to optical solutions, even though multicamera systems relying on markers, such as Vicon [55], Optitrak [56] and Qualisys [57] provide high usability in biomechanics, high positional information precision, with errors inferior to 1mm, and even support for third-party equipment, like force plates or EMG acquiring, there are significant limitations when used in clinical environ-ment, namely the limited measurement volume and the need of using a specialized laboratory with fixed cameras, not being ideal for multi-scenario applications [25].
Source Sensor examples Description Disadvantages Inertial Gyroscopes,
ac-celerometers, magnetometers
Portable inertial systems; No need for large capture areas or complex calibration; Small in size; Relatively inexpensive.
Positional drift in time; Need of frequent calibra-tion; Noise and bias er-rors; Impractical use for long time periods; Place-ment on the body seg-ments to track (demands preparation).
Acoustic Ultrasonic pulses (with audio re-ceivers and an array of audio transmitters)
Determination of body position through time-of-flight (ToF), tri-angulation or phase-coherence; Placement on body segments or fixation in the measurement vol-ume.
Physical characteristics of sound limit the accuracy, update rate and tracking range; Disturbances by reflection and occlusion; expensive; Wired.
Magnetic Magnetic transmit-ters
Three perpendicular coils emit a a magnetic field when a cur-rent is applied; Strength of mag-netic field is proportional to the distance of each coil from the field emitter; Three-dimensional measurement.
Magnetic and electrical interference; Attached to the body segments (de-mands preparation).
Mechanical Rigid or flexible goniometers; plan-tar pressure distri-bution sensor
Used mainly in wearables such as exoskeletons and insoles; Given information is occlusion free; Direct tracking of body joints angles in exoskeletons case.
Attached to the body; Po-sitional drift with time; Need of frequent calibra-tion; Heavy and cumber-some.
Optical Marker-based mul-ticamera (Vicon, Optitrak, Qualisys)
Determination of position by us-ing multiple cameras to track joints or bony anatomical land-marks, using markers placed on the body (which may be re-flective or active/light emitting), which are perceived by the cam-eras; No wires or other elec-tronic equipment.
Need of preparation; Markers attached to the body or need of a full body suit; Occlusion problems. Vision-based Markerless video-cameras(RGB or RGB-D sensors)
Single camera acquiring color and depth image; Enables a higher freedom of movement with no requirement of spe-cific or complex scene neither body preparation (typically a clear ample space with good-lighting).
Need of distinct process-ing softwares for differ-ent applications; 3D body pose estimation may be computationally demand-ing; Occlusion problems.
Lastly, vision-based solutions do not require sensor units or markers attached to the body, being less time-consuming and invasive to the users. In the context of this work, it is important that the sensing solutions are adaptable to different scenarios, such as rooms of different sizes or corridors, which might be achievable with these vision-based solutions, unlike optical ones. Furthermore, they ought to be inexpensive, as well as easy and quick to set-up, so they might be applicable in any clinic, even smaller ones. These characteristics make it also a good option targeting its application in ambulatory scenarios, shortening the distance between the doctors and the patients, allowing them to make assessments more often, with no need of leaving the comfort of their home.
3.2
Vision-based MoCap Work-flow
With the introduction of video camera systems, the production of more detailed movement-related studies was achievable, with a growing amount of published articles on this subject [8]. A survey of more than 130 human motion capture papers published since 1980 [54], allows perceiving what is the general workflow of vision-based motion capture, which is shown in Figure3.1.
Figure 3.1: General workflow of vision-based motion capture. Adapted from [54].
With respect to the main stages considered in the figure above, the first one is initialization, mainly concerning the camera calibration and adaptation to the scene characteristics. In other words, it relates to the actions necessary to ensure the correct interpretation of the current scene, focused on the comprehension of the initial pose of the subject and the model representing it. This step may be performed manually, with an operator specifying key points/frames, or automatically, identifying the start pose and the background scenario. Most times, assumptions or standard rules are taken into account to facilitate the system initialization and to reduce system’s complexity. Some examples of these restrictions are: only one subject inside the workspace at a time, no occlusions, camera in stationary state, static background, constant lighting, among others [54].
Regarding the tracking phase, it is defined as establishing coherent relations of the subject between frames. Generally, tracking may be divided in three distinct aspects: identify the human figure from the background, transform the segmented images in a reduced representation (e.g. skeleton-like) and tracking the body position frame to frame.
Pose estimation may be a post-processing step in a tracking algorithm or an active part of the tracking process. This may be as simple as positioning the center of mass or as complex as determining the positions, orientation and width of each limb. Pose estimation may be model-free, where no a priori model is used, or resort on a model: indirectly, by using a reference to constrain and guide the interpretation of measured data, or directly, by maintaining a model and updating it with the tracked data. Commonly, in MoCap systems, a human model is used, such as a geometric model of the human body.
The last phase, recognition, is usually carried out to classify the captured motion as action types, such as walking or running, and to analyse the performed movements. It may be seen as the processing of the tracked data.
3.3
Vision-based Mocap Systems
In the past years, MoCap systems have been researched and developed, aiming to aid clinicians to monitor and evaluate patients suffering from neurological diseases. After understanding the concepts of human motion and exploring the possibilities to capture human motion, it is important to understand the functioning of the systems in our case, vision-based ones, in order to identify what are the optimal specifications the sensors need to have.
Even though motion quantification systems bring many advantages for clinical environments, it is still relatively rare to see them in practice, due to some limitations mainly related to the prohibitive costs in technology, amount of space needed, time-consuming sessions, lack of clinical interpretation and validation of the results [8]. Furthermore, to evaluate the patients’ progression, it is required they regularly go to the hospital, spending time and money on transportation and increasing the medical expenses. Thus, the possibility of using an "at-home" system would allow more frequent assessments, without the need of leaving home, reducing costs and improving the contact between doctors and patients. All of this may be possible using vision-based systems.
Thus, an ideal solution that fulfills the clinical needs found, follows certain specifications, listed below:
• Multi-scenario motion acquisitions: it does not require a specific room or previous prepa-ration, which makes it suitable to use anywhere, even at ambulatory scenarios. Thus, any room free from obstacles or moving people, as well as proper light and flooring conditions; • Easy and quick set-up: the set-up only requires planting the camera in a stable and steady place, inside an ample and non-crowded room or corridor, powering it and connecting it with the computer, usually through USB connection. Then, the patient only needs to place itself in front of the camera, and it is ready to start;
• Affordable: the used cameras do not need to be replaced at each session, requiring only the initial investment; ;
• Non-invasive tracking: no need for markers or sensors placed in body, for the body is tracked with a static camera monitoring motion from distance;
• Validation: some validation studies having as reference optical systems have been published in the past years [8,18,58,59];
• On-the-fly results: right after the end of the acquisition, the processing and analysis of the data are made available. Indications on whether the values are within the normal patterns or not should also be given, aiding its interpretation.
Thus, after motion capture, the data needs to be processed and presented to the user, legibly and intuitively. This process involves three main phases: segmenting the movement of interest, processing data, which includes the computation of the features to be extracted, and providing proper feedback to the user, helping the interpretation of the given values.
The more robust vision-based systems there are rely on the Microsoft Kinect, being the first depth sensor majorly explored in this context. However, the results found using the Kinect may be interchangeable for vision-based systems relying on other depth sensors. For instance, in [60], the interchangeability of Kinect and Orbbec Astra for gesture recognition is evaluated. Hence, it is assumed that it is possible to drive continuity between the existing RGB-D cameras, which led us to broader research on other solutions, described in the following section.
3.4
RGB-D Sensors
Vision-based systems may rely on the common color cameras (RGB sensors) or on cameras which also acquire in-depth data (RGB-D sensors). Having in mind the goal of this work, to evolve a Kinect-based system to a multi-camera adaptable one, it becomes crucial to detail the characteris-tics of the Microsoft Kinect camera and to survey other cameras, in order to find the most adequate options.
As selection criteria, we considered real-time image acquisition, usability in different scenar-ios, minimal retailing price and the potentiality of integrating in the NeuroKinect system [61].
There are four types of depth estimation technologies which the studied sensors incorporate: passive stereo, active stereo, structured light (SL) and time of flight (ToF), schematically repre-sented in Figure3.2. SL together with passive and active stereo are stereo-based solution, based on comparing features of a scene from two viewpoints [62].
• Passive stereo: there are two cameras that match features of a scene, triangulating the dis-tance of each point to the sensor. It lacks when matching surfaces with no texture, like white walls or empty desks, meaning that it is preferable for outdoor environments usage, where textureless surfaces are rare;
• Active stereo: there are two cameras combined with a projector, which projects a random pattern, typically IR radiation. The projected light pattern allows to better track each point’s features, overcoming the difficulties tracking textureless surfaces associated with passive stereo, making this solution more versatile than the previous one;
• Structured light (SL): combination of single camera and single projector, which projects a known pattern, forming a stereo system, usually working in the IR spectrum so that the light pattern is not visible. The projection power needs to be limited due to safety reasons, leading to operation distances of only up to 4-5 meters. As more dots are projected, less power they have, and vice-versa, so there is a trade off between robustness and accuracy; • Time-of-flight (ToF): the distance of each point to the sensor is directly measured by
calcu-lating the time offset between a signal’s emission and reception. Similarly, it usually relies on IR radiation, invisible to the human eye.
Figure 3.2: Methodologies for depth estimation. a) Passive Stereo; b) Active stereo; c) Structured light; d) Time of flight. Adapted from [62].
3.4.1 Microsoft Kinect v2
The first depth sensor to be majorly investigated was the Microsoft Kinect, a low-cost sensor developed by PrimeSense, used to track the human body and quantify its movement [62,61].
the second version of the Microsoft Kinect (in Figure3.3) is a markerless sensor able to gather positional information about a subject’s motion. For the past decade, this sensor has been an object of interest for researchers, due to its affordable price, of about $150, and its immediate availability [18]. In addition, on 2012, Microsoft released the Kinect Software Development Kit (SDK) for Windows, which supports the development of systems including skeletal tracking. Its impact reached a wide range of areas, particularly the medical field, with applications such as games for authistic children or robotics for assisting doctors in operating rooms [63].
The Kinect was developed by Microsoft for the Xbox360, to revolutionize the way people play games, making the interaction between the body and virtual environment more natural and providing a more immersive and authentic experience. Depth sensors have enabled the capturing of human motion, adding a third dimension to the 2D plain image provided by the traditional RGB cameras [63].
Figure 3.3: The Microsoft Kinect v2 camera, developed for Xbox 360.
The Kinect physical constitution incorporates advanced sensing hardware, such as a depth sensor, a RGB camera and a four-microphone array. It acquires data up to 30 fps, with a resolution of 512x424 pixels, resulting in more than 300,000 points in each frame [64,65].
The functioning of the depth sensor is based on ToF principle, with an IR projector, that emits IR radiation, and an IR detector. The time offset between the signal’s departure and arrival allows the estimation of depth in the whole field of view, allowing the reconstruction of a depth map, which is encoded by gray values, as showed in Figure3.4. Black pixels indicate that the points are too far from the camera, too close or reflect poor IR lights (specular surfaces) [63,65].
Figure 3.4: Kinect depth sensing. a) IR dots seen by IR detector; b) Depth map in gray values. Adapted from [63].
Some studies have been made to compare the Kinect’s performance in clinical applications with optical marker-based systems, which are considered the "gold standard" in motion quantifi-cation, providing accurate results, especially for spatiotemporal parameters [18]. Comparing the joint angles computation precision of the Kinect with the Vicon system, it was verified that the
RGB-D sensor offers a range of disparity that assures enough precision for clinical rehabilitation assessments. In addition, similar results were obtained in the evaluation of individuals that experi-enced anterior cruciate ligament ruptures, showing that the Kinect skeletal model has appropriated accuracy for this purpose. Another study indicated that, for spinal cord injury rehabilitation pa-tients, the Kinect achieves competitive motion quantification accuracy results when compared with Optitrak system. For joint angle estimation, the camera can obtain results similar to inertial sen-sors, but due to its low sampling rate (30 samples per second), it may be insufficient to measure faster movements, such as hand gestures [58,59]. Thus, this tool has indeed proven to hold poten-tial for the assessment of clinically relevant features related to gait, providing accurate spapoten-tial and timing measurements for gross movements.
Notwithstanding, the system is prone to errors, which usually have three main sources: the sen-sor, the measurement setup and properties of the object. Sensor errors are primarly related with inadequate calibration and inaccurate measurement of disparities. Regarding the measurement setup errors, they are mainly due to lighting conditions and imaging geometry, such as occlusion, shadows, distance from the sensor, reflective floor surface, among others. The optimal measure-ment for the Kinect happens between around 0.5m and 5m of distance to the camera. Finally, the object properties also impact the accuracy of the results, since smooth and shiny surfaces that appear overexposed in the IR image prevent the correct measurement of disparities [63].
The launching of this camera lead to the development of many new cutting-edge depth cameras that limited Kinect adoption, resulting in its discontinuation in 2017 [62,61].
3.4.2 Stereolabs Zed
ZED (see Figure3.5) was developed by Stereolabs and it is able to estimate 3D information based on passive stereo methodology. This sensor has a dual 4MP Camera which allows the capturing of 3D stereo video with low-light sensitivity, even in challenging environments, such as low-light scenes [66].
Figure 3.5: Stereolabs Zed stero camera. [66]
This sensor has been developed mainly for autonomous car navigation and mapping. The main features of ZED are its high resolution and high frame rate and its ability to perceive depth indoors and outdoors at up to 20 m of distance. The resolution and frame rate are inter-dependable, so that for a resolution of 2208x1242 pixels it acquires at 15 fps, but for VGA resolution it reaches up to 100 fps [66,61].
To facilitate the development of ZED-based application, a SDK is provided, which helps to process the disparity map on the host machine, providing positional tracking, detecting the vari-ations in the camera position, and third-party integration with other technologies, such as Unity, ROSS, pcl, OpenCV, Matlab and Oculus [66,61]. However, this SDK does not provide tools for body tracking, so another SDK would have to be used for the purpose of our assessment.
This sensor is currently available in the market, costing around $500.
3.4.3 Orbbec
Orbbec is a company founded in 2013 specialized in the development of 3D in-depth sensors. It holds patents regarding these type of sensors, whose products have applications in different environments, such as smart home applications, industry, healthcare, robotics, 3D scanning and so on [67].
Beyond the potential of 3D sensors for gesture and face recognition, as well as three-dimensional map construction and measurement, they are highly recommended for gait quantification purposes, since they include a body tracking SDK, making Orbbec sensors smarter and more understanding of human bodies [67].
There are two main series of products developed by Orbbec: Persee and Astra, described below.
3.4.3.1 Orbbec Persee
Persee sensors (see Figure3.6) are the higher level version of Orbbec. They are considered a 3D camera-computer, as they combine a depth camera with an integrated computer. Even though some of their specifications are similar to the Kinect v2, Orbbec includes a built-in processor, making it more useful and versatile. Its range of capturing goes from 0.6m to up to 8m of distance, with optimal detection before 5m of distance. This sensor provides a sampling rate of around 30 fps and it has VGA resolution when capturing depth image and 1280x720 pixels when capturing RGB image. It has two incorporated microphones and an integrated processor with 2GB RAM, CPU Quad-Core Cortex A17 up to 1.8GHz and GPU Mali-T7 600MHz.
For it is also considered a computer, it has USB, Micro USB, HDMI and Micro SD card reader entries and supports Wi-Fi and Bluetooth connection, making it possible for this device to easily plug into a TV/monitor, or to run without a display.
It is highly compatible with applications developed in OpenNI and provides a development kit, Orbbec Astra SDK, which includes body tracking and gestural tracking. Its retail price is around $240.
3.4.3.2 Orbbec Astra
Astra series (see Figure3.7) include Astra, Astra S and Astra Pro sensors. The main difference between them and the previous sensor lies mainly on the computer component, which lacks in this version. The connection with Wi-Fi and Bluetooth is not possible and it does not have the integrated processor. On the other hand, similarly to Persee, it has high compatibility, supporting the same operating systems and the same development kits [67].
Figure 3.7: Orbbec Astra sensor. [67]
The three versions of Astra sensors are very similar, except in range of motion, narrower in Astra S (≈2m against ≈8m of other versions), and in RGB resolution, better in Astra Pro (1280x720 against 640x480). Both Astra and Persee rely on SL technology for depth estimation. The three versions cost around $149.00.
3.4.4 Intel RealSense
Intel has been one of the main depth camera developers in the past years. The latest cameras that have been introduced are RealSense 400 Series, whose use active stereo IR vision to calculate depth. Both versions, D415 and D435, showed in Figure 3.8, include a pair of depth sensors, a RGB camera and an infrared projector.
These cameras are designed to be small and portable, and to be used in both indoor and outdoor scenarios, providing an image resolution of up to 1280x720, the RGB frame rate of 30 fps and the depth frame rate of 90 fps, about 3 times higher than Kinect. Also, it has a dedicated color image signal processor for image adjustments and scaling color data and an active IR projector that illuminates the scene and enhance the depth data. Furthermore, these sensors have a long-range capability, going up to around 10 meters.
Figure 3.8: Intel RealSense Depth Camera D415 (above) and D435 (below). [68]
The Intel RealSense provides also a new cross-platform and open-source SDK to ease the access to sensor data stream and the application development both for Windows and Linux. D415 version costs around $150, while D435 reaches around $180. Apart from the retailing price, the main difference between both camera versions is the field of view, which is wider in D435, better for situations with moving objects or moving sensor, covering more area and avoiding blind spots [68].
3.4.5 ASUS Xtion Pro
Asus Xtion Pro estimates depth based on SL technology. It is basically composed by a fixed pattern IR emitter, an IR detector and an auxiliary RGB camera, acquiring color and depth simultaneously [61]. This sensor was created mainly for developers, for motion-sensing applications, such as games, physical rehabilitation, educative innovative application and so on.
Figure 3.9: Asus Xtion Pro Camera. [69]
The ideal range of capture is around 2.7m, from 0.8m to 3.5m, a short range for walking measurement. Besides, this camera is mainly suitable for indoor environments.