ContentslistsavailableatScienceDirect
Biomedical
Signal
Processing
and
Control
j ou rn a l h o m e p a g e :w w w . e l s e v i e r . c o m / l o c a t e / b s p c
Study
on
the
usage
feasibility
of
continuous-wave
radar
for
emotion
recognition
Carolina
Gouveia
a,b,∗,
Ana
Tomé
a,c,
Filipa
Barros
d,e,
Sandra
C.
Soares
d,e,f,
José
Vieira
a,b,c,
Pedro
Pinho
b,gaDepartamentodeEletrónica,Telecomunicac¸õeseInformática,UniversidadedeAveiro,Portugal bInstitutodeTelecomunicac¸ões,Aveiro,Portugal
cInstitutodeEngenhariaEletrónicaeTelemáticadeAveiro,UniversidadedeAveiro,Portugal
dCenterforHealthTechnologyandServicesResearch(CINTESIS.UA),DepartmentofEducationandPsychology,UniversityofAveiro,Aveiro,Portugal eWilliamJamesCenterforResearch(WJCR),DepartmentofEducationandPsychology,UniversityofAveiro,Aveiro,Portugal
fDepartmentofClinicalNeuroscience,DivisionofPsychology,KarolinskaInstitute,Stockholm,Sweden gInstitutoSuperiordeEngenhariadeLisboa,Portugal
a
r
t
i
c
l
e
i
n
f
o
Articlehistory: Received16April2019 Receivedinrevisedform 22December2019 Accepted24December2019 Availableonline9January2020 Keywords:
Continuouswaveradar Emotionrecognition Patternrecognition Support-vectormachine K-nearestneighbour RandomForest
a
b
s
t
r
a
c
t
Non-contactvitalsignsmonitoringhasawiderangeofapplications,suchasinsafedriveandinhealth care.Inmentalhealthcare,theuseofnon-invasivesignsholdsagreatpotential,asitwouldlikelyenhance thepatient’sadherencetotheuseofobjectivemeasurestoassesstheiremotionalexperiences,hence allowingformoreindividualizedandefficientdiagnosesandtreatment.Inordertoevaluatethe possi-bilityofemotionrecognitionusinganon-contactsystemforvitalsignsmonitoring,wehereinpresenta continuouswaveradarbasedontherespiratorysignalacquisition.Anexperimentalsetupwasdesigned toacquiretherespiratorysignalwhileparticipantswerewatchingvideosthateliciteddifferentemotions (fear,happinessandaneutralcondition).Signalwasregisteredusingaradar-basedsystemanda stan-dardcertifiedequipment.Theexperimentwasconductedtovalidatethesystemattwolevels:thesignal acquisitionandtheemotionrecognitionlevels.Vitalsignwasanalysedandthethreeemotionswere iden-tifiedusingdifferentclassificationalgorithms.Furthermore,theclassifierperformancewascompared, havinginmindthesignalacquiredbybothsystems.Threedifferentclassificationalgorithmswereused: thesupport-vectormachine,K-nearestneighbourandtheRandomForest.Theachievedaccuracyrates, forthethree-emotionclassification,werewithin60%and70%,whichindicatesthatitisindeedpossible toevaluatetheemotionalstateofanindividualusingvitalsignsdetectedremotely.
©2020TheAuthors.PublishedbyElsevierLtd.ThisisanopenaccessarticleundertheCCBY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).
1. Introduction
Emotionsare adaptive andmultidimensional responses
trig-geredbymeaningfuleventsand/orstimuli,thataffectthewaywe
think,feel,behaveandinteractwithothersinourdailylife[1,2].
Theseresponsesencompasschangesindifferentsystemsincluding
cognition,physiology,motivation,motorexpressionandsubjective
sensations,henceallowingforeffectiveself-regulationprocesses
andprovidingcrucialresourcestoasuccessfuladaptationtothe
environment[3,4].Thechangescanthenbeexpressedbydifferent
means,suchasviaverbalcommunication,facialexpressionsand
∗ Corresponding authorat:DepartamentodeEletrónica,Telecomunicac¸õese Informática,UniversidadedeAveiro,Portugal.
E-mailaddress:[email protected](C.Gouveia).
bodymotion[5],aswellasbymoresubtlecues,includingbiological
signs(increasedheartrateandrespiration)[6].
Although automatic emotion recognition represents a
chal-lengingprocess,it canprovideusefulinformationregarding the
individuals’well-beingand,generallyspeaking,contributeto
sys-tems’enhancement,accordingtotheuser’sneeds.Forexample,it
canbeusefultoadjustmedicalandpsychologicaltreatmentsor
evenaidindetectingtheveracityofinformationinforensic
inves-tigations.Additionally,emotionidentificationcanbeacrucialasset
intheHuman–ComputerInteraction(HCI)field,onceitenablesthe
adjustmentofthedesignanditsfunctionalfeaturestotheusers’
needs[5,7–9].
Emotionrecognitionthrough physiologicalsigns hasalready
beenreported,namelybyusingbreathing rate,bloodflow, skin
conductance,electroencephalogram(EEG),andelectrocardiogram
(ECG)[6],[8].However,mostofthesephysiological-basedsystems
stillrelyonthedirectcontactwithsensorsanditisrecognizedthat
https://doi.org/10.1016/j.bspc.2019.101835
1746-8094/©2020TheAuthors.PublishedbyElsevierLtd.ThisisanopenaccessarticleundertheCCBY-NC-NDlicense(http://creativecommons.org/licenses/by-nc-nd/4. 0/).
theindividual’sawarenessofthemonitoringprocessmayinfluence
theresults[6].Moreover,theuseinreallifecontexts,whichisof
greatvaluetoallowfortheestablishmentofindividualizedprofiles,
canalsointerferewiththesignal.Therefore,non-contactsystems
canbeadvantageousintheseapplications,astheycanremotely
monitorvitalsigns.
Recently, emotion recognition algorithms using vital signs
acquiredbyradarsystems,wasdonein[7,9],witha
continuous-wave (CW)radar and a frequency-modulated continuousradar
(FMCW)radar,respectively.In[9],theheartbeatsignalwastracked
andused toidentifypositive and negativeemotions(joy,
plea-sure,sadnessandanger),thatresultedfrommemoriestriggered
bymusicorphotos.Thesupport-vectormachine(SVM)
classifica-tionalgorithmwasimplemented,and72.3%ofaccuracyratewas
achieved,consideringthedatasetof12volunteers.Additionally,
[7]presentedanapplicationusingtheK-nearestneighbour(KNN)
asclassifier,usingtherespiratorysignaldetectedbyaheterodyne
CWradar,withanaccuracyrateof67.4%.Inthiscase,theclassifier
wastrainedalsotoidentifyfouremotions,nowincludingan
emo-tionallyneutralcondition(joy,sadness,fearandneutral).Adataset
from5subjectswasused,withwaveformfeatures(suchasmean
orstandarddeviation)andspectralfeaturesobtainedthroughthe
powerspectraldensitycomputationinseveralfrequencyintervals,
from0to0.4Hz.
SimilarstudiesusingCWradarswerecarriedoutin[10,11],but
focusedonthementalstressresponse.In[10],theSVMclassifier
wasusedtoidentifyifasubjectwasundermentalstress,physical
stressorinsteadystate.Theuseddatasetencompassedwaveform
andspectralfeatures,extractedfromthebreathingsignal,similarly
to[7].Thestressstatewasidentifiedwithabinarystrategy,withan
accuracyof77.5%and77.9%,forthesteadystate/mentalstresscase
andthesteadystate/physicalstresscase,respectively.Thebinary
strategywasalsotestedforthesteady/stressstatein[11],butusing
neuralnetworks(NN).Inthiswork,featureswereextractedfrom
therespiratorysignal.However,authorshadtolimittheamountof
data,whichledtoalimitednumberoffeaturesusedontheinput
oftheNN(asinglefeatureandcombinationsoftwofeatureswere
used).
Inthiswork,aradarsystembasedonasoftwaredefinedradio
(SDR)technology wasemployedto acquiretherespiratory
sig-nalsresultingfromtheinductionofthreedifferentemotions:fear,
happinessandanemotionallyneutralstatetoserveasacontrol
condition.Alongwiththerespiratorysignal,wealsoaimedat
cap-turingothertypesofbodymotionthatoccurnaturallyduetothe
individual’sreactiontospecificemotions.Thesebodymovements,
thathavebeenseenasasourceofdistortioninsimilarworks,hold
importantinformationregardinghumanresponsivenesstospecific
emotions.Asinpreviousworks,inwhichtheauthorsinduced
emo-tions[6,8],inthepresentstudyweusedfilmclipstoelicitspecific
emotions.Eachparticipantwasexposedtoasetofvideosinducing
thethreespecificemotions,whichwerepresentedinthree
differ-entsessions,separatedbyatleastoneweek,in ordertoisolate
thespecificemotions.Volunteersdidnothavepriorknowledgeof
video’scontentthatweregoingtowatchinthatday.
Fortheemotionrecognitionusingasetoffeaturesextracted
fromthesesignals,threedifferentclassificationalgorithmswere
used,inordertoevaluatewhich providedresultswiththebest
accuracy:SVM,KNNandtheRandomForest.Furthermore,afeature
selectionwasperformedtoincreasetheaccuracy.Therespiratory
signalsofthevolunteerswereacquiredusingsimultaneously,the
CWradarsystem,fromnowoncalledBio-Radar,andacertified
measurementequipment(BIOPAC),withachest-bandtransducer.
Bycomparingtheperformanceofclassifierswithsignalsacquired
usingbothsystems,weaimedatinvestigatingifsignalsremotely
measuredcanbeusedforemotionidentification.Thiswaspossible
toproveindeed,since theclassification resultsconcerning
Bio-RadarandBIOPACsignalswereinthesameorderofmagnitude.The
bio-radarsystemwascalibratedusingachest-wallsimulator
devel-opedforthispurposeandalreadyvalidatedusingthesamecertified
measurementequipment,in[12].Thebestresult,regardingthe
Bio-Radarsignal,wasachievedfortheRandomForestclassifier,being
65.2%and67.1%,beforeandafterthefeatureselection.
Thiswork is dividedas follows: in Section 2,the Bio-Radar
operationmodeisexplainedandtheexperimentalprocedureis
described.Thesignalprocessingalgorithmperformedbeforethe
classificationstageisexplainedindetail.Thissectionendswiththe
validationoftheBio-Radarsignalacquisition,wheresomeuseful
signalcharacteristicsareidentified.Thereafter,theclassifier
imple-mentationisexplainedinSection3,whichstartswiththefeatures’
extractionprocedureandexplainstheclassifierimplementation.
Thecross-validationmethodsarepresentedinSection4alongwith
theresults.ResultsarelaterdiscussedinSection5,througha
perfor-mancecomparisonbetweendifferentclassifiers.Weintendtofocus
speciallyintheclassifierperformanceconsideringbothmeasuring
systemsandinspectiftheiraccuracydiffersorifitisequivalent.
Finally,conclusionsarepresentedinSection6.
2. Experimentalprocedure
2.1. Continuouswaveradaroperationprincipal
Dopplerradarsforvitalsignsmeasurement,alsodefinedasthe
Bio-Radarsystems,consistinatechnologycapabletoacquirethe
respiratory and thecardiac signals, withoutinterfering directly
withtheindividual.Forthispurpose,ituseselectromagneticwaves,
whicharetransmittedtowardsthechest-wallofthesubjectunder
monitoring,andthereflectedechoisreceived.FromtheDoppler
effect,itispossibletorelatethereceivedsignalpropertieswith
thedistancechangebetweentheradarantennasandthesubject’s
chest-wall,whichmovesaccordingtothecardiopulmonary
func-tion.The CW radar continuouslytransmits a sinusoidalcarrier,
generateddigitally,andreceivestheechofromthereflectingtarget.
DuetotheDopplereffect,thereisaphasechangeasthesubject’s
chest-wallmovestowardsorawayfromtheradarand,hence,a
phasemodulationinthereceivedsignaliscreated[13].The
math-ematicalmodeloftheBio-RadarchannelresponseanditsMATLAB
simulationaredetailedin[14].
TheBio-RadarimplementedinthisworkisdepictedinFig.1
andit wasdeveloped in[13].It consistsin areal-time
measur-ingsystemimplementedwiththeLabVIEWsoftware.Signalswere
acquiredusingtwoantennas,onefortransmissionandtheother
forreception,andafront-endbasedinaSDRsystem.Theseradars
allowthedigitalconfigurationofitsinputandoutput(receiverand
transmitter),regardingtherequiredfrequencyandsamplingrate
ofthecurrentapplication.Intheframeworkofthisapplication,the
SDRusedwasanUSRPB210.Itoperatesinalimitedrangeof
car-riers(70MHzto6GHz)and,therefore,weselected5.8GHzasthe
carrierfrequency.
2.2. Vitalsignsacquisition
Theexperimentwasconductedwithninevolunteers,inthree
differentdays,spacedbyatleasttwodays(hence,awithin-subject’s
experimentaldesign).Theexperimentconsistedinmeasuringthe
respiratorysignalofeachsubjectandothermotioncharacteristics
relatedtohisemotionalstate,whiletheywerewatchingdifferent
setsofshortvideosthateliciteddifferentemotions(fear,
happi-ness andneutral emotion),similarly totheworks presented in
[15,16].Forexample,happinesswasinducedbycomedyvideos,
theneutralstatewasinducedbydocumentaryvideosand,finally,
Fig.1.Bio-Radarprototype:(a)hardwareset-upcomposedbyaSDRfront-endand twoarrayantennas,(b)signalacquisitionusingtheLabVIEWsoftware.
inthisexperimentwerepreviouslyselectedinpilotstudies,where
foreachvideoitwasverifiedhowintensewasthetriggered
emo-tion.Oneachtestingsessiononeemotionwasinducedbyusing
thesedifferenttypesofvideos.Duringtheexperiment,videoswere
shownaccordingtoanincreasingorderofemotionintensification,
inordertoallowforacontinuousinductionandaneffective
engage-mentofthesubjectthroughoutthesession.Participantsdidnot
havepriorknowledgeaboutthetypeofvideosthattheywereabout
towatch,toavoidforpotentialconfounds.Eachtestingsessiontook
around30minutes.
Sincethemaingoalofthisworkistovalidatetheusageofa
Bio-Radarsysteminemotionrecognition,respiratorysignalwas
acquiredusingBio-Radarandacertifiedmeasuringequipment,the
BIOPACMP160DataAcquisitionSystemwithAcknowledgment5
Software(fromBIOPACSystems,Inc.).TheBIOPACsystemis
con-nectedtoanacquisitionboard, whichhaveseveralmodulesfor
differenttypesofsignalacquisition,suchasECG,breathingand
bloodpressure.ThemoduleRSP100Cisfocusedintheprocessing
ofdataacquiredfromarespirationtransducerchestband,placed
aroundthechestcavityofthesubjectattest.Thistransducer
mea-surestherespiratoryeffortbyanalysingtheinstantaneousthoracic
perimeter[12].
The Bio-Radar signal is a lowpass signal with a bandwidth
equalto0.2–2Hz,whichcomprisestherespiratorybandwidthfor
ahealthyadult(0.2–0.4Hz)[17],thehyperpneacases(0.4–0.8Hz)
andthecardiacsignaturealongwithotherrandommotionsfrom
thebody (0.8–2Hz)[18].The receivedsignalhad tobevisually
inspectedduringthereal-timeacquisition,onceitwasrequired
todo aphaseadjustmentinrealtime,forseveralreasons.First,
thelocaloscillatorintheUSRPboardstabilizesinadifferentphase
valueeverytimethesystemrestarts,whichchangesthemeanvalue
ofthephaseandhencethearcpositioninthecomplexplane.The
distancebetweenthesubjectand theradaralsodeterminesthe
arcposition.Furthermore,thesubjectisnottotallystillduringthe
monitoringperiodandcannotkeepthesameamplitudeof
chest-wallmotion(infact,changesinamplitudeastheresultofdifferent
emotions,isexactlywhatwewanttocapture),henceweexpect
thatthemeanvalueofthephasechangesduringtheexperiment.
Ifthenominalphaseisnearor−inthebeginningofthe
exper-iment,itcanreacheasilythosevalues andwraps willoccur.In
thosecases,thephaseshouldbecorrectedbychangingthephase
ofthetransmittedsignal,digitallyandinrealtime.Theeffectof
thisphasecorrectionprocedureispresentedinFig.2.Thisfigure
hasawrapmarkedas‘W’andthreedifferentsegments(‘S1’,‘S2’
and‘S3’),whichresultfromthephaseadjustmentinthattime,to
avoidnewwrapoccurrences.
Regarding the acquisition details, the Bio-Radar signal is
receivedwithasamplingrateequaltofs1=100kHzandis
down-sampledinreal-timetofs2=1kHz.ThisisperformedinLabVIEW
Fig.3.BlockdiagramoftheBio-Radarsignalpre-processingbeforetheclassificationstage.
anditwasnecessarytoeasethevisualizationinreal-timeandto
decreasetheamountofdatatobestored,whichcanbeprocessed
inMATLABwithlowcomputationaleffort.TheBIOPACsignalis
acquiredwithasamplingrateequaltofs2.
2.3. Bio-Radarsignalprocessingalgorithm
Consideringthephaseadjustmentthatwasnecessaryto
per-formduringthesignalacquisition,thesignalprocessingalgorithm
implementedintheBio-Radarsignalshouldbeperformedwith
care.Inthissense,thealgorithmexecutesthefollowingsteps(see
blockdiagramfromFig.3).
SignalsareacquiredusingaSDR,whichhasanIn-phaseand
Quadrature(IQ)demodulatoranddeliversonitsoutputthe
com-plexbaseband signal,g(n),witha samplingfrequency equal to
fs2.Thus,thefirststepistodoasecondstageofdownsampling,
oncethesignals thatwe aredealing arelowpass signals.
Com-plexsignalgd(n) hasa newsamplingrateequal to fs3=100Hz.
Also, the BIOPAC signal is downsampled to this new sample
rate.
ThenextstepistoremovetheDCcomponentofthecomplex
signal.Inorder todo this,an algorithm wasdeveloped,having
inmind that thesignals extractionresultsfromthearctangent
computation.Inotherwords,itisnecessarytoguaranteethatthe
arc in the complex planedoes not cross the value.
Further-more,thephase transitions duetothephase correctionshould
alsoberemoved.Fig.4 shows analgorithm toperformthe DC
removal, that avoids the wrap occurrence and removes phase
transitions.
Thealgorithmstartedtoinspectiftherearephasetransitions
duetoouradjustments.Inthosecases,thesignalisdividedin
sub-segments(thecorrespondentsub-segmentpriorthetransitionand
afterthetransition)andthephasesarealigned.Then,withamore
uniformedsignal(regardingthemeanphasevalue),theellipse
fit-tingmethod[19]isappliedtoremovetheDCcomponentofthe
signal.Fig.5depictstheellipsefittingmethodanditsimportance
intheextractedsignal.First,theellipsethatbestfitstothearcis
determinedanditscentreiscomputed.Then,thecentre
coordi-natesaresubtractedfromthesignalinordertore-centreitbackto
theorigin.Fig.5(b)showsthatifthereceivedsignalhashighDC
component,thephasedemodulatedresultisdistortedand
atten-uated.Ontheotherhand,thesignalcanrecoveritsnormalform
aftertheDCremoval.
Inthisstage,itisrequiredtoinspectifthearcislocatedaround
thevalue.Inthiscase, thearc shouldberotatedtoavoidthe
wrapoccurrenceafterthearctangentcomputation.Evenwiththe
arclocatedfarfromthevalue,meanvaluefluctuationsduetothe
subjectrandommotioncanimplythatthearccrossestheorigin
ofthecomplexplane.Inthesecases,asmalloffsetisaddedjustto
avoidwraps,butitshouldbesmallenoughthemaintainthesignal
informationunchanged.
AftertheDCremoval,theresultingcomplexsignalisp(n).The
respiratory signal is recovered from p(n), by performingphase
demodulationwiththearctangentcomputation,resultinginthe
realsignalb(n).
Later,therespiratorysignalb(n)isdividedinsub-segmentswith
oneminutedurationforthefeatureextractionforclassification.
Fig.4. FlowchartofthealgorithmimplementedtoremovetheDCcomponent.
2.4. Bio-Radarsignalvalidation
Intheintroductorysection,wereferredtothenumerous
advan-tages of thevital signs acquisition using non-contact methods,
consideringtheemotionrecognitionframework.Nonetheless,one
needstoverifythatthesameresultsareachievedbyboth
con-ventionalsensorsandcontactlessmeans, andthat thiscanalso
contributefortheclassifiersperformanceenhancement.Forthis
purpose,vitalsignsweremeasuredusingtwomeasurement
set-upssimultaneously:theBio-Radarprototypefortheremotesignals
acquisitionandtheBIOPACsystem,astheconventional
measure-mentequipment.Fig.6showstwosignalsamplesobtainedduring
theFearstatetest,acquiredsimultaneouslywiththesetwo
sys-tems.Inthissample,itispossibletoobservethatthesubjectwas
frightenedtwiceandtheseeventsarerepresentedbytwopeaks
withhigheramplitudes,whencomparedwiththerestofthe
sig-nal(markedwitharedpointinthefigure).Immediatelyaftereach
momentoffright,itispossibletoidentifyanincreaseinthe
heart-beatrateintheBio-Radarsignal,duringtheexhaleperiods.This
Fig.5.Ellipsefittingmethod:(a)consideringthearcformedinthecomplexplane,theellipsethatbestfitstothearcisestimatedalongwithitscentre,(b)signalafterphase demodulationwithDCcomponent,(c)thearcisre-centredbysubtractingthecentrecoordinatesfromthesignal,(d)signalafterphasedemodulationwithoutDCcomponent.
Fig.6. RespiratorysignalacquiredduringtheFearstate:bytheBio-Radarsystem(top)andbytheBIOPACsystem(bottom).Peakswithreddotswerethemomentswhere subjectgotfrightened.Segment‘A’containsanincreaseoftheheartbeatrateandtheremainsegments(‘B’and‘C’)signthebodymotionofthesubject.(Forinterpretation ofthereferencestocolorinthisfigurecitation,thereaderisreferredtothewebversionofthisarticle.)
capturedintheBIOPACsignal.Fig.7showsazoom-inofthefirst
momentoffright,consideringthesignalpresentedinFig.6.Inthis
figure,theincreaseofheartbeatrateismoreclearintheBio-Radar
signalratherintheBIOPAC.
Theradarsystemcanalsodetecttherandommotionofthe
sub-ject,thatoccursnaturallyduetothesuddenreactionsofvideos
(suchaslaughorfright),orforcomfortingpurposes.Infact,this
ran-dommotionalongwithothersignalsignaturesthatarisefromthe
emotionalinduction,serveastoolsfortheproperemotion
recogni-tion.Forexample,thesubject’sdiscomfortcanbeobservedinthe
Bio-RadarsignalfromFig.6.Ifhe/shemovesslightlyhis/her
Fig.7.Zoom-inoftherespiratorysignalintheFearstate,duringthefirstfrightexperience:bytheBio-Radarsystem(top)andbytheBIOPACsystem(bottom).
Fig.8.Specificationofthesegmentationprocessforthedifferentfeaturecategories.
markedas‘B’and‘C’.Ontheotherhand,thistypeofeventisnot
evidentintheBIOPACsignal.
3. Classificationprocedure
3.1. Featuresextractions
Featureswereextractedfromoneminuteobservations.Forthis
purpose,thesignalwasdividedinoneminutesegments,performed
bySegmentationblock,asdepictedinFig.8.The“rawsignal”atthe
blockinputistheresultofphasedemodulation.Insidetheblock,
thissignalpassesthroughdifferentdataprocessing,accordingly
withthefeatures’categoriesthataregoingtobeused.
Duringthemonitoringperiod,changestotherespirationpattern
areexpectedaccordingtothenaturalreactionofthebodydueto
emotions,andotherbodymotions.Thus,statisticalandwaveform
characteristicscanbeidentified.In addition,thespectral
evalu-ationisalsocarriedout,not onlyfor therespiratorybandwidth
0.2–0.4Hz,butalsoforhigherfrequenciesthatcanrevealheartbeat
detectionorsuddenmotionsfromthesubject.
Forspectral features,theinputdataisonly dividedin1min
sub-segments,withoutanypriorprocessing.Ontheotherhand,
forstatisticalandwaveformfeatures,afilterwasappliedinthefull
signalbeforethesub-segmentationprocess.Thefilterusedwasa
2ndorderButterworthwithabandpassequalto0.05–1.5Hz.The
usageofthisfilterisimportanttocentretherespiratorysignalin
Table1
Featuresspecification:statisticalandwaveformfeatures.
Statisticalandwaveformfeatures
FT1 Meanvalue
FT2 Variance
FT3 Waveformwidth
FT4 Timebetweenpeaks
FT5 Respiratoryrate
zero,andtoavoidbiasedresultsonthemeanvalueduetodifferent phases,asshowedinFig.2.
Atotalof12featureswereextractedfromthesegmentedsignals
andtheyarepresentedinTables1and2.
Statisticalfeaturesencompasses:
• FT1–themeanvalueofthesegment;
• FT2–thevarianceconsideringtheaverageofallsegment
sam-ples.
Waveformfeaturesincludes:
• FT3–thewaveformwidth,whichisthemeanvalueofallpeak
widthsinthatsegment.Thewidthisdefineddistancebetween
thepointswheresignalinterceptsareferenceline.Inthiscase,
thereferencelineislocatedbeneaththepeakathalfofthepeak
Table2
Featuresspecification:spectralfeatures
Spectralfeatures
FT6 PowerSpectraldensityinthe
range0—0.1Hz
FT7 PowerSpectraldensityinthe
range0.1–0.2Hz
FT8 PowerSpectraldensityinthe
range0.2–0.3Hz
FT9 PowerSpectraldensityinthe
range0.3–0.4Hz
FT10 PowerSpectraldensityinthe
range0.4–0.9Hz
FT11 PowerSpectraldensityinthe
range0.9–1.5Hz
FT12 Ratioofthepowerspectral
densityvaluesbetween
low-frequencyand
high-frequencyrange
• FT4–thetimebetweenpeaks,wherethenumberofsamples betweentwopeaksiscountedanditsmeanvalueiscomputed • FT5 – the respiratory rateof the full segment,which canbe
obtainedbythroughtheestimationofthemaximumcomponent inpowerspectraldensity.
Finally,spectralfeaturesFT6-FT11aretheamplitudesofspectral componentsinthedifferentfrequencybands.Thisisalsoobtained throughthepowerspectraldensity,usingaslidingHamming win-dowwith50%ofoverlap.ThelastfeatureFT12relatesthespectral componentswithinthelowband(0.1–0.5Hz)andthehighband (0.6–1.5Hz).Itisexpectedthatthelowbandexpressesonlythe res-piratorypresenceandthehighbandexpressesarestlessbehaviour. Sincefeatures have differentnaturesand theirrange of val-uesaredifferentfromeachother,anormalizationisrequired.In thiscase,thez-scorenormalizationwasappliedtoallfeatures,as describedby
˜
xi=xi−mi
i (1)
where ˜xiisthenormalizedfeature,xiisthefeaturetobenormalized,
miisitsmeanvalueandiisitsstandarddeviation.
3.2. Classifiers
Inthiswork,threedifferentclassifierswereimplementedand theirperformancewascompared.Thesameclassification proce-durewasappliedforboththeBIOPACandtheBio-Radarsignals, inordertovalidatetheusageofthedevelopedprototypeforthis scope.
Theclassificationwasperformeddefiningdifferentproblems: binaryandmulticlass,bothusingadatasetwith221observations foreachemotion.Forthebinaryproblem,onlypairsofemotions wereconsidered,thereforethedatasetlengthhad442 observa-tions. On theother hand, themulticlass case considered three elicitedemotionsand afinaldatasetlengthequal to663 obser-vations.
Noticethattheseobservationswerechosenaftervisual inspec-tionofthesignal,whennoisysegmentsarediscarded.Thisresulted inanimbalanceddataset,sinceitwasnotpossibletohavethesame numberofobservationsforeachperson,ateachemotion.Thus,the numberofobservationswaslimitedtotheminimumdurationtime ofallexperiments,inordertobalancethedataset.
Threeclassificationalgorithmswerechosen–SVM,KNNand RandomForest, consideringtheirgood performance withshort datasets[20].Moreover,SVMandKNNwerealsousedinliterature (in[9,7],respectively)andsincethegoalofthisworkisto
rein-forcethepossibilitytouseradarsystemsinemotionrecognition,
Table3
Accuracyrateresultsforthebinaryclassification.
Happyvs.neutral Happyvs.fear Neutralvs.fear
SVM 70.8% 68.8% 72.2% Bio-Radar 79.2% 73.5% 81.7% BIOPAC KNN 72.6% 72.4% 72.4% Bio-Radar 75.8% 71.0% 79.2% BIOPAC Random Forest 77.4% 73.8% 77.8% Bio-Radar 83.9% 73.3% 83.0% BIOPAC Table4
Accuracyrateresultsformulticlassclassification.
SVM KNN RandomForest
Bio-Radar 58.1% 58.2% 65.2%
BIOPAC 65.2% 61.4% 69.1%
thatsamealgorithmsastheonesimplementedinliteratureshould beused.Inaddition,RandomForestispresentedasanalternate option.
Thesealgorithmswereimplementedforbothbinaryand multi-classclassification.TheaccuracyresultsareshowninTables3and4
,respectively,andarediscussedinSection5.Theirimplementation
consideredthefollowingfunctionalprinciples:
• SVMwasappliedwithaRadialBasisFunction(RBF)kernel[20],
inbothbinaryandmulticlassproblems.
–Regardingthemulticlasscase,sinceSVMalgorithmsarebased
inbinaryclassificationonly,adecisionstrategybasedinvoting
wasappliedinordertoreducethemulticlassprobleminaset
ofbinaryproblems.Inthissense,one-versus-one(OVO)
strat-egyselectstheclassthroughthemajorvote,withinC(C−1)/2
binaryclassifiers(SVMs)thatcanbetodealwiththeCclasses.
• TheKNNalgorithmusestheEuclideandistanceasmeasureof
proximityandtheclosestneighbour(withk=1)forthedecision
rule[21].Thesameclassifiercharacteristicswereusedforboth
binaryandmulticlassproblems.
• Finally,theRandomForestalgorithmconsistsinanensembleof
decisiontreestrainedinparalleltogiveacontributetothefinal
decision.Thisensembleclassifierwasimplementedwiththe
Bag-gingmode,i.e.,eachtreewastrainedwithabootstrapsampleof
thedataset,withoutnormalization[20,22].Inordertoprevent
thedeepgrowoftrees,aminimumleafsizewasassignedtofive
observations.Theout-of-bagscorewasthenusedtochoosethe
numberoftrees(seeFig.9)and70treeswereselected.
Table5
Accuracyrateresultsformulticlassclassificationwithalimitednumberoffeatures.
SVM KNN RandomForest
Bio-Radar 63.3% 66.2% 67.1%
BIOPAC 63.3% 76.5% 75.1%
–ThefeatureimpactcanalsobestudiedfromtheRandomForest method.Forthispurpose,therelevanceofthefeatureselection oneachnodewasanalysedandthesixmorerelevantfeatures werechosentocreate anewdataset.Theclassifiers perfor-mance,withthelimitednumberoffeatures, ispresentedin Table5.
4. Results
SVMandKNNweretrainedandtestedusingtheLeave-one-out
cross-validationstrategy.Notethatthedatasetissmall.
Leave-One-OutisanextremecaseofK-foldcross-validation.Consideringthat
thedatasethasNobservations,thisstrategyusesN−1
observa-tionsfortrainingandtheleftoutoneinstancefortesting[20].With
thismethodwehaveNcombinationsoftrainingsetswithdifferent
singleinstancesfortesting.Thefinalresultshowshowmanytimes
thepredictionwasrightonidentifyingthecorrectlabel.Therefore,
thebinomialdistributioncanbeusedtomodeltheaccuracyofthe
classifier.
InthecaseofRandomForest,theout-of-bagobservations
(sam-plesnotincludedinthebootstrapsample)canbeusedtoestimate
theperformanceoftheclassifier[22].
Consideringtheproceduredescribed intheprevioussection,
theachievedresultsforeachclassifierarepresentedinthetables
bellow.First,Table3presentstheresultsregardingthebinary
clas-sificationproblem.
Then,Table4presentstheresultsachievedformulticlass
prob-lem.
Finally, RandomForest classifier allowed theexecution of a
preliminarystudyregarding thefeatureimportance. Thisstudy
revealedwhichfeatureshadmoreinfluenceontheachievedresults
andthisinformationcanbeusedtopresumablyincreasethe
accu-racyresults,bydecreasingthenumberoffeaturesusedindataset.
Table5showstheobtainedresultsafterimplementingthe
classifi-cationalgorithmswithasmallernumberoffeatures.
5. Discussion
Inthissection,theresultsachievedforbothbinaryand
multi-classproblemsarediscussed.
5.1. Binaryproblem
Inthiscase,byanalysingtheobtainedaccuracies,itispossible
toverifythatresultsfrombothBio-RadarandBIOPACsignalswere
similar.Generally,theHappyvs.fearcaseistheonethatpresents
loweraccuracy,withadifferenceoflessthan10%,ifwecompare
withtheotherbinarycases.Thiscanbeexplainedbythesimilar
responsesforthesetwoemotions,leadingtoanapproximated
fea-tureresult,whichhinderstheclassidentification.Thealgorithm
withbetterperformancewastheRandomForestandtheworst
accuracyresultswereobtainedwiththeSVMclassifier,whichalso
presentedthebiggestdeviationbetweenBio-Radarand BIOPAC
signals,withamaximumdriftof10%,approximately.
Asidefromtheseresultsandinordertoproperlyvalidatethe
Bio-Radarsystemusage,theMcNemartest wasperformed[20],
withthemid-p-valuetest.Thistesthelpstounderstandwhichtype
ofsignalcouldbeusedtoidentifymoretruelabelsaccurately,or
iftheclassifieraccuracyissimilarusingbothsignals.Itverifiesif
thenullhypothesisisacceptedorrejected.Inthiscase,thenull
hypothesisstandsthattheclassifierwiththedifferentsignalshave
thesameaccuracyofpredictingcorrectlytheprovidedclasslabels.
TheMcNemartestwasperformedforeveryclassification
algo-rithm(SVM,KNNandRandomForest)andforeverybinarycase.
Toapplythetest,thepredictedvaluesofthetestsetofthe
leave-one-outstrategyloop,arestoredforallclassifiers.Then,thenull
hypothesisistested,e.g.,ifthepredictedlabelsoftwoclassifiers
havetheequalaccuracyforpredictingthetruelabels.Thetestwas
appliedwith5%significanceleveltocomparetheresultsof
classi-fiers,withinputtheBIOPACsignalsandradarsignals,respectively.
Sincebothpositiveandnegativeemotionsservemotivational
purposes(rewardorwithdrawal,respectively),similarautonomic
nervoussystemresponsesfavourenvironmentaladaptation
pur-poses, as to prepare the organism for appropriate behavioural
responses[23].Althoughthisremainsacontroversialtopicinthe
literature,severalstudieshavecorroboratedthisview,byshowing
anincreasedphysiologicalarousal(e.g.,increasedheartrate)not
onlyinnegativeemotionalstates(whichtrigger“fightorflight”
responses)butalsoinpositiveemotionalstates[24].Thecurrent
resultssupportsuchviewforrespiratorysignalscapturedusingthe
Bio-Radar.
5.2. Multiclassproblem
Theresultsformulticlassclassification(presentedinTable4),
wereagainsimilarwithinthedifferentclassifiersandthedifferent
acquisitionsystems.
Withadetailedanalysisitispossibletoobservethattheresults
fortheKNNandRandomForestclassifierwereapproximatelythe
sameforBio-RadarandBIOPACsignals,withonly3.2%and3.9%of
difference,respectively.Thedifferencewasslightlymoreevident
fortheSVM,with7.1%ofdifference.Forthemulticlasscase,the
bestclassificationalgorithmwastheRandomForest,with65.2%of
accuracyfortheBio-Radarsignaland69.1%fortheBIOPACsignal.
Consistentlywiththepreviousresults,theBIOPACsignalresultedin
betteraccuracyresultsthantheBio-Radar.However,theMcNemar
testperformedforeveryclassificationalgorithm,didnotrejected
thenullhypothesisatthe5%significancelevel,whichsustainthat
differencesbetweenaccuracyresultswithBio-RadarandBIOPAC
signalsarenotrelevant.
Analternatewaytoevaluatetheresultsisbycomputingthe
confidenceintervaloftheaccuracyresults.Forthispurpose,the
normalapproximationofbinomialconfidenceintervalwas
com-putedthroughby:
ˆ p±z
ˆ p(1− ˆp) N (2)where ˆpistheprobabilityofcorrectdecision,Nisthetotalnumber
ofobservationsandzistheequivalent1−˛
2 quantileofthe
stan-darddeviation.Inthiscase,˛=0.05wasconsidered,whichresults
inz=1.96.Infact, ˆp isalsoanestimateofaccuracy,itcanbeobtain
throughtherelation Ns
N,whereNsisthenumberofcorrect
deci-sions[25].Fig.10showstheperformanceofeachclassifierusing
theBio-RadarandBIOPACsignals,andtheirdistributionthrough
thecomputedintervals.
Resultsaredisposedinconsecutivepairsof‘name/signal’,with
therespectiveclassifiernameandsignaltype,where‘bR’stands
forBio-Radarand‘bP’standsforBIOPAC.Byanalysingthisgraph
itispossibletoobservethatresultsunderthesameclassifier
devi-ateslightlybetweenthedifferentsignals,bykeepinganinterval
Fig.10.Classificationaccuracyfordifferentclassifiersconsideringthedifferent sig-nals,Bio-Radar‘bR’andBIOPAC‘bP’,respectively.
5.3. Resultsafterfeatureselection
As mentioned previously, the Random Forest method also
allowstochecktherelevanceofthefeaturesinthedecision.Table5
showstheobtainedresults,foreachclassifierimplemented,with
themostrelevantfeatures.
IntheBio-RadarcasetheselectedfeatureswereFT6,FT7,FT12,
FT4,FT9andFT2(disposedindecreasingorderofimportance).In
theBIOPACcase,theselectedfeatureswereFT9,FT7,FT2,FT3,FT10
andFT8.Thereweresomecommonfeaturesusedbybothsystems,
suchasthevariance(FT2)andthepowerspectraldensityinthe
rangeof0.1–0.2Hz,(FT7)andintherangeof0.3–0.4Hz(FT9).These
featurescandepictmodificationsonthebreathingsignal
character-istics,i.e.,thechangeofthewaveformovertimeandthedynamic
rangevariationofthebreathingrateofahealthyperson,whichcan
occurduetorandommotionsortosuddenchangeofemotional
state(causedbylaughsorfrights).Itshouldbealsohighlightedthe
featuresthatwereselecteddifferentlyinthedifferentsignalcases.
Forexample,FT4,thatmeasuresthetimebetweenpeaksandFT12,
thatrepresentstheratiobetweenlowfrequencycomponentsand
highfrequencycomponents,wereonlyselectedontheBio-Radar
basedsystem.Thesefeaturescanencompasseithertheheartbeat
(mostlydetectedinapneaperiods),andrandombodymotionsthat
arenoteasilydetectedbytheBIOPAC,asseenpreviously.
Withthelimitednumberoffeatures,theresultsincreasedin
general(besideSVM for theBIOPACcase). Thisdeterminesthe
importanceoftheselectedfeaturesandprovestheirrelationwith
thecurrentemotionalstateofthesubject.
6. Conclusion
ThisworkaimedtovalidatetheusageoftheBio-Radarsystem
fortheemotionrecognitionthroughtheremotedetectionofvital
signs,inthiscasetherespiratorysignal.Threedifferentalgorithms
foremotionsidentificationwereappliedandcompared.Emotional
responseswereextractedfromvitalsignsacquiredsimultaneously
byanon-contactsystembasedonaCWDopplerradaranda
certi-fiedmeasuringequipment(BIOPAChereinusedasaconventional
method).Byanalysingbothsignalsvisually,itispossibletoobserve
thattheBio-Radarsystemwasmoresuccessfulonidentifyingthe
actualemotionalstateoftheindividual,thantheBIOPACsystem,
sinceitcandetectmoresubtlecharacteristics.Theexecuted
classi-fieralgorithmsweretheSVM,KNNandtheRandomForest,which
wereappliedtobothsignals.Regardingbinaryclassification,itwas
possibletoobserveadifficultytodistinguishbetweentheHappy
and the Fear emotional states, when compared withthe other
binarycases.Inbinaryandmulticlassclassificationproblems,the
BIOPACsignalpresentedbetterresults.Importantly,anddespite
themarginaldifferences,bothBio-RadarandBIOPACsignals
pre-sentedsimilarperformances,between65%and80%.Withina5%of
significancelevel,thesameclassifierusingthesedifferentsignals
havethesameaccuracy,accordingtoMcNemartestresult.
Gen-erally,theobtainedaccuracyresultsincreasedwhenthenumber
offeatureswaslimitedtothesixmoreimportantones,afterthe
featureimportanceevaluation.
In sum, thecurrent studycontributes toshowthat the
Bio-Radarcanindeedbeusedintheemotionrecognition,withasimilar
performancetothoseconventionalmethodsusedforvitalsignal
detection.Webelievethatitispossibletoenhancetheachieved
results,byimprovingtheset-upintwolevels.Firstly,regardingthe
monitoringscenario,thepresenceofmetallicsurfacesshouldbe
minimized,oncetheycorruptthereceptionoftheelectromagnetic
signals.Secondly,intermsofsignalprocessing,thedevelopment
ofstrategiestoremovetheDCcomponentinreal-timeandalso
totracktheoptimaldetectionpoint(i.e.,thecentreofthesubject’s
chest-wall),duringtheexperiment,cancontributefortheaccuracy
ofsignals’reception.Moreover,itisknownthatthecardiacsignal
couldalsobeseparatedfromtherespiratorysignal[26,27].Since
theirspectralcomponentsareclosedtoeachotherandthe
ampli-tudeofrespiratorysignalisaround10timesbiggerthanthecardiac
signal,theirseparationisachallengingtask.Thecardiacsignalhas
usefulinformationthatcouldhelpintheemotionalstate
identi-fication,thereforeitshouldbeexploredmethodsforanefficient
signaldecomposition.
Theseresultsrepresentanimportantsteptowardstheuseof
emotion recognitionsystemswithnon-invasive equipmentand
reinforces thenotionthat thesesystems representa promising
toolin severalcontexts,withmental healthcarerepresentinga
prime example,giventhepivotalroleof emotiondysregulation
inpsychiatricdisorders.Thistypeofsystemmayacttoprovide
individualizedprofilesand,therefore,moretailored-based
inter-ventions,hencereducingtheburdenofsuchdisorders.
Authors’contribution
Carolina Gouveia:conceptualization; software;investigation;
formalanalysis;datacuration;writing–originaldraft.
AnaTomé:conceptualization;methodology;validation;writing
-review&editing;supervision.
Filipa Barros:conceptualization; methodology;investigation;
resources;writing–review&editing.
SandraC.Soares:conceptualization;methodology;resources;
validation;writing–review&editing;supervision.
JoséVieira:methodology;validation;writing–review&
edit-ing;supervision.
PedroPinho:validation;writing–review&editing;supervision.
Acknowledgments
Thisworkis fundedbyFCT/MECthroughnationalfundsand
whenapplicableco-fundedbyFEDER–PT2020partnership
agree-ment under the project UID/EEA/50008/2019 and by National
PortugueseFundsthroughFCT–Fundac¸ãoparaaCiênciae
Tec-nologiaunderthePh.D.grantSFRH/BD/139847/2018.
Conflictofinterest:Nonedeclared.
References
[1]N.H.Frijda,Emotionexperienceanditsvarieties,Emot.Rev.1(3)(2009) 264–271.
[2]K.R.Scherer,Whatareemotions?andhowcantheybemeasured?Soc.Sci. Inform.44(4)(2005)695–729.
[3]L.F.Barret,Emotionsarereal,Emotion(Washington,DC)12(2012)413–429,
http://dx.doi.org/10.1037/a0027555.
[4]A.H.Fischer,A.S.R.Manstead,Socialfunctionsofemotion,Handb.Emot.3 (2008)456–465.
[5]S.Koelstra,etal.,Deap:adatabaseforemotionanalysisusingphysiological signals,IEEETrans.Affect.Comput.3(1)(2012)18–31.
[6]M.Raja,S.Sigg,ApplicabilityofRF-basedmethodsforemotionrecognition:a survey,2016IEEEInternationalConferenceonPervasiveComputingand CommunicationWorkshops(PerComWorkshops)(2016)1–6,http://dx.doi. org/10.1109/PERCOMW.2016.7457119.
[7]Q.Gao,J.Yan,H.Zhao,C.Ding,H.Hong,X.Zhu,Non-contactemotion recognitionviaCWDopplerradar,IEEEAsia-PacificMicrowaveConference (APMC)(2018)1468–1470.
[8]S.Katsigiannis,N.Ramzan,Dreamer:adatabaseforemotionrecognition throughEEGandECGsignalsfromwirelesslow-costoff-the-shelfdevices, IEEEJ.Biomed.HealthInformatics22(1)(2018)98–107.
[9]M.Zhao,F.Adib,D.Katabi,Emotionrecognitionusingwirelesssignals, Proceedingsofthe22ndAnnualInternationalConferenceonMobile ComputingandNetworking3(1)(2016)95–108.
[10]L.Anishchencko,Challengesandpotentialsolutionsofpsychophysiological statemonitoringwithbioradartechnology,Diagnostics8(4)(2018)73.
[11]J.Fernandez,L.Anishchencko,Mentalstressdetectionusingbioradar respiratorysignals,Biomed.SignalProcess.Control43(2018)244–249.
[12]C.Gouveia,D.Malafaia,J.N.Vieira,P.Pinho,Bio-radarperformanceevaluation fordifferentantennadesigns,URSIRadioSci.Bull.2018(364)(2018)30–38.
[13]C.Gouveia,Bio-Radar,UniversidadedeAveiro,Aveiro,Portugal,201712 (Master’sthesis).
[14]C.Gouveia,J.N.Vieira,P.Pinho,Areviewonmethodsforrandommotion detectionandcompensationinbio-radarsystems,Sensors19(3)(2019)604.
[15]J.Groot,etal.,Asniffofhappiness,Psychol.Sci.26(6)(2015)684–700.
[16]J.Ferreira,V.Parma,L.Alho,C.F.Silva,S.Soares,Emotionalbodyodorsas context:effectsoncardiacandsubjectiveresponses,Chem.Senses43(5) (2018)347–355.
[17]G.Yuan,N.A.Drost,R.A.McIvor,Respiratoryrateandbreathingpattern, McMasterUniv.Med.J.10(1)(2013)23–25.
[18]V.L.Clark,J.A.Kruse,Clinicalmethods:Thehistory,physical,andlaboratory examinations,JAMA264(21)(1990).
[19]O.Gal,EllipseFitUsingLeastSquaresCriterion,2018(Accessed)http://it. mathworks.com/matlabcentral/fileexchange/3215-fit-ellipse.
[20]E.Alpaydin,IntroductiontoMachineLearning,TheMITPress,Massachusetts InstituteofTechnology,Cambridge,MA,2004.
[21]P.-N.Tan,M.Steinbach,A.Karpatne,V.Kumar,IntroductiontoDataMining, 2nded.,Pearson,2018.
[22]Mathworks,Treebagger,2019(Accessed)https://www.mathworks.com/help/ stats/treebagger.html.
[23]J.Panksepp,AffectiveNeuroscience:TheFoundationsofHumanandAnimal Emotions,NewYorkOxfordUniversityPress,2004.
[24]N.R.Giuliani,K.McRae,J.J.Gross,Theup-anddown-regulationofamusement: experiential,behavioral,andautonomicconsequences,Emotion8(5)(2008) 714.
[25]T.M.Mitchell,MachineLearning,McGrawHill,1997.
[26]N.Petrochilos,M.Rezk,A.Høst-Madsen,V.Lubecke,O.Boric-Lubecke,Blind separationofhumanheartbeatsandbreathingbytheuseofadopplerradar remotesensing,IEEEInternationalConferenceonAcoustics,Speechand SignalProcessing1(2007)1–333.
[27]H.Zhang,etal.,Theseparationoftheheartbeatandrespiratorysignalofa DopplerradarbasedontheLMSadaptiveharmoniccancellationalgorithm, 6thInternationalSymposiumonComputationalIntelligenceandDesign1 (2013)362–364.
