Automatic calibration of the CAMBADA team vision system

2008 Informáti a

Ivo dos Santos

Pinheiro

Calibração Automáti a do Sistema de Visão da

Equipa CAMBADA

Automati Calibration of the CAMBADA Team

2008 Informáti a

Ivo dos Santos

Pinheiro

Calibração Automáti a do Sistema de Visão da

Equipa CAMBADA

Automati Calibration of the CAMBADA Team

Vision System

dissertação apresentada à Universidade de Aveiro para umprimento dos

requisitosne essáriosàobtençãodo graude MestreemEngenhariade

Ele -tróni ae Tele omuni ações,realizadasoba orientação ientí a doDoutor

Manuel Bernardo Salvador Cunha, Professor Auxiliar do Departamento de

Ele tróni a,Tele omuni açõeseInformáti adaUniversidade deAveiroedo

DoutorAntónio JoséRibeiroNeves,ProfessorAuxiliarConvidadodo

Depar-tamentodeEle tróni a,Tele omuni açõeseInformáti adaUniversidadede

presidente Doutor Paulo Jorge dos Santos GonçalvesFerreira

professor atedráti odaUniversidadedeAveiro

Doutor Luis Paulo Reis

professorauxiliar daFa uldadedeEngenhariadaUniversidadedoPorto

Doutor Manuel Bernardo Salvador Cunha

professorauxiliar daUniversidadedeAveiro

Doutor António José Ribeiro Neves

daram a levar esta dissertação a bom termo. Quero agrade er

espe ial-mente ao professor António José Neves, o-orientador da dissertação, pela

sua onstante disponibilidade, pa iên ia e auxílio impres indível na

resolu-ção de todosos problemas que fuien ontrando ao longodesta dissertação.

Quero agrade er tambémao professorManuel Bernardo Cunha, orientador

da dissertação, om quem dis uti alguns algoritmos de alibração de ores

e de âmaras. Um agrade imento a todosos elementos do proje to

CAM-BADA, em espe ial ao grupo da visão, os quais durante a preparação, e

durante as ompetiçõesmantiveramum onstanteespíritode amaradagem

dire ionais, alibração de âmarasdigitais, alibração de ores

resumo O re urso a amâras digitais em apli açõesrobóti as tem vindo a res er

signi ativamente nos últimos anos. As prin ipais áreas de apli ação

des-tes robs são a indústria e as apli ações militares,nosquais estas amâras

são usadas omo sensorquepermiteao rob retirarainformação relevante

do ambiente envolvente, a qual lhe permite tomar de isões. Para extrair

informação de uma imagem, omo a forma ou a or de obje tos, é

impor-tante que os parâmetros da âmara se en ontrem bem alibrados. Esta

dissertação estáinseridano proje toCAMBADA(CooperativeAutonomous

roBots w/ Advan ed Distributed Ar hite ture),desenvolvido pelo

Departa-mento de Ele tróni a, Tele omuni ações e Informáti a da Universidade de

Aveiroeapresenta um onjunto dealgoritmospara alibraçãodosprin ipais

parâmetrosde uma âmara digital,bem omoa alibração de oresde

inte-resseasso iadasaumadeterminadaapli açãodas âmaras. Estadissertação

propõe doisalgoritmosque permitem aos robs, de forma automáti a,

a-libraros parâmetros das âmarasdo seu sistemade visão. Estes algoritmos

baseiam-se em informação extraída das imagens adquiridas pelas âmaras

e na utilização de ompensadores. São propostos ainda três métodos para

alibraras ores,doisdelesdeformamanualmasintuitivosparaoutilizador,

e porúltimoumalgoritmoautomáti obaseadoemdete çãode ontornose

amera alibration, olor alibration

Abstra t In the past few years the use of digital amerasin roboti appli ationshas

been in reasingsigni antly. The main areas of appli ation of this robots

arethe industry and militaryappli ations, where these amerasare used as

a sensor that allows the robot to take the relevant information of the

sur-roundingenvironment,allowingittomakede isions. Toextra tinformation

from the a quired image, su h as shapes or olors, the amera alibration

pro edureisveryimportant. ThisthesisisinsertedintheCAMBADA

(Coop-erative Autonomous Mobile roBotsw/ Advan ed Distributed Ar hite ture)

proje t, developed by the Department of Ele troni s, Tele ommuni ation

and Informati softhe Universityof Aveiro, and presents several algorithms

to alibratethemostimportantparametersofadigital ameraandthe olors

ofinterestforaspe i appli ation. Thisthesisproposestwoalgorithmsthat

allow the robots to alibrate autonomously the parametersof their amera

systems. These algorithmsarebased onthe informationextra tedfromthe

a quiredimageandintheuseof ontrollers. Italsoproposesthreemethods

for olor alibration, two of them are manual butintuitiveto the user, and

the last one an automati algorithm based on edge dete tion and region

1 Introdu tion 1

1.1 The RoboCup MSL ase . . . 2

1.2 Thesis ontributions . . . 4 2 Digital Cameras 5 2.1 CCD sensor . . . 5 2.2 Camera parameters . . . 5 2.2.1 White-balan e . . . 6 2.2.2 Brightness . . . 7 2.2.3 Contrast . . . 7 2.2.4 Gain . . . 8 2.2.5 Exposure . . . 8 2.2.6 Shutter . . . 9 2.2.7 Gamma . . . 9 2.2.8 Saturation . . . 9 2.2.9 Hue . . . 10 2.3 Color spa es . . . 11 2.3.1 RGB . . . 11 2.3.2 YUV . . . 11 2.3.3 HSV . . . 14

3.1 Camera alibrationalgorithm . . . 18

3.2 Auto alibration of amera parameters usingPI ontrollers . . . 21

3.2.1 Run-time auto- alibration . . . 26

4 Color alibration 31

4.1 Automated algorithm for olor alibration . . . 33

5 Con lusion 37

A AutoCalibration manual 39

A.1 Introdu tion . . . 39

A.2 Howto usethe appli ation . . . 40

B LibAutoCalib 43

Introdu tion

In the past few years the use of digital ameras in roboti appli ations has been in reasing

signi antly. The main areas of appli ation of this robots are the industry and military

appli ations, where these ameras are used as a sensor that allows the robot to take the

relevant information of thesurroundingenvironment,allowing itto make de isions.

To extra t information from the a quired image, su h as shapes or olors, the amera

ali-bration pro edure is very important. If the amera is wrongly alibrated, the image details

are lost and itis impossible to re ognize anything basedon shape or olor,as an be seenin

Fig. 1.1.

Figure 1.1: On the left, an example of an image a quired by one of the CAMBADA team

robots withits amera wrongly alibrated. Ontheright, animage orre tly alibrated.

The major appli ations of ameras in roboti s vision o ur within ontrolled environments,

whi h means that the intensity and the type of light is always onstant. In this ase the

parametersof the amerasonly needto be adjustedon ebe ause theenvironment luminan e

van edDistributedAr hite ture)proje t,developedbytheDepartmentofEle troni s,

Tele om-muni ation and Informati s of the University of Aveiro [1℄. CAMBADA is a team of so er

robotsthatparti ipatesintheMiddleSizeLeague(MSL)ofROBOCUP[2℄. Itisa

multidis i-plinaryproje t,thatinvolvesknowledgeofme hani s,arti ial intelligen e, powerele troni s,

instrumentation, omputer ar hite ture, tele ommuni ations, software engineer, arti ial

vi-sion and ooperation, amongothers.

CAMBADA robots have a hybrid vision system integrating an omnidire tional and a

per-spe tive amera. Theomnidire tional partof thevision system[3℄isbased on a atadioptri

onguration implemented with a rewire amera (with a 1/3 CCD sensor and a 4.0mm

fo aldistan e lens) anda hyperboli mirror. Theperspe tive amera uses a low ost rewire

amera (BCL1.2Unibrain amera witha1/4 CCDsensorand a3.6mmfo aldistan elens).

The omnidire tional amera works at 30 frames per se ond (fps) in YUV4:2:2 mode with a

resolution of

640 × 480

pixels. The front amera works at 30 fpsin YUV4:1:1 mode with a resolution of

640 × 480

pixels.

Theomnidire tionalvision systemisusedto ndtheball,to dete tthepresen eofobsta les

andwhitelines. Inthe aseoftheMSL ompetitionofRoboCup,mostoftheteamsadoptthis

approa h for the vision system of the robots. The perspe tive vision is used to nd the ball

and obsta les in front of the robot at larger distan es,whi h aredi ult to dete tusing the

omnidire tional vision system, due to its very limited spa ial resolution at distan es greater

than 5 to 6meters.

Asetofalgorithmshasbeendevelopedtoextra tthe olorinformationofthea quiredimages

and, in a se ond phase, extra t the information of all obje ts of interest. The vision system

ar hite ture uses a distributed paradigm where the main tasks, namely, image a quisition,

olor extra tion, obje tdete tion and image visualization, areseparatedinseveral pro esses.

E ient olorextra tionalgorithmshavebeendevelopedbasedonlookuptablesandaradial

modelforobje tdete tion. Thevisionsystemisfastanda uratehavinga onstantpro essing

time independent fromtheenvironment around the robot [4℄.

1.1 The RoboCup MSL ase

The ultimate goal of the RoboCup proje t is, by 2050, develop a team of fully autonomous

humanoidrobotsthat anwinagainstthehumanworld hampionteaminso er[2℄. Itmeans

in a near future, the robots will have to play under natural light onditions and inoutdoor

system.

under ontrolled lighting onditions, as is the ase of arti ial illumination, as well in

non- ontrolled lighting onditions, su h as in outdoor elds. In outdoor elds the illumination

an hange slowly during the day, due to the movement of the sun, as well asfast in short

periods oftime dueto a partial andtemporally overing ofthesun by louds. It means that

the robots,have to adjustthe olors, inreal time,its olor segmentation values aswell asits

amera parameters to adaptto newlighting onditions [5 ℄.

Image analysis in the RoboCup domain is simplied, sin e obje ts are olor oded. Bla k

robotsplaywithanorangeballonagreeneldthathaswhitelines. Thus,the olor ofapixel

isastronghintforobje tdete tion. Thisfa t anbeexploited bydening olor lasses,using

alook-uptable(LUT)forfast olor lassi ation. Thetable onsistsof

16 777 216

entries(one of ea h

2

24

possible olor ombinations - 8 bits for red, 8 bits for greenand 8 bits for blue).

Ea h entry is8bitswide,whi hmeans thattheLUTo upies16MBytesintotal. Ifanother

olorspa eisused,thetablesizeisthesame, hangingonlythemeaning ofea h omponent.

Ea h bit expresses whether the olor is within the orresponding lass or not. This means

that a ertain olor an be assigned to several lasses at the same time. To lassify a pixel,

werst readthe pixel's olor and thenusethe olor asanindex into thetable.

The olor alibrationisperformedinHSV (Hue,Saturationand Value) olorspa e dueto its

spe ial hara teristi s. In the urrent setup, the image is a quired in RGB or YUV format

andthenis onvertedto an image oflabels using theappropriateLUT [4 ℄.

Asfar aswe an understand from thepublished work madebytheother teams ofRoboCup

MSL, most of them don't have any softwareto auto alibrate the amera, whi h means their

a quiredvideo.

In the team des ription papers of Brainstormers Tribots [6 ℄, NuBot [7℄, Te h United [8℄

andRobofoot ÉPM[9 ℄, some auto alibrationalgorithms arementioned, although they don't

present anydetailsabout them.

1.2 Thesis ontributions

Untilthebeginningofthiswork,theCAMBADAteamdidn'thaveanautomati appli ationto

alibratethe visionsystem. The amera parameters wereadjusted manuallyandthis pro ess

usually took mu h time and required an expert person. The olor alibration was made by

a quiring a video and, in oine mode, thevideo waspro essed and the olor rangefor ea h

olor was adjusted.

This thesis proposes two algorithms for amera parameters alibration, both automati ally

performedbythe robots. The rstone obtainssatisfa toryresults,but presentssome ontrol

problemsalongthetime. Moreover,itdoesn'tuseanymeasureofqualitytoensurethe orre t

alibration and itdoesn't evaluate thewhole image.

The se ondproposedalgorithm attainsbetter resultsandhassolved theproblems referredto

in the rst algorithm. It's based on a PI ontroller, responsible for the amera parameters

update,andusesasetofmeasures,namelyACM,Average,EntropyandMSVtoevaluatethe

quality of the a quired images. This improves the obtained results and guarantees that the

images have nearlythesame hara teristi s. Experimental resultsobtained bythisalgorithm

show that the quality measures of the image a quired always onverge to the same values.

Moreover, it is possible to apply this algorithm inrun-time inorder to guarantee that, even

whilevaryingtheilluminationofthe environment,therelevant olorsoftheimagewillremain

the same.

A method to alibrate olors using a HSV histogram is also proposed. It allows to visualize

easily what isthe olor rangeofa ertain olor lassandifapixel isinorout of thesele ted

olor range. This method uses the histogram of the three omponents, hue, saturation and

value tosele t the desired olor rangefor ea h olor lass.

Finally an automati method to alibrate the olor lasses using a anny edge dete tor is

also proposed. For ea h region obtained after applying the edge dete tor, a region growing

algorithm is applied inorder to obtain the olor range of thatregion. Ea h olor lass hasa

Digital Cameras

A digital amera is a devi e that aptures video or photographs digitally by re ording the

images aptured by a light sensitive sensor. It is formed basi ally by a lens, a olor sensor,

software for image treatment and hardware apable of pro ess and transmit the aptured

images,usually designated byframes[10, 11 ℄.

2.1 CCD sensor

Mostofthe olorsofthevisiblespe trum ouldbereprodu edbyaddingdistin tpartsofred,

greenandbluelight. ThispropertyisusedbymostofCCDsensorsto apture olorimages. It

onsistsinasquaregridof apa itors,whi h are hargedbyaphotosensitiveelement overed

by a lter that only allows one omponent of light (red, green and blue) to rea h thesensor

(seeFig.2.1). Theimageisformedmeasuringthe hargeofea h apa itorbya ontrol ir uit.

For ea hpixel, the CCDdoesn't a quiretheinformation of thethree omponents. Instead,a

Bayerpattern is used. It is a repeating

2 × 2

mosai pattern of light lters, withgreen ones at opposite ornersand redand blue inthe othertwo positions. The predominan e of green

takesadvantageofpropertiesofthehumanvisualsystem,whi hdeterminesbrightnessmostly

from greeninformation and isfar more sensitiveto brightness than tohue or saturation.

2.2 Camera parameters

For any parti ular appli ation the quality of an image a quired by a amera is dependent

parameters. Inthe followingse tionsitwillbeexplainedthe ommonparameters ofa amera

and their ee tsinthe image.

2.2.1 White-balan e

Thisparameterisoneofthemostimportant ongurableparametersofthe amera. Theimage

olorsappeardierent dependingontheilluminationunder whi htheimage wastaken. This

isduetothefa tthatdierentlightsour eshavedierent olortemperatures. Theadjustment

ofthisparametershouldbeperformedinorderto makeawhitesour etoappearwhiteonthe

image under dierent light onditions. Usually, when this parameter hasn't been adjusted,

the images have a red or blue tonality, as presented in Fig. 2.2. The ompensation of this

ee t an either be automati ally performed by the amera, or an be made manually. The

the white-balan e. Onthe left,an image a quired witha high value of thebluegain. Inthe

enter, animagewiththewhite-balan e orre tly alibrated. Ontheright,animage a quired

withahighvalueof the redgain.

2.2.2 Brightness

This parameter adjusts the bla k level of an image, whi h means that an oset is added or

subtra tedforea hpixel. Thisparameter,inthelimit, ouldleadtoimagesbrighterordarker

(see Fig.2.3). Usually,this parameter mustbeset manually.

Figure 2.3: Examples of images a quired in the roboti so er eld with dierent values of

brightness. On the left, an image a quired witha low value of brightness. In the enter, an

image orre tly a quired. Ontheright, animage a quired witha highvalue ofbrightness.

2.2.3 Contrast

This parameteris responsible to turnthe bright olorsmore bright and thedark olors more

dark. In a high ontrast image, edges an be seen more learly and thedierent elements of

an image area ented. Ina low ontrast image,thebrightness ofdierent elements isnearly

ontrast. Ontheleft,animagea quiredwithalow ontrast. Inthe enter,animage orre tly

a quired. Onthe right,an image a quiredwith ahigh ontrast.

2.2.4 Gain

Thisparameterisusuallyimplementedbyhardware. Whenthe hargeofea hCCD apa itor

ismeasured,itsvalueisampliedbyhardware,byagain fa torbeforethequantizationstep.

In reasing this fa tor makes the image brighter and in reases the ontrast but addsnoise to

the image,due to the fa tthattheoriginal noise of theimage isalso amplied. Usually,this

parameter an be set to the automati mode, whi h means that the amera automati ally

ontrol thevalue of thegain by an algorithm that evaluates thebrightness of the last frame

(seeFig.2.5).

Figure 2.5: Examples of images a quired in the roboti so er eld with dierent values of

gain. Ontheleft,animagea quiredwithalowvalueofgain. Inthe enter,animage orre tly

a quired. On the right,an image a quiredwith ahighvalueof gain.

2.2.5 Exposure

Exposure is the total amount of light allowed to fall on the image, whi h means that it is

relatedwiththetime thatthe CCDsensorisexposedtothelight inea hframe. Ahighvalue

of exposurewill leadto images more bright, whi h is thesame ee t ofthe gainbut without

Fig. 2.6).

Figure 2.6: Examples of images a quired in the roboti so er eld with dierent values of

exposure. On the left, an image a quired with a low value of exposure. In the enter, an

image orre tly a quired. Onthe right, animage a quired witha highvalue ofexposure.

2.2.6 Shutter

The shutterspeed determinestheamount oftimethattheshutter ofa amera isopened and

is therefore similar to the exposure parameter. The adjustment of this parameter allows to

ontrol themovement of anobje tinas ene. A lowshutterwill freeze theobje tand ahigh

shutter will makeit looksblurred astheobje t moves(see Fig.2.6).

2.2.7 Gamma

Thisparameter anbeimplementedbyhardwareor softwareinthe amera. Itisusuallyused

to for e an image to have the same visual aspe t on dierent monitors. For ea h pixel, the

operation

P c

= P o

1

gamma

is performed, where

P c

and

P o

are the orre ted and the original pixelvalues respe tively. The ee t of this parameter is to turn darker or brighter the dark

pixels relatedto thebright pixels. The adjustment of this parameter isusually set manually

(see Fig.2.7).

2.2.8 Saturation

Thesaturationofa olorreferstohowvibranta oloris. Alowsaturated olorisneartogray,

leading to an image where the olors look washed out. On the opposite, a high saturated

gamma. Onthe left,an image a quiredwith a low value of gamma. On theright, an image

a quiredwitha highervalueof gamma.

Figure 2.8: Examples of images a quired in the roboti so er eld with dierent values of

saturation. On the left,an image a quired witha low value of saturation. In the enter, an

image orre tly a quired. Onthe right, animage a quired witha highvalue ofsaturation.

2.2.9 Hue

This parameter allows to introdu e an oset in the olors spe trum. Usually, the amera

doesn't usethis parameter bydefault and doesn't have anautomati mode (see Fig.2.9).

Figure 2.9: Examples of images a quired in the roboti so er eld with dierent values of

hue Ontheleft,an image a quiredwitha lowvalueofhue. Inthe enter, animage orre tly

A olor spa e is a mathemati al model to digitally represent the olors. There exist many

olor spa es. Ea h one has dierent hara teristi s and is suitable for dierent appli ations.

Some of them use the three primary olors (red, green and blue) while others use dierent

on epts, su h asthe ase ofhue, saturation and value, related to the human visual system,

orluminan e and two hromati omponents, su h astheone usedfor television [13,14℄.

2.3.1 RGB

ThenameRGB omesfromRed,GreenandBlueinitials, thatarethethreeemissiveprimary

olors. This olorspa eusestheadditivepropertiesofthe olorsallowing toobtainalmostall

possible olorsofthe visiblespe trum. This olor spa eisusedmainly on omputergraphi s,

aswell inTV and video. This olor spa e isn't intuitive to human per eption, but have the

advantage to be very simple to implement and ompute. The RGB ould be visualized as

a ube with three axis, ea h one orresponding to red, green and blue (see Fig. 2.10). The

bottom orner, when

red

= green = blue = 0

is bla k, while on the opposite orner, where

red

= green = blue = 255

(fora 8 bit per hannel display system) iswhite. In thediagonal

ve torfrom bla kto white, arerepresentedthe gray levels, hara terized byhaving thesame

amount ofea h primary olor.

Figure2.10: The RGB olor spa erepresentation [15 , 13 ℄.

2.3.2 YUV

The YUV olor spa e separates the information of a olor into its luminan e (Y) and two

Mémoire (SECAM) omposite olor video standards. The biggest advantage of this olor

spa e relatively to theRGB is that the human visual per eption (in terms of both intensity

andspa ialresolution)oftheY omponentisgreater thantheper eptionoftheUandV

om-ponents. Duetothisknowledge,the UandV omponentsusuallyaresub-sampled(Fig.2.12)

and theimagerequiresasmallerbandwidthtobetransmitted anitismoree iently stored.

This pro ess introdu es some loss of quality in the re onstru ted image but it is assumed

that the human eye an't distinguish the dieren es. This is used as a rst step for image

ompression.

Figure2.11: The U-V olor plane onsidering Y=127 [16 ℄.

To onvert RGB to YUVthe following equations areused[13 ℄:

Y

= (0.257 ∗ R) + (0.504 ∗ G) + (0.098 ∗ B) + 16

V

= (0.439 ∗ R) − (0.368 ∗ G) − (0.071 ∗ B) + 128

U

= −(0.148 ∗ R) − (0.291 ∗ G) + (0.439 ∗ B) + 128

To onvert YUVto RGB the following equations areused[13 ℄:

R

= 1.164(Y − 16) + 1.596(V − 128)

G

= 1.164(Y − 16) − 0.813(V − 128) − 0.391(U − 128)

B

= 1.164(Y − 16) + 2.018(U − 128)

Sub-sampling s hemes

Be ause thehuman eye is less sensitive to the hrominan e than the luminan e, bandwidth

ould be optimized by storing more information about the luminan e than about

modeYUV420is a quiredusing aplanarrepresentation,i. e. the amera returnsea himage

omponent (thefullimage) at atime.

YUV444: ea h omponent has the same resolution without lossof information. The

infor-mation a quiredbythe amera hasthefollowing mapping:

Y

0

U

0

V

0

Y

1

U

1

V

1

Y

2

U

2

V

2

Y

3

U

3

V

3

Ea h pixelwill be mapped a ordingto thefollowing:

[Y

0

U

0

V

0

] [Y

1

U

1

V

1

] [Y

2

U

2

V

2

] [Y

3

U

3

V

3

]

YUV422: the U and V omponents are spatially sampled half times relatively to the Y

omponent, whi h means that the horizontal resolution is an half than the YUV444.

The information a quired bythe amera hasthe following mapping:

U

0

Y

0

V

1

Y

1

U

2

Y

2

V

3

Y

3

Ea h pixelwill be mapped a ordingto thefollowing:

[Y

0

U

0

V

1

] [Y

1

U

0

V

1

] [Y

2

U

2

V

3

] [Y

3

U

2

V

3

]

YUV411: thehorizontal resolution ofU andV omponents isdividedbyfour. Thevideo in

this format uses 6bytesto storeea h ma ropixel (onere tangle with

1 × 4

pixels).

U

0

Y

0

Y

1

V

2

Y

2

Y

3

Ea h pixelwill be mapped a ordingto thefollowing:

[Y

0

U

0

V

2

] [Y

1

U

0

V

2

] [Y

2

U

0

V

2

] [Y

3

U

0

V

2

]

YUV420: the horizontal and theverti al resolution of U and V omponentsare divided by

two. The video in this format uses 6 pixels to store ea h ma ropixel (onesquare with

2 × 2

pixels).

InFig.2.12,itispresentedagraphi alrepresentationoftheseveralYUVsub-samplings hemes

[17 ℄.

2.3.3 HSV

The olor re eptors in thehuman eye, known as ones, fromdierent areas of thespe trum,

withthegreatestsensitivityintheblue,greenandredpartofit. The olorinformationsignals

a quiredbythe onesarethenfurther pro essedinthevisual system. Nevertheless, aperson

annotmakeintuitiveestimatesoftheblue,greenandred omponentsofanyparti ular olor.

Onthe other hand, inthe per eption pro ess, a human an easily re ognize basi attributes

of olorsu hasintensity(brightness,lightness)I, saturationSandhue H.Thehuerepresents

the impression related to the dominant wavelength of the olor stimulus. The saturation

orresponds to relative olor purity. Colors with zero saturation are gray levels. Maximum

intensityissensedaspurewhite, minimumintensityaspurebla k. TheH,S,I omponentsof

theHSI olormodelare al ulatedfromformulaexpressingapproximately thepsy hophysi al

senseof these notions fromthe RGB oordinate systemto a ylindri al modelof per eptions

(seeFig.2.13a))[18 ℄.

Twootherderivations ofthegeneri HSI olorspa eareapplied usuallyin omputergraphi s

and image pro essing: HSV and HLS. Figures 2.13b) and 2.13 ) illustrate the geometri

interpretation of thesemodels. Theydierof theoriginal HSImodel intheexpression ofthe

intensity and saturation values. For the HSV model the olors be ome less saturated when

the intensity approa hes minimal level. In HLS the olors be ome less saturated when the

intensityapproa hes minimalor maximal levels. Important advantages of theHSI,HSV and

HLSmodelsoverother olorspa esaregood ompatibility withthehumanintuitionof olor.

They are intuitive olor spa es and allow an easy separation between hromati values and

a hromati values.

For image pro essing, a olor system based on the human per eption of olor (HSI, HLS or

Figure2.13: Graphi al representation of a)HSI,b) HSV and )HLS olor spa es [18℄.

ifthe image sour e is a quired using a YUV sampling s heme with lower resolution for the

hromati signals these models also allow an easy preservation of the luminan e omponent

intheform ofthe I,V orL signals. Dueto this fa t,intheeld ofroboti appli ations, HSV

isthe olorspa e most usedfor olor lassi ation whi h simpliesand makesmorea urate

the sele tion of olor ranges for ea h olor of interest[14 ℄. This olor spa e is represented in

Fig.2.14. .

Figure 2.14: Coni al representation oftheHSV olor spa e[19 ℄.

In this work it is used the HSV olor spa e. To onvert RGB to HSV it used the following

equations:

hǫ[0, 359]

sǫ[0, 1]

vǫ[0, 1]

max

= max(r, g, b)

min

= min(r, g, b)

h

=



































0

,ifmax=min

60

◦

_×

g−b

max−min

+ 0

◦

,ifmax=r andg

≥

b

60

◦

_×

g−b

max−min

+ 360

◦

,ifmax=r andg

<

b

60

◦

_×

g−b

max−min

+ 120

◦

,ifmax=g

60

◦

_×

g−b

max−min

+ 240

◦

,ifmax=b

s

=

(

0

,ifmax=0

max−min

min

,otherwise

v

= max

To onvert HSV to RGB itusedthefollowing equations:

h

i

= ⌊

60

h

⌋ mod 6

f

=

h

60

− ⌊

h

60

⌋

p

= v × (1 − s)

q

= v × (1 − f × s

t

= v × (1 − (1 − f ) × s)

Compute olor ve tor (r,g,b)

(r, g, b) =















































(v, t, p)

,if

h

i

= 0

(q, v, p)

,if

h

i

= 1

(p, v, t)

,if

h

i

= 2

(p, q, v)

,if

h

i

= 3

(t, p, v)

,if

h

i

= 4

(v, p, q)

,if

h

i

= 5

Auto alibration of amera parameters

The alibration of the amera is ru ial for olor-based obje t dete tion and has to be

per-formedwhen environmental onditions hange, for examplewhen usingdierent so erelds

orwhen lighting onditions hanges.

Using the amera in auto mode has several problems and annot be used is some roboti

appli ations. In the aseoftheCAMBADArobots anddue to thefa tthattheenvironment

aroundtheserobotshasspe i hara teristi s, su hasthegreeneldand thebla kbodyof

the robotsthat lls the most partof the image (see Fig.3.1), theauto mode of the ameras

donot perform well.

Figure3.1: An exampleof amaskusedto sele tthepixelsto be pro essedbythe

omnidire -tional visionsystem. White points representsthearea thatwill be pro essed.

The image a quired bythe amera inauto mode isoverexposeddue to thelarge bla k areas

existing in the image and the white-balan e is not orre tly alibrated, leading to erroneous

olors intheimage. Thisisdue to thefa tthatmost of amerasusealgorithms thatassume

ized person. This alibrationpro ess usuallytookabout10 minutesfor thefull onguration

of ea h robot's vision system. So an algorithm that alibrates automati ally the amera is

a great help to optimize the pro ess. In the following it will be explained the developed

algorithms and dis ussed theexperimentsresults obtained.

Figure3.2: Anexample of an imagea quired bythe amera of one robot oftheCAMBADA

team.

3.1 Camera alibration algorithm

The rst developed algorithm doesn't analyze the entire image but uses a sele tion of two

areas with spe i hara teristi s, namely a bla k and a white areas. It only adjusts the

white-balan e, the gain and brightness parameters. It is assumed that, when the amera is

alibrated,the sele ted areashave thefollowing hara teristi s:

•

thewhite areashould be white:

 in the YUV olor spa e this means that the average value of U and V should be

127. Ifthewhite-balan eisnot orre tly ongured, thesevaluesaredierentfrom

127 and theimage doesnothave the orre t olors;

 intheRGB olorspa etheaveragevalueshouldequalforthethree omponentsand

near(255,255,255). Ifnot,theentire imageiseithertoodarkandgainparameter

needs to beadjusted, or there is apredominan e of one of theprimary olors and

the white-balan e needsto be adjusted;

•

the bla k area should be bla k  in the RGB olor spa e this means that the average

values of R,G and B should be lose to zero. Ifthe brightness is too high we observe

Thealgorithm performsasfollows:

1. A newframe is a quiredbythe amera.

2. Forthe sele tedbla kandwhiteareas,itis al ulatedthemaximum,minimumandthe

averagevaluesofY,UandV inthe YUV olor spa eandR,GandBintheRGB olor

spa e.

3. The adjustment of white-balan e is made based on the average value of white area in

theYUV olorspa e. IftheUvalueissmallerthan127thebluegainofwhite-balan eis

in remented,elseifitisgreaterthan127thebluegainisde remented. Thesameismade

for redgain ofwhite-balan e, butusing asreferen ethemean value ofV omponent of

thewhite area.

4. The adjustment of the gain parameter of the amera is based by the maximum and

minimum values of the R, G and B values in the white area. If any of the maximum

values is equal to 255 the gain will be de remented. On the other ase, if any of the

minimumvalues issmallerthan 200 thegainis in remented.

5. The adjustment of brightness is performed using the bla k area and it is adjusted in

order to guaranty thattheblue value intheRGB olor spa eis smallerthan 5 and, at

the same time, guaranty that any of the mean values of R, G and B never is greater

than or equal to 2.

The hangesintheparameters,asdes ribedabove,aremadeusingsomepredened onstants

obtained experimentally. Thisisa possible drawba k ofthealgorithm.

Theexperimentalresultsshowsthatthisalgorithmpresentsasatisfa toryperforman e.

How-ever it also shows some problems, namely it takes some time to onverge and only partial

evaluation of the image to obtain useful information. Moreover, it depends on the initial

parameters set to the amera. Some results obtained with this algorithm are presented in

0

100

200

300

400

500

600

700

800

0

200

400

600

800

1000

1200

1400

1600

Value

Frame

WB_RED

WB_BLUE

Exposure

Gain

Brightness

200

300

400

500

600

700

800

900

1000

1100

0

100

200

300

400

500

600

700

800

Value

Frame

WB_RED

WB_BLUE

Exposure

Gain

Brightness

Figure3.4: Experimentalresults usingthe automated alibration pro eduredes ribed inthis

se tion. On the left, results obtained starting with all the parameters of the amera set to

zero. On the right, results obtained with all theparameters set to the maximum value. On

the top, the initial image a quired. In the middle, the image obtained after applying the

automated alibrationpro edure. Thelastrow ontains graphi sshowingtheevolution ofthe

This se tionpresents an evolutionof the algorithm des ribed intheprevious se tion. It also

usestworeferen eareasintheimage, namelyawhiteareato alibrate thewhite-balan e and

abla kareato alibratethebrightness. However,thehistogram oftheintensitiesoftheimage

isusedto alibrate the exposure andthegain ofthe amera.

Anhistogramof theintensitiesofan imageisagraphi alrepresentationoftheintensitypixel

valueshownasbars orrespondingtothe ountoftheo urren esofallpossiblevaluesinthe

image. For animage representedusing8bits perpixel, thepossiblevaluesarebetween 0and

255. Image histograms an indi ate the nature of the light onditions, the exposure of the

imageand whetherit isunderexposedor overexposed.

Theassumptions usedbytheproposedalgorithm are:

•

the white area should be white  inthe YUV olor spa e this means that the average

value ofU and V should be 127. Ifthe white-balan e is not orre tly ongured, these

values aredierent from 127and theimage doesnot have the orre t olors;

•

the bla k area should be bla k  inthe RGB olor spa e, this means that the average

valuesofR,GandBshouldbe losetozero. Ifthebrightness istoohigh,itisobserved

that the bla kregion be omesblue, resulting ina degradation oftheimage;

•

thehistogramofintensitiesshould be enteredaround 127andshouldspanallintensity

values. Dividingthehistogram into regions,theleftregionsrepresentdark olors,while

the right regions represent light olors. An underexposed image will be leaning to the

left,while anoverexposedimage willbeleaning to theright inthehistogram.

Statisti almeasures anbeextra tedfromdigitalimagestoquantifytheimagequality[20 ,21 ℄.

A number of typi al measures used in the literature an be omputed from the image gray

level histogram,namely:

•

Mean:

µ

=

N −1

X

i=0

iP

(i);

(3.1)

•

Entropy:

E

= −

N −1

X

i=0

P

(i)log(P (i));

(3.2)

•

Absolute Central Moment (ACM):

ACM

=

N −1

X

i=0

|i − µ|P (i);

(3.3)

•

Mean Sample Value(MSV):

M SV

=

P

4

j=0

(j + 1)x

j

P

4

j=0

x

j

;

(3.4)

where

N

is the number of possible gray values inthe histogram (typi ally, 256),

P

(i)

is the probabilityof ea hgrayvalue,

x(j)

isthe sumof thegrayvalues inregion

j

of thehistogram (in the proposedapproa h we divided the histogram into ve regions). The image is orre t

when

µ

≈ 127

,

E

≈ 8

,

ACM

≈ 50

and

M SV

≈ 2.5

. These measures allow to analyze the performan e ofthe automated alibrationalgorithm. Moreover, theinformation of MSV will

also beusedto alibrate the exposureand thegainof the amera.

Thealgorithm onguresthemostimportantparametersofthe amera: exposure,gain,

white-balan e and brightness. For theUnibrainFire-i ameras, thedynami range ofthese

param-etersare:

•

exposure: 0-511;

•

gain: 0-255;

•

white-balan e:0-255, both for theredand theblue hannels;

•

brightness: 128-383.

For thePoint Grey Flea 2 amera, thedynami rangeof theparameters are:

•

exposure: 1-1023;

•

gain: 0-683;

•

white-balan e: 1-1023, both for theredand theblue hannels;

•

brightness: 0-255.

For ea h one of these parameters, a PI ontroller was implemented. PI ontrollers are used

insteadof proportional ontrollers asthey resultinbetter ontrol havingno stationaryerror.

The onstants of the ontroller have been obtained experimentally for both ameras,

guar-anteeing the stability of the system and an a eptable time to rea h the desired referen e

gen e of theparameter underanalysis has been attained. The algorithm stops when all the

parameters have onverged. Thealgorithm isoutlinedinFig.3.5.

Brightness

Calibration

Black

Region

R, G, B

Camera

Parameters

Get

Set new

Parameters

Calibration

Exposure / gain

Intensities

Histogram

MSV

Camera

Parameters

Get

Set new

Parameters

White balance

Calibration

Region

White

U, V

Camera

Parameters

Get

Set new

Parameters

Figure 3.5: Overview of the automated alibration pro edure. The algorithm exe utes one

module at a time, hanging between them when the onvergen e of the parameter under

analysishasbeen attained.

Tomeasuretheperforman eofthis alibrationalgorithm,testshavebeen ondu tedusingthe

amerawithdierentinitial ongurations. InFig.3.6,theexperimentalresultsarepresented

both when the algorithm startswith theparameters of the amera set to zeroand setto the

maximumvalue. Asit anseen,the ongurationobtainedafterusingtheproposedalgorithm

isapproximately thesameindependently oftheinitial ongurationofthe amera. Moreover,

the algorithm onvergesfast, between 60 and 70framesto onverge.

InFig.3.9ispresentedanimagea quiredwiththe amerainautomode. Theresultsobtained

using the amera with theparameters inautomode areoverexposedandthewhitebalan eis

not orre tly ongured. Thisisduetothefa tthatthe ameraanalyzestheentireimageand,

aswe an seeinFig.3.9, there arelarge bla k regions orresponding to therobot itself. The

implemented algorithm uses a mask to sele t the region of interest to alibrate the amera

using only the valid pixels. Moreover, and due to the hanges in the environment around

the robot as it moves, leaving the amera in auto mode leads to undesirable hanges inthe

parameters ofthe amera, ausing problemsto the orre t olor lassi ation.

Table 3.1 presents the value of the statisti al measures des ribed to evaluate the quality

of digital images, regarding the experimental results presented in Fig. 3.6. These results

onrms that the amera is orre tly ongured after applying the automated alibration

pro edure, sin e the results obtained are near the optimal. Moreover, independently of the

0

200

400

600

800

1000

1200

0

20

40

60

80

100

120

Value

Frame

WB_RED

WB_BLUE

Exposure

Gain

Brightness

0

200

400

600

800

1000

1200

0

50

100

150

200

250

Value

Frame

WB_RED

WB_BLUE

Exposure

Gain

Brightness

Figure3.6: Someexperimentsusingtheautomated alibrationpro edure. Ontheleft,results

obtained starting with all the parameters of the amera set to zero. On the right, results

obtained with all the parameters set to the maximum value. On the top, the initial image

a quired. Inthe middle, the image obtained afterapplying theautomated alibration

pro e-dure. The last row ontains the graphi s showing the evolution of the parameters along the

time.

A ording to the experimental results presented in Table 3.1, we on lude that the MSV

measureisthe bestone in lassifyingthe qualityof animage. Although important,theother

measures annot distinguishbetween two imagesthathave lose hara teristi s.

his-mode. Onthe right, animage a quired afterapplying theautomated alibrationalgorithm.

Experiment  ACM Average Entropy MSV

Parameters Initial 111.00 16.00 0.00 1.00

setto zero Final 39.18 101.95 6.88 2.56

Parameters. Initial 92.29 219.03 2.35 4.74

set to maximum Final 42.19 98.59 6.85 2.47

Camera Initial 68.22 173.73 6.87 3.88

AutoMode Final 40.00 101.14 6.85 2.54

Table 3.1: Statisti al measures obtained for the images presented in Figs. 3.6 and 3.7. The

initial values refer to the images obtained with the amera before applying the proposed

automated alibration pro edure. The nal values refer to the images a quired with the

ameras ongured withtheproposedalgorithm.

tograms presented in Fig.3.8. The histogram of theimage obtained after applying the

pro-posedautomated alibrationpro edure(Fig.3.8b)is enteredneartheintensity127,whi his

a desirable property, as visually onrmed in Fig.3.6. Thehistogram of theimage a quired

using the amera in auto mode (Fig. 3.8a) shows that the image is overexposed, leading to

the majorityof thepixels to have bright olors.

This algorithm have also been tested outdoor,undernatural light. Figure3.9 showsthatthe

algorithm workswellevenwithdierentlight onditions. It onrmsthatthealgorithm ould

0

1000

2000

3000

4000

5000

6000

0

50

100

150

200

250

300

Intensity

Histogram of the intensities, MSV = 4.74

a)

0

500

1000

1500

2000

2500

3000

3500

4000

4500

0

50

100

150

200

250

300

Intensity

Histogram of the intensities, MSV = 2.47

b)

Figure3.8: Thehistogram ofthe intensitiesofthetwoimages presentedinFig.3.6. a) shows

the histogram of the image obtained withthe amera inautomode. b) shows thehistogram

of theimage obtained afterapplying theautomated alibration pro edure.

3.2.1 Run-time auto- alibration

Sometimes it is ne essary to adjust the amera parameters during the game be ause the

illuminationalongtheeldisn't onstant andthenatureofthelightanditsintensity hanges

during the game. That usually makes the olors be ome out of the range where they were

supposed to be. Consequently the robot an't identify where the ball is or identify its own

lo ationonthe eld. Inordertoover omethisproblem,itis ru ialtokeepapro essrunning

able to adjust the amera parameters during the game, and insuring that the image olors

an be kept onstant independently of the light. In this se tion, it will be dis ussed an

it is possible to observe, the olors are washed out. That happens be ause the amera's

auto-exposure algorithm tries to ompensate the bla k around themirror. On theright, the

sameimage withthe amera alibrated usingthe implementedalgorithm. Asitis possible to

observe,the olorsand their ontours aremu h moredened.

it ould ae t the other pro esses running in the omputer of the robots, for example the

a quisitionand image pro essingprogram [23℄and theagent software.

The developed algorithm is based on the assumptions des ribed in the previous se tion but

only adjusts the gain and exposure. The only measure made for ea h frame is the MSV

(Equation 3.5). As mentioned before, it allows to know if the image is underexposed or

overexposed. Thismeasure isfast to be al ulated, inparti ular when theimage is a quired

on theYUV olor spa e, dueto thefa tthattheluminan e ofea h pixelis a quireddire tly

from the amera without any furtherarithmeti operation. This algorithm isin luded inthe

image a quisition and pro essingappli ation,whi h istheappli ation runningon therobot's

omputerresponsiblefora quiring,pro essandanalyzetherelevantinformationofea hframe

[23 ℄, andtakesabout 2ms perframe toexe ute. Aftertheframe isa quired, thealgorithm is

alledto al ulate theMSVofthe frame,thenthegainandtheexposureareadjustedinsu h

waythat theMSVof thenext frameisnear 2.5.

TestshavebeenperformedonCAMBADAeld,whereitispossibletoswit honandothree

rows of uores ent lamps. Inthe rsttest, therobot is pla ed inthemiddle of theeld and

the teststartswithallthelampson(seeFig.3.10a)). Then,onebyone,alltherowsoflamps

areswit hedo. Afterthat, the se ondrowisswit hedon,followed bytherstrowandthen

the thirdrow.

An exampleofanimage a quiredintheexperiment des ribedaboveispresentedinFig.3.11.

evolu-Figure3.10: Ina) therepresentationof CAMBADA's eld,for therstrun-timeexperiment.

Therobotpositionisrepresentedasabla k ir le,andtherowswiththelampsarerepresented

with the blue olor, showing the arrangement of the lamps on CAMBADA's eld. The test

has started with all rows of lamps on. Then, from the row one to the row three, the rows

lampswereshutdownone byone. After that,the se ond row isswit hedon,followed bythe

rst row and then by thethird row. On the right, the red arrows represents the movement

alongthe eldperformedbytherobotduringthese ondrun-timetest,startingwiththerows

one and three on. Then, inthe middle of thetest, the rst and the third rows are swit hed

o, and these ondrowis swit hedon at thesame time.

Figure3.11: AnimageoftheCAMBADA'seldwithonlythese ondrowoflampson, ausing

a non-uniform light overthe eld. As itis possible to observe, it auses a great variation of

thealgorithm triestoadjustthe ameraparameters, insu hwaythattheMSVisalways 2.5.

Whenbothparameters,gainandexposure,hitsthemaximum,thereisnowaytomaintainthe

MSVat2.5andthe residualerrorisa umulatedbytheintegral ontroller. The onsequen e

isthatthealgorithm doesn'trea t immediatelywhentheMSVde reases. Ina realsituation,

thisdoesn't auseanyproblembe ause theillumination hangesslowlyalongthetime.

300

400

500

600

700

800

900

1000

1100

0

100

200

300

400

500

600

700

800

900

1000

Value

Frame

Gain

Exposure

1

1.5

2

2.5

3

3.5

4

0

100

200

300

400

500

600

700

800

900

1000

Value

Frame

MSV

Figure 3.12: Graphi s of rst run-time test. It shows that the algorithm adapts qui kly to

small hangesinlight onditions maintaining theMSVvalueat 2.5.

These ondexperimentperformedstartswiththerobotpla edinthemiddleoftheeld. Then

itmovesinthe eld makingthetraje tory marked intheFig.3.10b) withtheredarrow. In

the beginning of the test, only the rst and third rows of lamps are swit hed on. Then, in

the middle of the test, the rst and the third rows are swit hed o, and the se ond row is

swit hed on at thesame time.

Theexperimental results arepresented inFig. 3.13,where thegraphi spresent theevolution

oftheexposureand gainparameters ofthe amera and theMSV value. As an be observed,

pla edonthe eld,meaning thattheobje tsofinterest, su hasthewhitelines,are orre tly dete ted.

200

300

400

500

600

700

800

900

1000

1100

0

200

400

600

800

1000

1200

1400

1600

Value

Frame

Gain

Exposure

1.6

1.8

2

2.2

2.4

2.6

2.8

3

0

200

400

600

800

1000

1200

1400

1600

Value

Frame

MSV

Figure3.13: Graphi softhese ondrun-timetest. Itshowsthatthealgorithm adaptsqui kly

todierent light onditions maintaining theMSVvalue at2.5.

Regardingtheexperimentalresultsobtained,itispossibleto on ludethattheadjustmentof

thegainandexposureparameters ontributesigni antlyto the olor lassi ation. Itmakes

the robot to be ableto always lassify the olors, even under light variations alongthe eld.

Consequently, the robot always knows where it ispla ed on theeldand an make theright

de isions and perform the right behaviors. The ee tiveness of the proposed algorithm was

onrmedonROBOTICA2008where thisalgorithm wasusedandtheteamCAMBADAhas

Color alibration

In the RoboCup domain image analysis is simplied, sin e obje ts are olor oded. Bla k

robotsplaywithanorangeballonagreeneldthathaswhitelines. Thus,the olor ofapixel

isa strong hint for obje t dete tion. Dueto this fa t, the orre tobje t dete tionmay have

asigni ant dependen e onthe orre t olor lassi ation.

The alibration of the olor range asso iated to ea h olor lass is another pro ess that has

to be performed before ea h game be ause the olors of an image a quired by a amera are

dependent of the s ene and also of the amera alibration. These two pro edures have to

mat h ompletelyinorderto allowtherobot tore ognize therelevant obje tsexisting inthe

environment aroundit.

Untilnow,the olor alibrationpro edurehasbeenmadeoine,whi hmeansitwasne essary

topreviouslya quireavideosequen e. Then,withanoineappli ation,theusersele tssome

pixelsanda ordingtothetheir olorvalueaparti ular olor lassis reated. Thispro edure

hasto berepeatedfor ea hrelevant olor. Usually,the olor alibrationisperformedinHSV

(Hue,Saturation andValue) olor spa edue to itsspe ial hara teristi s (seeFig. 4.1)[14℄.

Themain drawba kof this pro edureisthefa t thatithasto beperformedoine,meaning

thatwhenthe amera alibration or the olorsof the s ene hange, itis ne essaryto a quire

anewvideo andthenpro ess itagain. Thesesteps usuallyrequiresseveralminutesto adjust

the information for ea h olor. Another problem is that, using an oine pro edure, there is

nowayto test the olor alibration resultinreal-time and inall positionsof theeld.

In this hapter three developed methods will be presented allowing online olor alibration.

Thesemethods make possible to alibrate the olorsdire tlyon therobot.

brate the olorsinoine mode [24℄.

area for ea h olor lass. For ea h area, it is al ulated the maximum and minimum values

in HSV olor spa e. These values are used to sele t the upper and lower boundaries of the

olor lass. If the olor range wasn't orre tly alibrated, a manual adjustment has to be

madeformaximumandminimumvaluesofhue,saturationandvalueusingaslidersinterfa e.

This pro edure an present some problems parti ularly when the sele ted area hasn't an

homogeneous olor or, sometimes, when thereis a pixel witha quitedierent olor fromthe

desired one,leading to olorswrongly alibrated orsomewhat lessa urate. Furthermorethe

use ofsliders to sele tthe olor rangeis not mu h intuitive.

In orderto improvethemethoddes ribed above,these ondmethodisan evolution fromthe

rst one. Itwasdevelopedan interfa e thatusesthehistogram ofthethree olor omponents

(hue, saturation and value) to sele t the desired olor range for ea h olor lass. For ea h

olor, a set of pixels are sele ted orresponding to the olor lass that is under alibration.

Thevalueofea hpixel,inHSV olorspa e,ismarkedonthehistogram ofhue,saturationand

value. Then,itissele tedtherangeofhuefor thesele ted olorandthesaturationandvalue

histograms are updatedwith theimage pixels in luded inthis range of hue. This pro edure

segmented. Onthe right, the orrespondinghistogram. Fromtop to bottom: hue, saturation

andvalueof the image.

4.1 Automated algorithm for olor alibration

A third method was implemented using HSV histograms and a annyedge dete tor [25 , 26 ,

27,28℄. The annyedgedete torndstheedgesinareaswithstrongintensity ontrasts. This

isthemost usededgedete torin imagepro essing. Applying the annyedgedete tor tothe

image a quired, allthetransitionsbetween olorswithhigh ontrast aremarked,asshown in

Fig.4.3.

Figure4.3: Onthe left,animagea quiredonthe CAMBADA'seld,towhi hthe annyedge

dete tor was applied. On the right, the same image with strong edges after onsidering as

edgetheneighboring pixels ofthe originaledges obtained bythe annyedgedete tor.

Hue 79 177

Saturation 26 64

Value 84 207

Table 4.1: Color ranges obtained for the green olor lass using the des ribed algorithm

a ording to the Fig.4.4.

.

orre tthisee t,itis onsideredasedgetheneighboringpixelsoftheoriginaledgesobtained

bythe annyedgedete tor. Withthistransformedimage,whereareaswiththesame olorare

welldelimited,itispossibleto al ulatethemaximumandminimumboundariesinHSV olor

spa e for olor lassi ation. First, it is ne essary to nd at least one pixel of the intended

olor in ea h one of the delimited areas. For that, using the assumption that most of the

image pixels are green, the hue, saturation and value of the olor with the higher level of

o urren e in the HSV histogram aredetermined. Using theHSV histogram, itis possible to

knowwhi h are the values, inHSV olor spa e, of the pixelthat appears more often. Then,

with a threshold of 10, it is sele ted at leastone pixel inea h delimited area. Now, using a

region growingalgorithm, allpixels delimitedaresele ted. Withall greenpixels sele ted,itis

al ulatedthe maximumandminimumvaluesinHSV olorspa eforthe olorunderanalysis.

The right side of the Fig. 4.4, shows the eld image with the green olor segmented. The

results obtained with this algorithm are presented on Table 4.1. As it an be seen, all the

green pixels aresegmented onrmingthat thisalgorithm workswell.

Figure4.4: Onthe left,animage a quiredontheCAMBADA'seld. Ontheright,thesame

urrentimplementationhasahigh omputational omplexity(above100mstopro essasingle

Con lusion

There are many problems behind the use of ameras in roboti appli ations. One of them,

andthemostdi ultto over ome,istheuseof amerasinroboti appli ationsundernatural

light. To a hieve therstgoalofRoboCupMSL ompetition,itisne essarytoover omethis

problem.

In Chapter 1, it was dis ussed the most ommon problems behind the use of ameras in

RoboCup MSL ompetition. It was also dis ussed how CAMBADA team and the other

RoboCupMSL teamssolve this problems.

In Chapter 2,it wasintrodu ed some notions about amera parameters and theRGB,YUV

andHSV olorspa es,tobetterunderstandthealgorithmsdis ussedinthefollowing hapters.

InChapter3, itwasdis ussed the two implemented algorithms to auto alibrate the ameras

atthebeginningofea hgame. Thisdoesn'tsolvethewholeproblembutminimizestheee ts

of thelight variations during the tournament. By omparing the two developed algorithms,

itis easy to on lude that the se ond algorithm, usingPI ontrollersand the MSV measure,

hasbetterresults. Thathappensmainlybe auseitusesthePI ontrollerstoapproximatethe

ameraparameterstotheidealparametersandtheMSVmeasurethatevaluatestheluminan e

of theimage. Comparingto the se ondalgorithm,therstdevelopedalgorithm doesn't work

so well. One of the problems is thatit takesmu h more timeto onverge. Another problem

isthatitonly uses theinformation oftwo sele ted areas,and not thewhole image.

It was also presented a run-time algorithm that an be used withthe image a quisition and

pro essingappli ation. Itminimizestheee tsoflightingvariationsalongtheeldandduring

the game. That isa typi alproblem thatCAMBADA team hasfa ed,until now.

analysis of a sele ted area for ea h olor, isn't themost a urate due to thefa t that if one

pixel of the sele ted area is quite dierent of the sele ted olor, that will ause an in orre t

olor alibration. It ispossible to adjustthe olor ranges using aninterfa e withsliders.

The se ond method, using HSV histograms, makes the pro ess of manual olor alibration

easier, be ause sele ting some pixels on the image allows to visualize their position on HSV

histogram. Sele ting the maximum and minimum boundaries of hue, saturation and value

histograms areupdatedonlywith the image pixelsin luded on thesele ted rangeof hue.

The last developed method use an edge dete tion and a region growing algorithm. The

ex-perimentalresultsobtainedwiththedes ribedalgorithm arepromising. However, the urrent

implementation hasahigh omputational omplexityand annotbeusedinreal-time. Inthe

future, with some optimization this drawba k ould be solved. This method with the

am-era alibration runningon robot's omputer, will probablyallowthe robot to have its vision

systemalwaysoptimizedfor thebestperforman eon olor lassi ation andobje tdete tion.

Now withthe proposedalgorithm for amera parameters and olor alibrationmethods, itis

possible to alibrate the vision system of the robots without the need for either a previous

video a quisition or for the presen eof an expert person. This an now be madedire tly on

the robot,andvisualized inreal-time ifthe olorsare orre tly alibrated.

This thesis has ontributed signi antly to the good results obtained by the CAMBADA

team in thePortuguese Roboti s Open, ROBOTICA 2008 [29℄. The work presented in this

thesiswasalsousedinotherroboti appli ations,asisthe aseoftheRota2008andRatozinger

teams,ofUniversityofAveiro,intheAutonomousdriving ontest. Thisworkhas ontributed

to their olor and ameras alibration.

The results obtained prove thatthealgorithms presentedinthis thesiswill ontributeto the

main goals ofRoboCupleague.

As futurework,the following lines an be pointedout:

•

Improvement of the olor alibration performan e in su h way that it an be made

automati ally inoine and alsoinrun-time mode.

•

Improvement ofthewhite-balan ingalgorithm,insu hwaythatitdoesn'tneedawhite

referen e or,ifneeded, the referen emustbesear hed intheimage.

•

Development of analgorithm to automati ally alibrate the amera saturation.

•

Implementation of an algorithm integrated into the software of the robot to alibrate

AutoCalibration manual

A.1 Introdu tion

This manual intends to explain how to use the alibration tool designed for CAMBADA

team,to alibratetheir rewire ameras. Thisappli ationisusedtoadjustautomati allythe

parameters ofthe amera. To useit,theappli ation only needsto knowwhere thebla kand

white referen es are. These are the ne essary referen es used by the alibration algorithm.

The amera's parameters areadjusted by theanalyses of thesele ted bla k and white areas

andthewholevalidimage, untilthe "thewhite isreally whiteand thebla kis reallybla k".

Thealgorithm ould be explainedina simple way su has:

•

The white-balan e (the gain of blue and red hannels respe tively), is adjusted until

the average values of U and V (in YUV olor spa e), of the sele ted white zone are

approximately 127.

•

Thegainandexposureareadjustedinsu hwaythattheluminan ehistogramis entered

around 127.

•

The brightness is adjusted from the zone sele ted as bla k. It is adjusted until the

average value ofea h RGB hannelis around 2.0.

Thisappli ation also allows the user to alibrate the olor range, using theinformation

The use of this appli ation is simple. To start the appli ation the following ommand line

format is used: ./AutoCalibration - f # {options} The only mandatory parameter is the

onguration le(option - f r?. onf), where the information of theinitial parameters of the

amera,the olorrangeforea h olor,themaskthatsele tsthevalidpixels,andthedistan e

map are stored. This appli ation was developed only for the ameras used by CAMBADA

team,whi hareaPointGreyFlea2and aUnibrainFire-iBCL.Touseoneofthese ameras,

theparameter - am mustbeinitialized (option- am #{ Unibr: UnibrainFire-iBCL,Point:

PointGreyFlea2}). Theparameterfps(frames-per-se ond)alsohastobe ongured(option

-fps #{3,7,15, 30}).

When theappli ation starts, the image a quired by the amera is diplayed inthe omputer

s reen. To alibratethe amera parameters, rst itisne essaryto sele tthewhiteand bla k

areas. Forthis,presstheCkeyandinthes reenisdisplayedthenumberofea h olor. Then,

press thekey numberof the desired olor. When the olor is sele ted,with themouse sele t

theareaforthat olor. ThekeySallowstoseethesele tedzones forea h olorinthes reen.

Tostartthe alibrationalgorithmpresstheAkey. Inthes reenwillbedisplayedthemessage

"AutoCalibrating..."andthe urrent values ofgain, exposure,brightness and white-balan e.

After a fewse onds, when the parameters had stabilyzed, you should press again theA key

to stopthe alibration algorithm. To store the orre t parameters into the onguration le

pressU.Itispossible tousethe amera's auto alibrationalgorithmat anytime,pressingthe

Kkey. To desa tivate the amera's algorithm press againtheKkey.

To alibrate the olors, rst press the E key. It will be displayed in the s reen the HSV

histogram. To sele t thedesired olor to be alibrated, press inthe"HSV - Histogram

Win-dow", with the mouse's left key, the respe tive square olor. Then press the key I. It will

be displayed, in a new window, an enlarged image of the mouse's pointed area. To know

the values, in HSV olor spa e, of thesele ted pixel, press the mouse's left key. The values

of the urrent pixel will be displayed on the image and in the HSV-Histogram window, the

urrent pixel position will be displayed with white lines, in the HSV-Histogram. To display

the segmentedimage, press the Y key. Ifa ertainpixel isnot segmented, for therespe tive

olor, sele tit with themouse. Onthe HSV-Histogram window, the redlines arethe upper

andlowerboundariesofthesele ted olor. Ifa ertainwhiteline isoutof therange, tomove

thelowerboundarie,inthe HSV-Histogram window, presson thewhitelinewiththemouse's

left button. To move the upperboundarie, in the HSV-Histogram window, sele t the white

Now will be explained the fun tion of ea h key and the meaning of the arguments that is

possible to passto theappli ation:

Appli ation:

-h Help

- f Conguration le (Obligatory)

Camera:

-mode #4 4:RGB, 1:YUV 411,2:YUV422

- am #{Point} Point: Point GrayFlea2,Unibr: Unibrain Fire-iBCL

-fps #{15} 3:3.75, 7:7.5,15, 30 -wb #:#{-1:-1} -sht #{0} -exposure #-1 -gain #{-1} -brightness #{-1} -gamma #{-1} -saturation #{-1} -sharpness #{-1}

H Visualizeon the s reen the fun tionof ea h key

C Allowsto sele t aspe i olor area

A A tivate/dea tivatethe alibration algorithm

S Shows the sele ted olor areas

K A tivate/dea tivatethe amera's alibration algorithm

U Updatethe ongurationle

P Pause

I Allowsto know the HSV information ofa ertainpixel

Shows the enlarged areaaround themousepointer

Y Segment the image

E Displays the HSV histogram

R Displays the RGB histogram

G Displays the Luminan eHistogram

LibAutoCalib

•

void Rgb2Yuv( double*RGB, olor_yuv&yuvPix ) - Converta RGBpixel to YUV

•

void Rgb2Yuv2(double *RGB, olor_yuv&yuvPix )- Converta RGB pixelto YUV

•

void ConvertYuv2Rgb(unsigned har bun[℄,unsigned har buout[℄, int mode)-

Con-vert aYUVframe to RGB

•

voidAverageAreaRGB(unsigned har*imgBuf,inty1,inty2,intx1,intx2,Stats*RGB)

- Cal ulates theaveragevalueinRGB modeof thesele ted areafor ea h hannel

•

voidStdAreaRGB(unsigned har*imgBuf,int y1,int y2,int x1,int x2,Stats*

statsval-ues)-Cal ulatesthestandarddeviationvalueinRGBmodeofthesele tedareaforea h

hannel;

•

void MaxMinAreaRGB(unsigned har* imgBuf, int y1, int y2, int x1, int x2, Stats*

statsvalues)-Cal ulatesthemaximumandminimumvaluesinRGBmodeofthesele ted

area forea h hannel

•

voidAverageAreaHSV(unsigned har*imgBuf,inty1,inty2,intx1,intx2,Stats*HSV)

- Cal ulates theaveragevalueinHSV mode of thesele tedarea forea h hannel

•

voidStdAreaHSV(unsigned har*imgBuf, inty1, int y2,int x1, intx2, Stats*

statsval-ues)-Cal ulatesthestandarddeviationvalueinRGBmodeofthesele tedareaforea h

hannel

•

void MaxMinAreaHSV(unsigned har* imgBuf, int y1, int y2, int x1, int x2, Stats*

statsvalues)-Cal ulatesthemaximumandminimumvaluesinRGBmodeofthesele ted

•

voida mCal (unsigned har*buf,unsigned har*mask,ImageStats*stats)- Cal ulates

image statisti s ofthe validpixels

•

voidSegmentateImage(unsigned har*SegImage,unsigned har*mask,ColorRange* r

) - Segment the image

•

void Cal Histogram(unsigned har* imgBuf, unsigned har* mask, int* histogram,

Im-ageStats* stats)- Cal ulates thehistogram

•

ImageStats GetHistParam ( onstImageHolder& image, unsigned har*mask) -

Cal u-lates histogramparameters

•

void CalibrateGainExposure(ImageStats stats, CameraSettings* s) - Cal ulates the

Gain and Exposure values to set to the amera a ording to theinformation passed

bystats.

•

Class ImageAnalyzer

 ImageAnalyzer() - Default onstru tor

 ImageAnalyzer(enum iMode Mode, unsigned Rows, unsigned Cols, unsigned har

*ImgBuf, unsigned har *Mask)- Contru tor

 

ImageAnalyzer() - Destru tor

 void Cal ImgStats()- Cal ulates image satisti s

 void DrawHistogram() - Drawhistogram

 void SetColourCoord(unsigned olor, int x1, int y1, int x2, int y2) - Set olor

oordinates

 void Cal ImgAreaStats()- Cal ulates areasatisti s

 void AverageArea(unsigned olor) - Cal ulates the average value of the sele ted

area forea h hannel

 void MaxMinArea(unsigned olor) - Cal ulates the average value of the sele ted

area forea h hannel

 void ZoomPoint(CvPoint ptCenter) - Zooms thesele ted point

 void SegmentateImg (ColorRange * r)- Segmentates theimage

 void DrawRgbHistogram() - Draws the RGB histogram