H A RTZ, Z. M. A. 2 4 8
Cad. Saúde Pública, Rio de Jane iro , 15(2):229-259, abr-jun, 1999
Eva l u a t i o n : th e Fren ch che fs are still s e a rching for “la n ou v elle cu isi n e”
I wish t o b e gin b y c o m p lim e n t in g Zu l m i ra Ha rtz fo r h e r a rt icle o n t h e in stit u tion a lizatio n of e valu a t io n in t h e Fre n ch co n te xt , s in ce in a few p a ges sh e p rovid es a b rillian t an a lysis o f th e h i s t o ry o f p r o g ra m e va lu a t ion in Fra n c e . Se c-o n d , sh e ra ise s a n u m b e r c-o f im p c-o r t a n t issu e s on wh ich I wo u ld like to co m m en t b rie fly h e re.
Th e first p o in t con cer n s t h e role o f e va l u a -tion in o u r p a rliam en t a ry d e m o cra c i e s. In d e e d , in s t it u t io n a liza t io n o f e va lu a t io n s h o u ld b e se en as an at te m p t to re s t o re to elect ed officia ls t h e p ow e r t o co n t ro l th e fu n c tio n in g o f p u b lic a d m i n i s t ra t io n , t h e so ca lled t ech n ocra c y. Be
-G é ra rd de Po u vo u rv i l l e
Grou p e Im a g e , École N a tion a le d e la San té Pu b l i q u e S a i n t - Ma u r i c e , Fra n c e . e va lu at ion in vo l ves in terests lin ke d to t h e
power sp h powere, espe cia lly wh e n it is a m a tt power of p u b -lic p o -licy e va lu a t ion . A p o -licy’s su cce ss o r fa il-u re m ea n s t h e a ccil-u m il-u lat io n or lo ss of p o litica l a n d sym b olic ca p it a l e it h e r b y t h o se wh o go v-e rn o r b y t h o s v-e in o p p o sit io n . Eva lu at io n ca n a lso b e u sed by m a n a ge rs in sid e in stitu tion s as a t o ol t o con tro l su b o rd i n a t e s. T h u s, re s i s t a n c e t o t h e o u t sid e e va l u a t o r, q u a n t ifica t io n , a n d o b jec t ifica t io n ca n b e gre a t e r, wh ile st ra t e g i e s t o e xp a n d in d ivid u a l p owe r (e m p o we rm e n t ) m ay b e m o re read ily accep te d a n d m o re o rg a n -ic wh e n t h e re is a t re n d t ow a rd s d e c e n t r a l i z a-t io n a n d d e m o cra a-t iz aa-t io n o f d e c isio n -m akin g p ro c e s s e s.
A n a l y ze d fro m a d iffe r e n t a n gle , t h e s u r m e s u rea p p ro ach m ay b e se e n a s a d ju st in g t h e e va l u a t i o n’s m e t h od ological st ra t e gy t o it s ob -je c t . In t h is se n se, it sh o u ld also b e p re f e r re d , wh ile th e m a in p rob le m s in vo l ve m ore th e con -s t r u ct io n o f t h e o b je c t (Bo u r d ie u , 1989) a n d t re a t m e n t o f t h e o ry a s a gu id e fo r e va l u a t i o n (Ch e n , 1990) t h a n t h e o p p os it io n b e t we e n q u a l i t a t i ve a n d q u an t it at ive t e ch n iq u e s, wh ich (a s Ha r t z p o in t s o u t q u it e a p p ro p ria t e ly) ca n b e ar ticu la t ed in act u a l st u d ies.
An o t h e r im p o rt a n t p o in t ra ise d b y t h e a u t ho r is t h e d istin ct io n b e twee n e va lu a tion p ro -g ram s a n d p o licies. Alth o u -gh th e b ib lio -gra p h y she q u ote s d o es n ot m ake t h is d istin ction , I b e-l i e ve it is n ecessa ry from b o th a th eo ret ica e-l an d m e t h o d o lo gica l p o in t o f vie w. Pu b lic p o lic ie s re la te to t h e st a te, i.e., th e p ower fie ld ; eva l u a t-in g p o lic ie s t-in vo l ve s n o t o n ly ju d gt-in g t h e a d e q u a c y, pe rt i n e n c e, e ffect ive n e s s, efficien cy, a n d legitim acy of gove rn m en t al in t en tio n s a n d a c t i o n s, b u t e sp e cia lly a n a lyz in g t h e n a t u re o f t h e st a t e a n d t h e p o lit ic al p owe r in vo l ve d in d ra ft in g t h e m . Me a n w h i l e , p ro g ra m s re l a t e m o re t o a p o licy’s tech n ical an d op erat io n al d im e n sion , i.e., its im at e ria l im a n ife stat ion a s o b -j e c t i ve s, go a ls, re s o u rc e s, a n d a c t ivit ie s, a n d t h e ir e va lu a t ion re q u i re s a set o f m e t h o d s a n d t e ch n iq u e s t h a t are d iffere n t fro m t h o se n e ed -e d t o a n a lyz-e p o lic i-e s. By c o m b in in g p o lici-e s a n d p ro g ra m s in t h e sam e o b ject , o n e ru n s t h e risk o f red ucin g p olit ics to te ch n iq u e or e ven t o p la n n e d p o licy, o r (in ve rse ly) focu sin g on ly o n t h e p o litic a l s id e o f t e ch n iq u e. I d o n o t m e a n t o sa y (e sp e cia lly a t t h e lo ca l gove r n m e n t a l or e ve n in s t it u t io n a l le ve l) t h a t p r o g ram e va l u a -t io n is a m e re ly -t e ch n ic a l issu e . Fo r e xa m p le, d e pe n d in g o n t h e o b je ct or issu e t o b e e va l u a t -e d , con t-e xtu a l a n alysis ca n b -e a n in v-e s t i g a t i o n s t ra t e gy fo r e xp la in in g t h e p roc e sse s in vo l ve d in im p le m en t in g a p ro g ra m .
La st co m es t h e relat io n sh ip b et ween e va l u -a t io n -an d t h e d e cis io n - m -a kin g p ro c e s s. Mo re
t h a n a t h e o re t ical, t e ch n ica l, or m e t h od o logi-ca l q u e st io n , t h is is a p o litilogi-ca l an d eth ilogi-cal issu e, i n vo lvin g c h o ic e s. Th a t is, fa ce d wit h va ri o u s ra t io n a le s t h at in te rf e re with t h e m an a ge m e n t p ro c e s s, in st it u tio n aliz at ion of e valu at io n fo r a p u b lic h e alt h syst e m m e a n s se e kin g t o e n su re t h e h e ge m o n y of th e te ch n ica l/ he alth ra t i o n a l e in t h e d e c isio n -m akin g p ro c e s s , i.e., p ri o ri t i z-in g h e a lt h n e ed s ove r z-in st it u t io n a l cor p ora t i s t o r e ve n e xt e rn a l p re s s e s. It m e an s d e ve l o p i n g m a n a ge m e n t b a se d o n id e n t ificat io n of p ro b -l e m s, o rga n izin g su p p -ly t h ro u gh p ro g ra m m e d a c t i o n s, a n d e m p h as izin g co n t r o l o f ris ks a n d c a u se s t h ro u gh a t e rri t o r ial fo cu s, wit h so cial p a rt icip a t io n . In o t h e r wo rd s, it m e a n s ch a n gin g t h e cu rre n t h e a lt h c a re m o d e l. Th is d e m a n d s n o t o n ly e va lu a t in g, b u t e sp e c ia lly in -t e r ven in g -to ch a n ge -th e cou n -tr y ’s h e a l-th re a l i -t y
su r m esu re.
BOURDIEU, P., 1989. O Poder Si m b ó l i c o. Rio d e Ja n e i-ro : Be rt ra n d Bra sil/ Lisb o a : D i f e l .
BOURDIEU , P. & WACQUA N T, L. J. D., 1992. R é p o n s e s . Pou r Un e An t h ropologie Réflex i ve. Pa r is: Se u il. C H E L I M S K Y, E., 1997. Th e Co m in g Tra n s f o rm a t i o n s
In Eva lu at io n . In : Evaluation for the 21st Ce n t u ry : A Ha n d b o o k(E. Ch e lim sky & W. R. Sh a d ish , e d s. ) , p p . 1-26, Th o u sa n d Oa ks: Sa ge Pu b l i c a t i o n s. CH EN , H ., 1990. T h e o ry - D r i ven Ev a l u a t i o n s. Ne w
AVALIAÇÃO DE PRO GRAMAS E PO LÍTICAS DE SAÚDE NA FRANÇA 2 4 9
Cad . Saúd e Públic a, Rio d e Jane iro , 15(2):229-259, ab r-jun, 1999
cau se p ub lic a ction h as b ecom e so d ive rse an d co m p le x, it is o b vio u s t h at m e m b e rs o f Pa r l i a -m e n t a re in cre a sin gly le s s a b le t o ju d ge b y t h e m s e l ve s wh e t h er t h e p o licie s t h e y h a ve ap -p rove d a re a ct u ally a ch ie vin g th e ir o b je c tive s. Alt h o u gh t h e re a re in d e p e n d e n t in sp e c t io n b o d ie s in Fra n c e , q u it e o ft e n t h e ir in ve s t i g a -t ion s a re m or e a u d i-t s -t h a n -t r u e e va l u a -t i o n s ( i . e., t h e st u d y o f t h e re la t io n s h ip b e t we e n m e an s a n d en d s). From th at p ersp e ct ive, I m u s t say th at I a m q u it e a t tracte d by t h e US p h ilo so -p h y q u ot e d b y H a rt z, o f ex a n tes o c i a l e x p e r i-m e n t a t ion . If o n e re c a lls t h e fa i-m o u s d ist in c-t io n m a d e b y H irs ch m a n (1970), im p ro v i n g d e m o c ra cy (t h e p ower of vo ice ) is t h us a legitim at e goa l. But ilegitim p le legitim en ta t io n o f such e va l u a -t io n s ru n s u p a ga in s-t n u m e r o u s o b s -t a cle s: m em b e rs of Pa rlia m e n t st ill d ep e n d h ea vily o n t h e p u b lic a d m in is t ra t io n in t e rm s o f e xp e r -t ise a n d a cc e s s -t o d a -t a , a n d -t h e ir own -t im e is s c a rc e. Th e p u b lic a d m in istratio n is a lso re l u c -ta n t t o rele ase in for m a t io n a n d d a-ta , an d th ere is st ill a st ro n g fe e lin g t h a t m em b e rs o f Pa r l i a -m e n t , n o t b e in g e xp e r t s, will fa il t o gr as p t h e co m p le xit y o f a ll t h e is su es. Is t h ere a so lu t io n t o t h e p r ob le m , o t h e r t h a n t o m e r e ly p ro v i d e m o re re s o u rces an d re g u l a t o ry p ower to Pa r l i a-m en t to con d u ct au to n oa-m ou s e valu at io n s?
In d e e d , if e va lu at io n w e re c on s id e re d a n i n t r in sic p a rt o f p u b lic se r vice s m a n a ge m e n t , as Ha rt z su ggests, th en at least p a rt of th e work wo u ld b e d o n e o n a d e ce n t r a l i ze d b a sis. It seem s to m e th at wh at is at stake is m uch m ore t h an ju s t ra t io n a liz in g p ro d u c t io n o f p u b lic s e rv i c e s, i.e., ju st on e m o re m a n ager ial fash ion . Th e tu rn of th is cen tu ry is m arked b y e m p h at ic re ject io n o f t h e We l f a re St a t e, o n t h e b a sis t h a t p u b lic a d m in is tr a t io n , n o t u n d e r t h e p re s s u re o f co m p e t it io n , c a n n o t b e e fficie n t ly m a n a ge d . Go ve rn m e n t s a p p e a r t o b e u n d e r p re s -s u re fro m in t e r n a t ion a l fin a n cia l m a rke t -s a n d a re f o r c e d t o d e m on s t ra t e t h e ir c a p a cit y t o m a s t e r t h e p u b lic se ct o r’s growt h . Th is ge n e r a te s h ea vy id eo lo gica l p re s s u re t o tra n sfe r p ro -visio n o f se rvice s t o t h e p ri va t e s e ct o r, u n d e r t h e a ssu m p t io n th at if su ch ser vice s co m e u n -d e r co m p e titive p re s s u re, th ere will b e a st ro n g i n c e n t i ve fo r m o re e fficie n t u se o f re s o u rc e s. Howe ve r, b e c a u se p u b lic se rvice s m u st b e d e -l i ve re d u n d e r t h e c on s t ra in t s o f a c ce ss ib i-lit y a n d e q u it y, p r i va t izat io n o f d e live r y ge n e ra l l y goe s with so m e fo rm of co n tract wit h t h e stat e, lim it in g co m p e t it io n a n d d e m a rca t in g se r v i c e p rov i d e r s’ a u t o n o m y. So t h e q u e s tion is, if t h e ga m e is n ot re al co m p e tition , wh y is it im p o ssi-b le t o a ch ie ve th e s a m e re su lt s wit h p u ssi-b lic s e rv i c e s, to wh ich m o re a u t on o m y is given , b u t f ro m wh ich m o re a cco u n t a b ilit y sh o u ld b e e
x-p e ct e d ? T h u s, e valu a t io n wo u ld b e t h e co n d i-t io n fo r a cc o u n i-t a b ilii-t y a n d o n e o f i-t h e n e ce s-s a r y c o n d it io n s-s fo r m o d e rn izin g t h e p u b lic s e c t o r.
An ot h e r way to d ecen t ra l i ze evalua t io n is t o e n c o u ra ge it s d e ve lop m e n t a s a le gitim a t e sci-e n t ific a ct ivity, wh ich is n o t a lwa ys th sci-e ca ssci-e, a t le a s t in Fra n c e . Eva l u a t i ve re s e a rch , as it h a s b e e n la b e le d b y Ha r t z , is s t ill co n sid e re d “a p -p lie d re s e a rc h”, a s co m -p are d t o c on ce -p t u a l o r b a sic re s e a rch . Sin ce it is o ft en fu n d e d by p u b -lic agen cie s (o r p ri va t e firm s, for t h e econ o m ic e va lu a t io n o f d r u gs), ve r su s p u b lic re s e a rc h m o n e y, a ca d e m ic e va lu a t io n co m m it t e e s a ls o c on sid e r it m e re c o n s u lt a n c y. It is a c kn ow l-e d gl-e d t h a t s u c h w o rk is a t ru l-e sl-e rvicl-e t o t h l-e c o m m u n i t y, b u t t h e re a re m a jo r d o u b t s a s t o wh e t h e r it is re a lly sc ie n ce . I sh a ll n o t a n swe r t h is q u estion co m p le te ly. I sh all m erely su ggest t h a t m o b iliza tio n o f scie n t ific m e t h od s t o e m -p i r ica lly e x-p lo re -p u b lic a ct io n an d e x-p la in it s im p a ct o n so c ie t y se e m s t o m e t o b e q u it e close t o scie n ce, if o n e co n sid er s t h a t t h e m ain a im o f scie n ce is t o e xp lo re o u r re a lit y wit h c o n t ro lle d m et ho d s. It is t ru e th at p u b lic a gen -c ie s o ft e n a sk re s e a r-c h t e a m s t h e im p o s sib le t a sk o f e va lu a t io n p ro g r am s “d u e ye s t e rd a y ”, a n d if p ossib le a t n o e xp e n se t o t h e b u d get (af-t e r a ll, re s e a rc h is alre a d y fu n d e d b y p u b lic m o n e y e lse wh e re ), t h u s fo st e rin g “q u i c k a n d -d i r ty” st u -d ie s. Wh en st u -d ie s a re fu n -d e -d b y t h e p ri vat e se c t o r, t h e y a re s e e n as d e fe n d in g p ri-va t e in t e re s t s, an d a s su ch su s p e ct e d of b ia s. For exam p le, d r ug re s e a rch in Fran ce is fun d ed m ain ly b y t h e d ru g in d u st ry. T h u s , it a llow s p u b lic a ge n cie s to d isq u a lify th e stu d ies o n th e b a sis o f t h e ir p a rt isa n sh ip an d t o b e m o an t h e la ck of in d e p e n d en t stu d ies. Fu n d e d b y wh o m ? T h e re f o r e, allo w m e t o p le ad fo r sp e c ia l re -s e a rch fu n d in g fo r e va l u a t i ve re -s e a rch an d fo r t h e d evelop m e n t of a stru c t u red re s e a rch co m -m u n it y to d e fin e go od scien t ific wo rk.
H A RTZ, Z. M . A. 2 5 0
Cad. Saúd e Públic a, Rio d e Jane iro , 15(2):229-259, abr-jun, 1999
Th e a rt icle b y Zu l m i ra Ha r t z d e scr ib es t h e in i-t i a i-t i ve s, a i-t le a s i-t a s p ro p o s a l s, ch a r a c i-t e ri z i n g th e im p le m e n t atio n o f a n ove rall p ub lic p olicy a n d a s p ecific p u b lic p o lic y fo r t h e h e a lt h sec-t o r fo r e va lu a sec-t io n o f p olicie s a n d p ro g ra m s ( w h e re “p o l i c i e s” an d “p ro g ra m s” ca n a ls o b e se e n a s pro g ram s an d te ch n ologies, b ut d o n ot in c lu d e p o licie s in t h e s e n s e o f “p o l i t i c s”) in Fra n c e b e gin n in g in t h e 1980s. In o rd e r t o d e -ve lop so m e co m p arison s an d estab lish a n alyti-ca l alyti-ca t egories for eva lu a t io n p olicie s th e au t h o r d e s c rib e s sp e cific a sp e cts of p olicy im p le m en tation in cou n t rie s like t h e Un ited St a t e s, Ca n a -d a, an -d Au s t ra l i a .
Th e ove ra ll ju st ifica t io n fo r d e ve lo p in g h e r re s e a rch a s p re se n t e d in t h e p a p e r is b a sed o n t h e p re m ise t h a t kn o wle d ge o f t h e r e alit y o f o t h e rs fo st e rs a b e t t e r u n d e rst a n d in g o f o u r own , a n d m o re s p e cifica lly t h at a c o u n t r y ’s p u b lic p olicie s a n d p ro g ra m s ca n b e im p rove d ( i . e., b e m a d e m o re a p p ro p r ia t e t o t h e ir o b je c-t i ve s, m o re e ffe c c-t ive , m o re d e m o c ra c-t ic, e c-t c. ) u s in g o t h e r c o u n t ri e s’ e xp e r i e n c e s. In o t h e r w o rd s, n o t o n ly is t h e re n o t a h ist o ric a l d e t e rm in isrm o r “ i n e xo ra b ilit y” (a t le a s t n o t a n ab -so lu t e o n e ); ra t h e r, ra t i o n a l i ze d co lle ct ive a c-tion s are p ossib le, an d th ey a re stren gth en ed to a ce rt a in ext e n t wh e n b a se d o n kn owled ge ac -cep t ed a s t ru e.
Th e s e p re m is e s a re o b vio u sly a re f e re n c e fo r a m ajo r p o rtio n o f re s e a rch a ct ivity, p a r t i c-u la rly in t h e fie ld o f Co l l e c t i ve He a lt h , b c-u t it is
H . Ma ria Du t i l h Nova e s
De p a rtam en to d e Med icin a Pre ve n t i va , Fa cu ld ad e d e Me d i c i n a , Un i versid ad e d e Sã o Pa u l o, Sã o Pa u l o, Bra s i l .
Ha r t z m e n t io n s a n o t h e r im p or t a n t a sp e ct o f e va lu a t io n , t h at it is a n e ve r-e n d in g p ro c e s s. Eva lu at io n s a lwa ys co m e e it h e r t o o so o n (t h e p ro g ra m d o e s n o t a p p e a r t o h a ve b e e n fu lly im p le m e n t e d a n d s t ab ilize d ) o r t o o la t e (ir re -versib ility h a s b ee n cre ate d ). Bu t th is ju d gm e n t is b ased o n an erro n eo u s p erce p t io n of eva l u a -t io n , co n ce ive d o f a s a o n e - sh o -t ju d gm e n -t a -t o n e given m om e n t in t im e , a llowin g for a st op -o r- g-o d e cis i-o n . Ra t h e r, e va lu a t i-o n s h -o u ld b e se e n a s a le ar n in g p ro c e s s , e ach st e p id e n t ify-in g wh at is a lre ad y kn own a n d wh at rem aify-in s t o b e le arn e d .
Fi n a l l y, if o n e se e s e valu a t io n a s a p ro c e s s t o im p ro ve co n d it io n s fo r d e m o cra t ic d e b at e in ou r p a r liam e n t a r y s yst e m s, t h e n on e m u s t ra ise t h e issu e o f e q u a l a cce ss t o t h e e xp e rt i s e re q u i re d fo r e va lu at io n . Eq u a l a c ce ss h a s t w o m a in d im e n s io n s. Th e first re lat e s t o p u b lic d i s c l o s u re o f e va lu a t io n s co n d u ct e d b y p u b lic s e r vic e s o r p a rlia m e n t a r y o ffice s, i.e ., r e s u l t s th at sh ou ld b e p u b licized as wid ely a s p o ssib le. I a m aw are t h at m a n y sh a re a p e ssim istic vie w o f h u m a n n a t u re a s t o wh e th e r a cce ss to in fo r-m a t io n a n d q u a lit y kn o wle d ge ir-m p r ove s o u r societ ies (Re vel, 1988). A Ma c h i a ve llian vie w o f g ove r n a n c e a ls o t e n d s t o a rgu e ove r la Raison
d ’ É t a t. Bu t a n o rga n izat io n se ld o m h a s th e ca -p a city to a d a -p t itse lf fro m th e in sid e, a n d it of-te n n e e d s “e xo ge n o u s sh o c ks” t o im p rove. Mo re ove r, in t h e c as e o f p u b lic se r v i c e s, c it i-ze n s a re o ft e n ca p t ive c u st o m e rs. Eva l u a t i o n m a ke s p u b lic services m o re a ccou n ta b le to th e p e o p le th e y a re su p p ose d t o se r ve. My se co n d p o in t is m o re u t o p ia n . In d e m o c ra t ic n at io n s, a cc e s s t o fr e e le ga l c o u n se l is gu a ra n t e e d for t h ose wh o ca n n ot afford t o h ire a lawyer t o d e-fe n d th e m in cou r t . T h e re is n o gu a ran t ee t h at t h is la wye r will d o t h e b e s t wo rk in t h e wo rld , b u t a t lea st fre e le ga l aid is p rovid e d . Access t o e va lu a t ion is ce r t ain ly n o t d ist r ib u t e d e q u a lly a m on g socioe con om ic cate go ri e s. So m e act ors h a ve t h e re s o u rce s t o b u ild t h e ir ow n e va l u a -t io n o f p u b lic se rvice s a n d u se -t h is -t o lo b b y, p ro m o t e, o r p r ot e c t t h e ir in t e re s t s. Is it p o ssib le t o im a gin e t h a t p o lit ic a lly w e a ke r co n st it u e n cies cou ld h o p e t o co u n t e rb a la n ce eco n o m ic power a n d b e su p po rt ed b y p u b lic m on -ey to d e velo p th eir own eva l u a t i o n ?
Ha ve we re a lly m e t t h e s e re q u i re m e n t s in Fra n c e, a s t h e re c o rd o f ach ie vem en ts list ed b y H a rt z m igh t s u gge st ? Ac t u a l l y, w e a re s t ill fa r s h o r t o f m a n y o b je c t ive s , in b o t h t h e h e a lt h s e c t or a n d o t h e rs. Th e ro le o f p a rlia m e n t a ry e va lu a t io n is m o d e s t b e ca u s e o f t h e m o d e s t l e ve l o f re s o u rc e s in ve st e d , e va lu a t io n is fa r f ro m b ein g acce p te d a s a n orm al m a n a ge m en t p ra ct ic e in p u b lic se r v i c e s, a n d e va l u a t i ve re
-s e a rch la ck-s le git im acy. Bu t it m a y w ell b e th a t h e alt h will b e a m od el fo r ot h e r se ct o rs of p u b -lic in t e r ve n t io n , b e c a u se o f se ve re e xo g e n o u s s h o c k s, n o t o n ly fin a n c ia l, b u t a lso s cie n t ific, a n d t h ro u gh t h e e m e rge n c e o f m a jo r p u b lic h e a lt h issu e s su ch a s “n e w p ove rt y ”, AIDS, p ri-o n s, p ri-o p ulatiri-o n a gin g, a n d ri-o th e rs.
H IRSCH MAN , A. O., 1970. Ex i t , Voice an d Loy a l t y. Bo sto n : Ha rva rd Un i ve rsit y Pre s s.
