Cad . Saúd e Púb lic a, Rio d e Jane iro , 15(2):229-259, ab r-jun, 1999
Inst itutionalizing t he evaluation
of health programs and policies in France:
cuisine int ern a t i o n a l e
over fast food
and
sur mesure
over re a d y - m a d e
Institucio nalizando a avaliação de p ro g ramas
e p o líticas d e saúde : c ulinária e co rte -co stura
nas liçõ e s franc e sas
1 De p a rtam en to de Ep i d e m i o l o g i a , Escola Na cion a l d e Saú d e Pú b ica, Fu n d ação Osw ald o Cru z. Ru a Leop old o Bu lh ões 1480, 8oa n d a r, Rio d e Ja n e i ro, R J 2 1 0 4 1 - 2 1 0 , Bra s i l .
Zu l m i ra M . A . Ha rtz 1
A b s t r a c t Th e p u rp ose of t h is a rticle is t o d escrib e seve ral ch ron ologica l m ileston es in in st it u -tion alizin g th e ev alu a-tion of p u blic p ro g ram s an d p olicies in Fran ce from a govern m en tal p er-s p e c t i ve an d in the h ealth er-sector, er-situ atin g er-su ch re f e ren ceer-s in th e in tern ation al con tex t . Th e in er- sti-tu t ion a l n a sti-tu re of evalu ation im p lies in tegratin g it in to a n action -orien ted m od el, lin k in g an a-lyt ical act ivities to m a n agem en t, th u s con stit u tin g t he form u lation of a n eva lu a tion p olicy for p olicy eva l u a t i o n . Th e stu d y focu ses on issu es rela ted t o th e st ru ct u re ,p ra c t i c e , a n d u tilization of e va lu ation resu lts as w ell as oth er ch aracteristics p rovid in g the Fren ch m od el w ith a certain resistan ce to tra d ition al “ f a s t f o o d ”or “re a d y m a d e”m eth odological ap p ro a c h e s . The in stitu tion -alization of sectorial evalu ation ap p ears m ore p rom isin g th an th at of t he gove r n m e n t ’s cen tra l-i zed ch an n el, d esp l-ite th e w ork d evelop ed by a Scl-ien t l-ifl-ic Evalu atl-ion Cou n cl-il, a n d su ggests av-en u es for reflection an d d eba te p erta in in g to the Braz ilian Un ified Health Sy s t e m .
Key word s Pro g ram Eva l u a t i o n ; Health Po l i c y ; Health Sy s t e m ; Health Pl a n n i n g
R e s u m o Este texto tem o p rop ósito d e d escre ver a lgu n s m arcos cron ológicos d e in stitu cion a liza-ção da ava lializa-ção d e p ro g ram as/p olíticas pú b licas da Fra n ç a , n a p ersp ectiv a govern am en tal e n o setor saú de, c o n t extu aliz an d o-os n o âm bito in tern acion al. O cará ter in stitu cion a l d a ava l i a ç ã o su p õe in tegrá -la em u m m od elo orien ta d o p ara ação, ligan d o ativida d es an alíticas às d e gestão, con st it u in d o a ssim u m a form u lação d a p olít ica d e a valia çã o p a ra a val ia çã o d e p o lít ica s. S ã o focaliz ad as qu estões relacion a d as à estru t u ra , à p rática e à u t iliza çã o d os resu ltados d a ava l i a-ç ã o, b em com o ou tras ca racteríst icas qu e con ferem ao m od elo fran cês u m a certa resistên cia ao
“fa st food ”ou “p r ê t - à - p o rt e r”d as ab ord a gen s m etod ológica s tra d i c i o n a i s . A ava liação setorial se re vela m ais p ro m i s s o ra em su a in stu cion a lização d o qu e o d isp ositivo cen tra liza d o do gove r-n o, ap esa r d o trab alh o d eser-n volvido por u m Cor-n selh o Cier-n tífico d e Ava l i a ç ã o, su gerir-n d o p ista s d e re f l exão e deb ate n a p ersp ectiv a d o SUS.
“ On e im p ortan t qu estion th en , w h en on e com -p a res ex-perien ces from on e cou n try to a n oth er, is w h at lesson s ca n be learn t...there is a lw a ys a q u estion abou t learn in g from th e errors of ot h-e r s . Doh-es on h-e ha vh-e to go throu gh t hh-e sam h-e path, to be con vin ced th rou gh p erson al exp erien ce of w h a t is right or w ro n g , e v en t h ou gh on e h as been told, or can on e sk ip stages t o com e d ire c t l y to w h at seem th e right p oin ts?”( Po u vo u rv i l l e, 1 9 9 7 : 1 7 0 ) .
I n t ro d u c t i o n
Wh ile evalu ation of p ub lic p ro g ram s an d p oli-cie s ap p e ars t o b e a c o n sen su s in ligh t of th e n eed to kn ow th e effects of such in terve n t i o n s, t h e m od els u se d in th eir in stit u t io n alizat io n , th at is, th e stru c t u re s, age n c ies in ch arg e, ob -j e c t i ve s, m e th od s, an d u tilization of re s u l t s va ry from on e coun try to an other. In Brazil an d oth e r Sou th Am erica n c ou n tri e s, th e p ra c t i c e o f eva lu a tion as a p u b lic fu n c tio n is ra re, al-thou gh it h as occu p ied in creasin g space in th e le ga l a n d tech n ic al a n d sc ie n t ific lite ra t u re ( Ha rtz & Po u vo u rv i l l e, 1998). I b elieve, h owe v-e r, th at on v-e can takv-e ad van ta gv-e of th v-e lv-esson s l e a r n ed fro m th e m o re ad van ce d c ou n tries in e valu ation p ro g ram s for th e evaluation of p rog ra m s, as in th e case of th e US m odel for arogen -c ie s in -ch ar ge of p u b li-c h ealth in t erve n t i o n s ( PH S, 1996) o r th e d iffic u lties e xp e r ien c ed by t h ose wh o h ave m ore rec en tly b egu n to b u ild an evalu ation p olicy for the evaluation of p olic i e s. Fran olice is on e of th e latter, with th e ad va n -t age o f b ein g on e of -t h e Eu ro p e an co u n -t ri e s with a m a jo r in fluen ce on Bra z i l’s state m od el for p u b lic ad m in istrat ion , b esid e s d isp layin g g reater sim ilarity to our u n iversity an d scien tif-ic re s e a rch in fra s t ru c t u re, a s high light ed by Novaes (1992) in citin g it as a re f e ren ce for u n -d e rstan -d in g th e -d eterm in an ts in the in cor p oration an d dissem in ation of n ew m ed ical tech -n ologies by the h ealth sector.
My ob jective h ere is to p resen t th e ch ara c-t e r isc-tic s of c-t h e Fren ch exp e rie n ce b ased on a re vie w of t h e lit era t u re, wit h a c h ro n o l o g i c a l s u m m a ry of som e even ts related to th e eva l u a-tion of p u blic policies in gen eral, first, an d then c o n c e n t rate on the h ealth sector, wh ere I a d d e d t o th e b ib lio gra p h ic al in fo rm a tion t hr ou gh a visit an d in ter views with th e p e op le in ch arg e of the Fren ch evalu ation u n its at the m in ister i-al le vel, ad a p tin g th e in t ervie w p ro tocol used b y Love (1996). Th is p r oc ess is ac c om p a n ied b y c om m e n ts on o th er in te r n at ion al exp e ri-e n cri-es with in stitu tion alization , d r awin g on com m on den om in ators or a certain sp ecificity H A RTZ, Z. M. A.
2 3 0
C ad. Saúde Pública, Rio de Janeiro , 15(2):229-259, abr-jun, 1999
in t he Fren c h co n te xt wh ic h c on tr ib u ted to a reflection a n d d eb ate on ad ap tab le/ avo i d a b l e path ways or sh ortcuts for a sectorial eva l u a t i o n p olicy for the cou n try.
I begin with som e in form ation ab out Fra n c e an d its h ealth system . Th is deve lop ed cou n tr y, with n early 60 m illion in h a bitan ts, is a p arlia m e n t a ry dem ocratic rep u b lic wh ose Co n s t i t u -tion defin es health as a fun d am en tal right of all c i t i ze n s, sp e cific ally en su r in g h ealth p ro t e c-tion for m oth ers, ch ildren , an d th e elderly. T h e g ove r n m e n t (Cou n c il o f Min ist ers) e xec ut es th e p ar lia m e n t ar y laws an d is resp on sib le fo r the p ro g ra m s a n d p olicies th a t h ave b een ap -p roved . Loc al an d region a l gove rn m e n t s are re sp on sib le fo r ad m in isterin g lo c al serv i c e s un d er the aegis of the Mi n i s t ri e s, as in th e case of t he p u b lic h o sp ital n e twork (t wo- t hird s of th e 247,813 b ed s). Th e ce n tra l gove rn m en t is in c h a rge of h ea lt h p lan n in g, estab lish in g th e m ec han ism s for regu la tion an d con tr ol (We i l l , 1995). Fran ce sp en t 9.8% of its GNP (Gross Na-tio n a l Pro d u ct ) o n h e alt h c are in 1993 a n d , with in th e Eu rop ean cou n tri e s, Fran ce has th e high est exp en d itu re. Th e value p er cap ita was $1,835 (in pu rchasin g power p arities) com p are d wit h $3,299 (USA), $1,213 (UK), a n d $1,815 ( Ge rm an y). Cove rage again st the fin an cial c o s t s of illn ess is n early u n ive rsal, with less th an 1% of th e p op u lation still withou t in su ran ce (To n n elier & Lu c a s, 1996). The cou n try ’s level of in -d u s t rial -d evelop m en t is sim ilar to that of su ch other n ation s as Can ada an d th e Un ited St a t e s, an d it faces th e sam e challen ges of re s t ru c t u r-in g its h ea lth system to m ee t n ew d e m a n d s f ro m th e eld er ly an d oth er p op u lation gro u p s ( Battista et al.,1995).
Cad . Saúd e Pública, Rio d e Jane iro , 15(2):229-259, ab r-jun, 1999
rea lity on wh ich on e in ten d s to a ct (He n ra rd , 1996a). Jorjan i (1994:69) sp eaks of “e va l u a t i o n of p u blic p ro g ra m s”, p rop osin g a co n ce p t u a l f ra m e w o rk co n sid e rin g in terse cto r ia l asp e cts of in terve n t i o n s, re p resen tin g “n ew ap pro a c h e s to govern an ce th at are chan gin g the basic t h i n k -in g of m od ern politics”.
The fo cu s on e va lu at ion of p ro g ra m s is a p ri o rity all over the world , in clu din g the acade-m ic litera t u re on the su bject, sin ce it is d efin ed b et ter a s an a p p r oach , favo rin g an em p iri c a l e valu ation of resu lts a n alo gou s to a scien tific te st o f t h e valid ity o f t he or ies t h a t ve r ify h y-p otheses y-p ertain in g to the association betwe e n th e m ea n s em p loyed an d t he effects ob tain ed ( Pe r ret, 1995). In Fra n c e, what is gen erally m e n -tion ed is th e evalu a-tion of p ub lic p olicies, bu t th is usually pertain s to evaluation of p ro g ra m s, p a rt ic u la rly with re g a rd to tr an sfe r s fro m th e econ om ic to th e social sector. In En glan d , a p a-p er on a-p olicy eva lu ation d eals sim u ltan eou sly with p r o g ram s a n d p o licies, wh ere th e n otion of p olicy is sim ilar to th at of th e Fren ch logic as th e m ea n s em p loyed t o a ch ie ve ob je ct ive s ( Pe r ret, 1994). Th e In t e r- A m e rican In stitu te for Soc ial De ve lop m e n t (In d e s), in its co u rse fo r La tin Am e ric an an d Ca r ib b e an le ad e r s, c om -b in e s in a c om m o n t ra in in g o-b je ct ive th e d e-sign a n d m an a ge m e n t of so cia l p o lic ie s a n d p rogram s.Sum m in g up , in Fran ce, as elsewhere, th e d istin ction is n ot always clear between p oli-cies an d p ro g ram s in eva l u a t i ve re s e a rch (Pe r-ret, 1995), as in the case of Germ an y (Wollm an n , 1997), Swed en (Fu ru b o & Sa n d ah l, 1996), a n d Sp ain (Ra m o s, 1996; Ba l l a rt, 1997), wh ich ju sti-fies th is p re f e ren c e for t rea tin g t h e m jo in t ly f rom th e p oin t of view of in stitution alization .
To in stitu tion alize evalu ation in th e sen se e m p l oyed h ere m ean s to in tegrate it in to an or-gan ization al system whose b eh avior it is cap
a-ble of in fluen cin g, i.e., an action -orien ted m o d-el n ec essa rily lin kin g a n alytica l an d m an a ge-m en t activities (Ma y n e, 1992). The in stitu tion al n a t u re of evalu ation also p resu p poses a form a l d efin ition of th e re s p e c t i ve com m an d re s p o n -sib ilities (i.e., th ose wh o com m ission th e eva l-u a t i o n ) an d th e eva l l-u a t o r s, so th at in p ri n c i p l e th e resu lts o f the kn owle d ge p rod u ced can b e ap p rop riated an d in tegrated in to their own v i e w o f rea lity (Pe r ret, 1995). Ac c o rd in g t o Ma y n e (1992), th e d ecision to in stitu tion alize eva l u a-tion at th e fed era l gove rn m en t level re q u i res a n ation al d efin ition of a m in im um set of p olicy gu id elin es to b e in cor p orated in to ou r d iscu s-sion , such as:
• p u r p oses a n d re s o u rce s at trib u t ed to th e e valuation (stru c t u re ) ;
• location an d m e th od ological ap p roach es of the tier(s) in the evaluation (p ra c t i c e ) ;
• relation s esta b lish ed wit h m a n agem en t an d decision -m akin g (utilization ).
Evaluation of public policies in France ( Table 1)
Bio n (1994) lin ks th e b egin n in g of t h e Fre n c h in stitu tion alization p rocess to th e p u b lication of th e R a p p o rt De l e a u(1986), b u t th e foun d in g o f th e Office Pa rl e m e n t a i re d’ É va lu ation d es Choix Scien tifiqu es et Te c h n o l o g i q u e s ( 1 9 8 3 ) m e rits equ al atten tion , h avin g been in sp ired by th e Office of Tech n ology Assessm en t, lin ked to t h e US Co n g re s s, an im p or tan t m ilesto n e in the in stitution alization of Co n g ression al eva l u-a t i ve p ru-a c t i c e s. Th e ob jective of th e Office Pa r-l e m e n t a i reis to in form Pa rlia m e n t in su c h a way a s to o r ien t d e cision - m a kin g (Ma u r y, 1997). It co n sist s of eigh t De p u ties a n d eigh t Sen ators with n o h iera rch ic al ascen da n cy a n d
Tab le 1
Institutio nalizatio n o f e valuatio n in France .
P e r i o d R e f e rence Events
1 9 8 3 O ffic e Parle me ntaire d’ Évaluatio n de s Cho ix Scie ntifiq ue s e t Te c h n o l o g i q u e s 1 9 8 6 R a p p o rt De le au
1 9 8 8 R a p p o rt Vi v e re t
1 9 8 9 Prime Minister’s Me mo randum
1 9 9 0 D e c re e o n the e valuatio n o f p ublic p o licie s: I n t e rministerial Evaluatio n Co mmittee (Cime ) Sc ie ntific Evaluatio n C o uncil (CSE)
H A RTZ, Z. M. A.
2 3 2
Cad . Saúde Púb lic a, Rio de Janeiro , 15(2):229-259, ab r-jun, 1999
is a ssist ed b y a 15-m em b er scien t ific b o ard . D em an d fo r eva lu ation s o n issu e s m a y com e f rom th e two Ch am b ers th em selve s, th e ch air-m an of a p olitic a l p art y, a grou p of Se n a t o r s (m in im u m of 40), or D ep u t ies (m in im u m o f 60), or by its own d ecision . A feasib ility an aly-sis b ased on state-of- the-art kn owled ge of th e issu e at h an d p reced e s all stu d ies, a n d on e or m o re ra p p o rteurs are ch osen an d p rovided w i t h t he n e ce ssar y m ea n s to p erf o rm th eir tasks (with the p ossibility of perf o rm in g au d its in all state a gen cies excep t on issu es of n ation a l d e-fen se). Desp ite all th ese facilities, Ma u r y (1997) p oin ts to the lim its of its con trib ution : th e work is con cen trated on p rob lem s rath er th an p u b -lic p o-lic y p er se; t h e au to n om y an d p owe r o f t h e Scien tific Co u n c il are lim ited an d su b jec t t o d elega tion with ou t p rop er m et h o d ological adju stm en ts; an d n o re p o rt has b een p ro d u c e d n or h as the Bo a rd m et sin ce 1994. Soon , le p ar-l em en t a, u n p eu co m m e M . Jo u rd a in fa it d e la p ro s e , to u jou rs fa it d e l’év alu ation ( Ma u r y, 1997:2).
Th e R a p p o rt De l e a u(1986) p resen ts eva l u a-tion in the Fren ch p olitical an d ad m in istra t i ve con text as a m ean s to iden tify an d m easure th e effects of p ub lic p olicies with m eth od ological ri g o r, a d o p t in g an exp er im en ta l or cogn it ive p e r s p e c t i ve, u tilizin g objective data an d exter-n a l e valu at ors b a sed oexter-n th e c o exter-n ce p t of on n e p eu t pa s être ju ge et p artie ( Bion , 1994). Me a n -w h i l e, the R a p p o rt Vi ve re t(1988) cri t i c i zes th e p recedin g rep ort, assum in g a p olitical app ro a c h t o evalu at io n t h a t is n o lo n ger co n sid er ed a m e a s u rem en t tool. Ac c o rdin g to its view, elect-ed re p re s e n t a t i ves are th e on es th at can legiti-m ately p ass ju d glegiti-m en t on th e value of p olicies, c o n s i d e rin g th eir in h eren t su bjectivity. Eva l u a-tion is th u s seen as n ec essa rily con tra d i c t o ry, e n s u rin g th e p lu r ality o f p oin ts of view an d ad din g at r i b u n i c i e n n efun ction (Bion , 1994).
In 1989 th e Prim e Mi n i s t e r’s Me m o ra n d u m , c o n s i d e red the backb on e for re n ovatin g p ub lic a d m i n i s t ratio n , took on th e task of p ro m o t i n g p olicy evalu at ion , resu ltin g in 1990 in th e De-c ree on th e Evalu ation of Pu bliDe-c Po l i De-c i e s. In s t i-tu tion alization was form a l i zed u n d er the aegis of the Ex e c u t i ve Bran ch an d con sid ered essen -tially re l e van t for th e cen tra l gove r n m en t. T h e offic ial text d rew on ten d en cies fro m p re v i o u s re p o r ts (Pe r ret, 1995) an d c re at ed t h e In t e r-m i n i s t e ria l Evaluation Cor-m r-m ittee (Cir-m e) an d th e Scien tific Evalu ation Cou n cil (CSE).
Th e Cim e is cha ired by th e Prim e Mi n i s t e r, h avin g as p erm an en t m em b ers the Min isters of Plan n in g, Ec o n o m y, Bud ge t, In t e ri o r, an d Pu b-lic Affairs, bu t with n o re p resen tation from Pa r-liam en t, th u s m akin g it im p ossib le for th e
lat-ter to con ven e it (Ma u r y, 1997). An eva l u a t i o n by t he Cim e p r ovid es th e re s p e c t i ve Mi n i s t e r wit h th e p ossib ilit y o f a cce ssin g in form a t i o n on p ro g ra m s, thus in a sense cuttin g across va r-iou s ad m in istra t i ve ech elon s.
Th e CSE is n am ed by th e Prim e Min ister for a term of six ye a r s, an d its p articipan ts are ch o-sen on th e basis of p roven com p eten ce an d are com m ission ed to d efin e a deon tology for eva l-u ation with re g a rd t o t h e fo rm an d m et h od s bu t n ot the bien fon d éof a give n p roject. Its au -ton o m y is in ten d e d to b e gu aran t eed b y t h e n o n - re n ewa l of it s t erm an d th e fa ct th at it is n ot lim ited to dem an ds raised by the Cim e. It is also su p p o sed to gath er an d p u b lish in form a-tion p er tain in g to evalu aa-tion an d con tribu te to th e train in g of sp ecialists an d th e deve l o p m e n t of eva l u a t i ve re s e a rch . Pr i o rity p r oje ct s fo r Cim e are exp ected to be an alyzed by the CSE in resp o n se to fu n d in g by th e FN DE (Fon d N a -tion a l d e Dévelop pem en t d e l’Éva l u a t i o n). T h e Co u n c i l’s roles also in clu de m ethodological as-sistan ce (CSE, 1996) an d th e fin al op in ion in a st u d y is b ased on th e c o her en c e b et ween t h e re p o r t an d o t h e r stu d ies on t h e sam e th em e, as well as th e recom m en d ation s an d a n a lysis. Th e se at tr ib u tion s a llow th e Cou n cil t o c ap i-t a l i ze on e xp e rie n ce s a n d i-to b e co m e fam ilia r with t h e lim it s an d fac ilitie s e xp e r ien c e d b y e valu ators (CSE, 1996). Th e Econ om ic an d So-cial Co u n cil, a t h ird co n st it u t ion al a sse m b ly, with re p re s e n t a t i ve s m ain ly fr om a m on g t h e so c ial p a r t n e r s, m ay u se u p t o 20% o f t h e FN DE with th e con d ition th at t h e p ro jec ts b e su b m it te d to th e CSE (Pe r ret , 1995). The C o-m issa riat Gén éra l du Pl a np re p a res th e ru l i n g s by the Cim e, over se e s th e en forcem en t of d e -c i s i o n s, m an ages th e FNDE, an d raises an d or-g a n i ze s t h e d em a n d fo r eva lu a tio n fr om th e Mi n i s t ri e s.
T h eM e m o ran d u m on St a t e Region al Con -t rac-tu aliza -t ion (1994-1998) p rovid es fo r t h e e valu at io n of cer t ain p ri o r ity p ro g ra m s, an d p rojects are expected to com p ly with the a b ove -m e n tio n ed gu id elin e s. Ac c o rd in g to Mo n n i e r (1995), even wh ile th e p ractice of con tra c t u a l-iza tion in a con text o f co- resp o n sib ility ru n s th e risk of b ein g lim ited to a sim p le re g u l a t i o n , it is in terestin g to n ote th at certain adm in istration s h ave p e r m an en t in st itu istration ally est ab -lish e d m ech an ism s, a n d th ese loc al d e vic es h a ve gen erally im itated th e in stitu tion al arc h i-t e c i-t u re ad op i-ted by i-the Prim e Mi n i s i-t e r.
Cad . Saúde Pública, Rio d e Jane iro , 15(2):229-259, ab r-jun, 1999
e valuation s we re the sam e as those of th e 1983 ru lin g, b u t wit h m ore lim ite d p owe rs (Ma u ry, 1997). Pe r ret (1995), a m em b er of the CSE, also reca lls th at o th er a ge n cie s su c h as th e CNE (Com ité Na tion a l d ’ É valu at ion d es Ét ablisse-m en ts Pu bliqu es à Ca ra c t è re Scien tifiqu e, Cu l-t u rel ou Pro f e s s i o n n e l), CN ER (Com it é N a t i o-n a le d ’ É valu atioo-n d e la Re c h e rc h e), a n d th e
Cou r des Com p tes m a y a lso p a ss ju d gm en t on th e e ffic acy of p u b lic a ct io n s. Ne ve rt h e l e s s, Ma u r y (1997) o b ser ve s th at t h e le gislation is n ot p re cise ab ou t th e o b je ct ives o f th e n e w p a r l i a m e n t a ry office an d se es ot he r issu es as p roblem atic: th e absen c e of a Scien tific Co u n -cil (tu rn ed d own by th e Sen ate) an d the lack of a c er tain p er son n el alloca tion or fixed b u d get.
The international cont ext
To get a b etter u n d erstan d in g of th e tren d , we fin d it re l e va n t to a d d a fe w r e m a rks on o th er e x p e rien ces arou n d th e wor ld . From a less re-s t r icte d h ire-sto ric al p e r re-sp ec tive, th e roo tre-s of e va lu atio n of so cia l p o licies or p ro g ra m s d at e far back, b ut they we re on ly form ally exp re s s e d in th e West in th e 1960s, th rou gh th e Pl a n n i n g Pro g ram m in g Bu d getin g System (PPBS) logic, b egin n in g in th e Un ited States an d followed by Can a d a , Ge rm a n y, a n d Swed en . Du r in g th e 1980s there was an explosive tren d in its im ple-m e n t at ion in th e Ne t h e r l a n d s, De n ple-m a rk, an d Sw i t zerlan d (Alb aeck, 1996). Du rin g that sa m e p e riod a n ew tren d also em erged in th e Un i t e d St at es a n d Can ad a wh ich ha s rece n tly ta ke n h old in th e Eu rop ean cou n tri e s, i.e., th e d eve l-op m en t of salien ce p erf o rm an ce aud its by the Su p r em e Aud it In s t i t u t i o n s : e f f i c i e n c y, effec-t i ve n e s s, good m an agem en effec-t, an d services qu al-ity h ave b ecom e th e focu s of issu es, con tra ry to th e em p h asis p reviou sly p lac ed o n legal a n d p ro c e d u ra l re q u i rem en ts in gove r n m e n t e x-p e n d i t u re s ( Po llit, 1997). Th is sam e t ren d h as been ob served in other cou n tries wh ere audit-in g sp h eres h a ve p u sh ed th e d evelo p m en t of e va lu ation t o the p oin t o f fo m e n t in g th e cre-ation of Eu rop ean societies of evaluators sin ce 1994. On e th u s un d erst an d s wh y th e Ge n e ra l Ac co u n t in g Offic e (GAO) cr eat ed in 1980 a n d t h e Pr o g r am Eva lu a t io n Me t h o d o lo gy Di v i-sio n ( PE M D - G AO) p lay su ch an im p ortan t ro l e in resp on d in g to d e m an d s b y Co n g ress a n d f rom d ifferen t cou n tri e s, m akin g th eir stu d ies a n d gu id e lin es ava i l a b l e. A goo d exa m p le in th e b iom e d ic al fie ld is th e p u b lic a tion C ro s s Design Sy n t h e s i s( G AO, 1992). As a top agen c y for verifyin g p ublic accoun ts (sim ilar to Fra n c e’s
la Cou r d es com ptes) an d lin ked to th e Am eri-can Co n g re s s, th is d ivision of th e GAO
special-i zed special-in eva lu atspecial-ion ha s a h u n d red e m p loye e s ( Pe r ret, 1995).
With re g a rd to th e stru c t u re o f in stitu tio n -alization m odels, m ost gove rn m en ts th at h ave im p le m en ted re l a t i vely suc cessfu l e va l u a t i o n p olicies (even thou gh with d ifferen t ap p ro a c h-es) have in clud ed in terest on the p art of bu dget s e c t o r s, as in the case of Can ada, Un ited St a t e s, Au s t ralia, an d Great Britain (Ma y n e, 1992). Acc o rd in g to the AcclassifiAccation p roposed by Mo n -n ier (1995), Fra-n c e wo u ld b e i-n clu d e d i-n t h is g ro u p o f cou n tr ies wit h greater m an agem en t e f f i c a c y: scien tific cri t e ria as th e b asis for legit-i m a c y; p roblem -solvlegit-in g; m ode rn legit-izatlegit-ion of p u b-lic services; an d u se of in dep en den t eva l u a t o r s with n o direct lin k to the bud get.
Hu d son & Mayn e (1992) com p are d th e Can a d ian an d US exp e rien c es an d fou n d th a t d e sp it e th e h ea vy in flue n ce from th e Un i t e d St a t e s, co n sid ered th e p ion e er in th e e va l u a -tion of social p ro g ra m s, th e two coun tries d if-fer in th a t evalu a tio n in th e Un ite d St at es is p a r t of a logic o f so cia l e xp erim e n t at io n t o ju d ge t h e effectiven ess of soc ial p ro g ra m s a p r i o r i, while in Can ada the im p lem en tation of such p ro g ram s is p ri m a rily con dition ed by th e p olitical debate. In relation to sp ecific fin an cial re s o u rc e s, althou gh there are few re f e ren ces in the litera t u re, Ca n a da sp en t U$28.5 m illion in 1991-1992 (CES, 1994), while th e Un ited St a t e s p asse d a law in t he 1970s au th or izin g th e u se of 1% of the fun ds allocated for health p rogram s to be used in evalu ation , a total of US$100 m il-lion (GAO, 1993).
a-H A RTZ, Z. M . A.
2 3 4
Cad. Saúde Pública, Rio d e Jane iro , 15(2):229-259, abr-jun, 1999
tion guidelin es. Th e p rin cip le of m in isterial re-sp o n sib ilit y d et er m in e s t h e p ra ct ice o f p ro-g ra m e va lu at io n , as it h a s d et erm in e d th at o f o th er s in th e a d m in istr atio n o f t h e Ca n a d i a n p a r l i a m e n t a r y syste m , wh ere e valu at io n a s a p rocess is m ore sectorial than cen tra l( Ma rc e a u et al., 1992).
Ac c o rd in g t o Hu d so n & Mayn e (1992), th e efficac y of th is Can ad ian in stitu t ion al eva l u a-tion device hin ged on an official m ech an ism of e x t e r n al assessm e n t of th e in ter n ally p er-f o rm ed evaluation p ri n c i p l e s. On e in dicator oer-f th e su ccess of th is n ew gu idelin e cou ld b e th e i n c e n t i ve for p r ofession alization exp ressed by th e Can adian Evalu ation Society (CES), foun d-e d in 1981 an d wh ich h a d 1400 m d-e m b d-e rs by 1991. On th e oth er han d , th is system was n eve r a l l - i n c l u s i ve. A re p o rt fr om 1993 sh owed t h a t o n ly on e -fo u rt h of p u b lic exp en d itu res h a d b een evalu ated from 1985 to 1992, while eva l ua tion s h uad b een su b ject to p ro b l e m s, c on c en t ratin g on ly on sm all ob servation al u n its with out an alyzin g the join t effects or e ver q u estion -in g th e ove rall cost o r e xisten ce of th e re s p e c-t i ve p ro g ram (CES, 1994).
Th e closin g of the OCG, as p ar t of the re o r-gan ization of th e evalu ation system con du cted b y t h e Tre a s u r y Cou n cil, su ggests th at a go od f o rm ula had still n ot been reach ed , a fac t th at was an ticip ated b y Ma rceau et al. (1992) wh en he stated th at the p olicy-m akin g an d adm in ist ra ist i ve m ach in e r y sistill fa ce d p r ob lem s in a d -ju stin g to th is “f o re ign b od y”, i.e., e va l u a t i o n . Ac c o rdin g to the n ew in stitu tion alization m od-e l, od-evalu at io n a n d vod-e r ific at ion wod-e rod-e su b ord i-n ated directly to th e Se c re t a riat of the Tre a s u ry Cou n cil, alth ou gh th eir e xecu tion is still a re-sp on sib ility of the Mi n i s t ri e s, wh o n ow have an in d ep en d en t p rofe ssion al in c h arge (v i s - à - v i s
the activities evaluated ), organ ization al visibil-i t y, an d d visibil-irect acc ess to th e Assvisibil-istan t Mvisibil-i n visibil-i s t e r ( Tu rgeo n , 1997). Th e Se c re t a ria t is lim ite d t o s u p e r visin g th e a ct ual im p lem en ta tion of t h e n ew p olicy an d th e evaluators (b ased o n th eir role with the m an agers an d p ro g ram heads) at-tem pt to facilitate q uality im p rovem en t an d ef-ficien cy in p ro c e d u res (th e m easures em p loye d m u st in clud e asse ssm e n t by u sers o f t h e re-s p e c t i ve p ub lic re-serv i c e re-s ) .
Ac c o rd in g to Mu l l e r-Clem m & Ba rn e s (1997), sin c e 1993 th is ch an ge resu lt ed in th e in ability of th e OCG to red uce the gap betwe e n p rom ises a n d p erf o rm an ce in gove rn m en t ac-t i v i ac-t i e s. Th is also in creased ac-th e r isk of a ac-th row-b a ck to th e h a rd fac ts row-by co n stitu tin g a sin gle g ove rn m en ta l sect or for au d itin g an d eva l u a-tion , m argin alizin g th e so-c alled soft p ro g ra m e va lu a tion p ro d u c t s. Th e issu e re m a in in g for
th e authors was “w h ether or n ot this order is too tall for th e evalu ator to fill”.
In m y op in ion this ch an ge ascribes a role to th e Tre a s u r y Cou n cil t h a t is sim ilar to th a t of the Fin an ce Mi n i s t ry in th e Au s t ralian m od el. In a six-year in ve n t o ry of the Au s t ralian system by Ma c Kay (1994), on e n otes excellen t p erf o r-m a n c e, to th e p o in t of h avin g p erf o r r-m e d 290 e va l u a t i o n s, th e m e r it of wh ic h is attr i b u t e d p a rticu larly to th e role of catalyst/ coord i n a t o r p l a yed by the Fin an ce Mi n i s t ry. Ac c o rd in g to th e au th or, “ . . . f i n a n c e’s a dvoca cy of ev a l u a t i o n as a w ort hw h ile activity is m ore lik ely to be in -flu en tial w ith portfolios th an if evalu ation h ad been th e respon sibility of a stan d-alon e sp ecialist organ izat ion th at ha s p erc e i ved as tan gen -tial to m ain stream govern m en t activities”, with th e Can adian m odel fittin g in to th is latter cate-g o r y. In Au s t ralia th ere has been a relaxation of c o n t ro l s, offering m ethodological sup port w h e n req u ested a n d fa vo rin g the co n str uc tion of n e t w o rks of eva l u a t o r s, wh ich “h elp s th e m an -agers m an age”. Without gettin g in vo l ved dire c t-ly in th e p ro c e s s, t he y c om p le m e n t th e wor k p h ilosoph y with th e slogan “lettin g the m an ag-er m an age”. Aw a rd s are p rovided to ach ieve the p la n n ed re s u l t s, a n d a th re e - year e va l u a t i o n p lan is m an d atory for each p ro g ram , th e re p o rt on wh ic h is usu a lly p u b lish ed . It is b elieve d th at th is p u b licat ion fo st ers u t ilizat ion of th e d at a an d q u a lity im p rovem en t in p rim e a p -p roaches in ad d ition to avoid in g “[steerin g] the e valu ation aw ay from the d ifficu lt q u estion s or su sp ect are a s”( Ma c Ka y, 1994).
c-Cad . Saúde Pública, Rio d e Jane iro , 15(2):229-259, ab r-jun, 1999
t e ristic of which was to kn ow th e p ro g ra m’s im -p act, relyin g on th e u se of q uan titative d ata. It c o r resp on d e d to a m o d el su ch a s the Ci v i c Textbook Vi ew of the Role of Scien ce an d Eva l u -ation Re s e a rc hc ri t i c i zed by Sab atier (1997) due to th e su p p osition t h at sc ien tists an d eva l u a -tors are n eu tral an d that th eir d ata are p re s e n t-ed in “u n b iast-ed fash ion to p olicy m akers”. T h e la tte r th eo ry, su r m esu re, id e n tifie d w ith th e c o n s t ructivist p arad igm , of a m ore q u alitative n a t u re, p laced the evaluator in a p osition of lis-te n in g t o th e p olitica l aren a , seekin g gre a t e r u t ilizat io n of th e re su lts t h r ou gh th e p erm an ean t ian vo l vem ean t of the va riou s p layers ian flu -en cin g th e d ecision -m akin g p ro c e s s. Th is char-a c t e ri ze d t h e “f o u r th st char-age o f evchar-a l u char-a t i o n” char-as iden tified by Gu b a & Lin coln (1990, a p u dCo n t a n d riopoulos et al., 1997), wh ich alth ough en -c om p a ssin g the a-c -cu m u la tio n of kn ow l e d g e f rom p reviou s sta ge s (valid it y of m easu re s, p ro g ra m effective n e s s, an d valu e ju d gm en ts), focused on th e n egotiation b etween p laye r s. It was also an alogou s to “e m p owe rm en t eva l u a -t i o n”, wh ich c ou ld b e -tran sla-ted as év a l u a t i o n axée su r l’au ton om ie,p rovid in g for active p ar -ticip ation by p ro g ram head s (an d ideally users, too) in th e evalu ation p rocess an d resultin g de-cision s (Rowe, 1 9 9 7 ) .
Ac c o rd in g to Ch en (1997), who form u l a t e d th e con ce p t of “t h e o r y- d ri ven eva l u a t i o n”, it is c r u cia l to recon cile q u an tita tive an d q u alitat i ve a n a lyalitatic al alitatec hn iq u es, wh ich are n oalitat con -flictin g, rath er con stitu tin g d ifferen t p ossibili-ties availa b le t o evalua to rs to d ea l with th e sp ecificity of th e p rob lem s en tru sted to th em . Th is focu s is b e co m in g a c on sen su s am on gst th e va r iou s in stitu t io n alizatio n m od e ls stu d -ie d . T h u s, on e a lrea d y n ot es in th e Ca n a d i a n gu id elin es from 1991 th a t evalu a tion s we re to b e d e sign e d w it h m u lt ip le lin e s of e vid en ce, i n clu d in g c arefully se lec ted q u an tita tive a n d q u a l i t a t i ve d ata ( Mc Qu een , 1992). In Sw i t ze r -lan d , wit h a p olitica l stru c t u re of th e t yp ically i n t e n s i ve d irect dem ocratic p articip ation type, d ata from evalu ation s are u sed exten sively for co n sen sus-b uild in g at th e fed eral a n d ca n ton l e vel to shed light on p rob lem s an d foster solu -t i o n s, em p loyin g m ul-tip le ap p ro a c h e s : ” m o re p o s i t i v i s t m o re con str u ct ivist; q u an t ita tive q u a l i t a t i ve; d ist an t p ar ticip an t; p ro s p e c t i ve -re t ro s p e c t i ve” ( Bu ssm an n , 1996).
Co n t ra ry to this in ten sive use of eva l u a t i o n as re p o rt ed b y Sw i t ze rla n d , a p re l i m i n a ry in -ve n t o r y of th e first fi-ve years of the Fren ch gov-e r n m gov-en t al ru lin g p r gov-esgov-e n tgov-ed by Pgov-e r rgov-e t (1995) s h owed th at on ly 17 p rojects we re an alyzed by th e CSE, wh ile Weill (1995) p o in t ed o ut th a t h e alt h p olic ies we re n ot a m on gst even th is
sm all gro u p. On th e p ositive sid e, Pe r ret iden ti-fied th e op en n atu re of d iscussion s with in eva l-u at ion sp h er e s, allowin g fo r a b r oad gr asp o f th e evalu ation issu e an d its resu lts by p olitical p l a ye r s, a lth ou gh th e effects on th e d e cision -m akin g p rocess we re in sufficien t. Leca (1997), c h a i rm an of the CSE, is som e what skep tic al of ru lin g, n otin g th at “e valu ation ap p ears to b e a s a c red cow ”, wh ere d e sp ite th e lofty in au gu ra-t io n o f ra-t h e CSE (in clu d in g even a sea ra-t fo r ra-th e Pre sid en t of th e Fr en c h Re p u b lic), in t ere s t wan ed in sub m ittin g p rojec ts d u rin g th e p eri-od fr om 1991 to 1996 (with 10, 7, 6, 4, 1, an d 0 p rojects p er ye a r, re s p e c t i ve l y ) .
e-H A RTZ, Z. M. A.
2 3 6
Cad . Saúde Púb lic a, Rio de Jane iro , 15(2):229-259, ab r-jun, 1999
s t roy th e p r in c ip le of im p u tation , in ord er for e valu ation to em erge from its m odest sp ace of m an a gem e n t c on t ro l to ach ie ve a gen er al re-co m p osition of relation s b etween ad m in istra-tion an d p olitics, it p resum es a resp on se to th e i n q u i ry in to the for m of d em ocracy con ceive d for th e Fren ch p olitical system (Rou ban , 1993).
C h ronology of sectorial evaluation in health (Table 2)
Th e p er in a ta l p ro g r am im p lem en t ed in th e 1970s wa s a m ileston e in sec torial in stitu tion -alization , illu stratin g th e relation s b etween d e-cision -m akin g an d tech n oscien tific kn ow l e d g e g e n e ra te d by eva l u a t i ve re s e a r ch . Ac c o rd i n g t o Ch ap ala in (1996a), an alysis an d eva l u a t i o n m eth odology was u sed for the first tim e in p u b-lic h ea lth at th e n at io n al le ve l. An ec on om ic a n a lysis (ex - a n t e) an d e p id e m io lo gica l su r -veys co n d u c ted by the In s e r m (In stitu t Na t i o-n al Spécialisé eo-n Re c h e rch e M éd icale) su p p ort-ed th e p olicy ch oices exp ressort-ed as ad m in istra-t i ve an d re g u l a istra-t o ry m easu res an d bu dgeistra-t cre d-its (Ration alisation des Ch oix Bu d g é t a i re s-RC B). Th is p olicy was based on PPBS, a m eth odologi-c a l le ga odologi-c y o f t h e US d e fe n se u se d in re n e w-in g p u blic ad m w-in istration an d w-in stitu tion alize d e valuation p ractices m en tion ed ab ove. On th e oth er h an d, it d id n ot in vo l ve the sam e US am -b ition s of m axim izin g the exp ected -ben efits as a fu n ction of costs, in cor p oratin g from th e b e-gin n in g th e critical view of Am er ican theore t
i-cian s wh o in th e late 1960s called atten tion to th e im p ortan ce of rep lacin g th e “o p t i m i z a t i o n” of p u b lic p olicy resu lts with th e sea rc h fo r a “s a t i s f a c t o ry n ature” in th e im provem en t of t h i s sam e p olic y. T h u s, p er in a tal stu d ie s we re a l-ways con sid e red in st ru m en ts fo r n egotiat io n b e t ween evaluators an dd é c i d e u r s, with a s t ro n g rela tion sh ip d eve lop ed righ t from t he b e gin -n i-n g of t h e stu d y (Ch a p a la i-n ,1996a). A-n o th e r i n i t i a t i ve th at d e se rves h igh ligh t in g in vo l ve s th e stu d ies b y th e Grou p e d e Réflexion su r l’É-con om ie des Tra n s p o rts Ur b a i n s( Mi n i s t è re des Tra n s p o rts/ École Nation ale Su p e ri e u re d es Mi-n e s d e Pa r is – 1974-1978), focu siMi-n g o Mi-n th e c o u n t ry ’s h igh way safety p ro g ram , wh ich ca n be viewed as a va n t - g a rd i s t e san d close to “e m -p owe rm en t eva l u a t i o n” or su r m esu ree va l u a-tion , un derscorin g th e m ech an ism s for n egoti-at io n an d id e n tifyin g a n eva l u a t o r’s lim it s as
l’hom m e d’ é t u d e, in th e cro s s road s between in -tellec tua l an d p olit ica l e n d eavo r. In a d d it ion , th ey con c eive d of an d p r ovid ed an exc e l l e n t e x e rcise in the ir ro le a s p ilot co m m it te e, d e -fin ed by th em as a gro u p act ivit y fa cilit at in g th e wo rk of th e c o m m a n d i t a i re in tu rn in g a vagu e q u estion in to th e ob ject of re s e a rch , ac -co m p a n yin g its -co n str u ct io n a n d p r ov i d i n g tec h n ic al su p p or t t o t h e re s e a rc he rs in th eir m ethod ological an d op eration al op tion s.
Th e year 1982 wit n essed th e fo u n d in g of th e Ced it (Com ité d ’ É valu a tion et de Di f u s i o n des In n ova tion s Te c h n o l o g i q u e s), lin ked to the p u b lic a ssistan ce d ep ar t m e n t of th e Pa ri s i a n p u b lic h o sp it al syste m (Ap - HP). Its ob jec tive
Tab le 2
Institutio nalizatio n o f sec to rial e valuatio n in he alth.
P e r i o d R e f e rence Event s
1 9 7 0 - 1 9 8 0 Evaluatio n o f p erinatal c are po lic ie s
1 9 8 2 C re atio n o f Ced it (C o mité d’ Évaluatio n e t de Difusio n de s Inno vatio ns Te c h n o l o g i q u e s) 1 9 8 5 R e p o rt b y Ministry o f Health o n e valuatio n o f te chno lo g ie s
1 9 8 6 C reatio n o f So feste c (So cie té Franç aise p o ur l’ Évaluatio n de s So ins e t Te c h n o l o g i e s) 1 9 8 7 C reatio n o f C o mité Natio nal p o ur l’ Évaluatio n Mé dicale de s So ins de Santé ( C N E M ) 1 9 8 9 C reatio n o f Ag e nce Natio nale p o ur le Dé ve lo p p e me nt de l’ Évaluatio n Mé dic ale ( A n d e m ) 1 9 9 0 Spe c ial c o mmittee fo r re s e a rch in p re ve ntio n and e valuatio n
1 9 9 1 Ho sp ital Ac t: Re g io nal Co mmittee s fo r the Evaluatio n o f Med ical and Ho sp ital C are (Cre mes) and Evaluatio n Bure a u in the Ho sp ital Dep artment, Ministry o f Health
1 9 9 2 Ac t o n RMO s-R é f é re nc e s Mé dic ale s O p p o sable s
1 9 9 3 B u reau fo r health evaluatio n and e c o no mic s, DGS-D i re ctio n Gé né rale de Santé Pub liq ue, M i n i s t ry o f He alth
1 9 9 4 D e c re e o n the missio n o f de ce ntralized se rv i c e s
Cad . Saúd e Públic a, Rio d e Jane iro , 15(2):229-259, ab r-jun, 1999
was to su m m ari ze scien tific da ta an d con d u ct st u d ies p r ior t o th e d issem in a tio n of tec hn o -logica l in n ova t i o n s, t h ere by h elp in g im p rove th e sec torial in stitution aliza tio n m od el. So m e of its stu dies showed the p ractical im p ort a n c e of the role of evalu a tors in m ediatin g b etwe e n scien tific kn owledge an d decision -m akin g, f ro m b oth t h e m ed ic al p o in t o f view (e ffic acy an d risks), th e organ ization al or econ om ic p oin t of view (im p act on fu n ction in g an d b u d get), an d th e ch a llen ges p osed by su c h p erf o rm an ce in t e rm s of p rofessio n al train in g (Po u vo u rville & Mi n v i e l l e, 1995).
In 1985, th e con clusion s from th e Re p o rt by th e Mi n i s t ry of Hea lth em p h asize d th e d efi-cien cies in the evaluation of m edical techn olo-g y, with Ced it a p p ea rin olo-g a s th e o n ly ava i l a b l e s p e c i a l i zed age n cy in Fra n c e. Th e re p o r t re com m en d ed th e crea tio n of a sp ec ialized au -ton om ou s fou n d ation at a con sen su s m eetin g that on ly occu rred with the creation of the An -dem (Agen ce Nation ale pou r le Déve l o p p e m e n t de l’Évalu ation Médicale), m ade p ossible when th e So cialist s regain ed th e m ajo rity in Pa r l i am en t (Weill, 1995). Howe ve r, I feel th at two in -t e rven in g fac-tors con -trib u -ted -to p u sh forw a rd an d im plem en t th e re p o r t’s re c o m m e n d a t i o n s. Th e first was the creation of th e So f e s t e c( S o c i e-té Fra n çaise p ou r l’Év alu at ion d es Soin s et Te c h n o l o g i e s ) ,c o n s i d e red a Fre n ch version of the In t e r n ation al Society for Quality Assura n c e with th e m ain objective of b rin gin g exp er ts t o-get h e r fr om va rio u s in st itu tio n s t o d issem i-n at e m et h o d s ai-n d te ch i-n iq u es wh ose re s u l t s h ad b een evalu ated . The secon d wa s th e n am -in g of th e Com ité N ation al p ou r l’Éva l u a t i o n M édicale d es Soin s d e San té (CNEM), wh ich was com m ission ed to discuss ethical p ro b l e m s an d m ethodological issues in in stitution al e va l-u a tion wit h a vie w t ow a rd s d efin in g n a tion a l p ri o rities an d b rough t lead er s an d auth o ri t i e s to ge th e r from th e h ea lth se ct or, b u t h a d n o b u d ge t o r fo r m a l tim eta b le o f its ow n (We i l l , 1 9 9 5 ) .
The creation of An d em in 1989 with the au-ton om y recom m en d ed in th e 1985 re p o r t fos-t e red fos-the d issem in a fos-tion o f evid en c e-b a se d k n owled ge in m ed ical p ractice an d h elp ed d e-fin e m e th od s fo r t ec h n o logica l e valu at io n . It also ser ved as a scien tific co n su ltin g b od y for th e Nation al Health In s u ran ce Fu n d (CNAMTS ) an d p h ysician s’ u n ion s. Th e origin al b ud get of US$1.5 m illion h ad in creased to U$5 m illion by 1992. Evalu at io n th em es are p rop osed by its Bo a rd (re p re s e n t a t i ves fr o m th e Mi n i s t ries of Health , Ed u cation , Re s e a rc h , an d Agr i c u l t u re, C N A M TS, CN EM, et c.). Evalua tion of m e d ica l te ch n olo gies h a s fo rm al statu s a s a n a tion al
p roje ct: Th e e m p h asis o n tec h n olo gy a ssess-m en t ssess-m u st b e p la ce d in t h e wid er co n text o f the Fren ch gove rn m e n t’s con cern ab out lack of e valu atio n of p u b lic p ro g ram s in gen eral d u r-in g a tim e of econ om ic difficulties. The n eed to assess p ub lic p olicy an d p ro g ram s was in d icat-e d by sicat-e vicat-e ra l ricat-e p o r ts a s a m u ch -n icat-eicat-e d icat-ed goal (Weill, 1995). From 1992 on th ere was close col a b o ration with the Agen cy for Heacolth Ca re Po co l-icy and Re s e a rch (AHCPR) an d by 1996 over 1 0 0 con sen su s m eetin gs h ad b een h eld (Du r ieu x & Ra vau d, 1997).
Th e field of p u b lic h ealth was en ha n ced t h rou gh th e Special Com m ittee for Re s e a rch in Pre ven tion an d Evalu ation , create d by In s e rm , o r igin ally as an a d h oc c om m ittee, with fu n d s com in g from th e n ation al health in su ran ce sys-t em . Th is c o m m isys-t sys-tee h as b e en an im p o rsys-t a n sys-t catalyst, sin ce ep idem iologists, econ om ists, a n d social sc ien tists are n ow m u ch m o re in vo l ve d than before in evaluation p rojects (Weill, 1995). Th e 1990 a d m in ist ra t i ve r e f o rm , re c o m -m en ding the decentralization of services thro u g h
Schèm as Région au x d es Organ isation s San itai-re s( S ROSS), was a stron g argu m ent for the Ho s-p it al Act of 1991 in th e ss-p h e re of h ea lth ca re system re f o rm . The n ew re g u l a t o ry fra m e w o rk re v i e wed the p rin cip les of th e 1970 Act, wh ic h o rg a n i zed m ed ic al c are on ly on th e b a sis o f s t ru c t u ra l in d ic a tor s o f th e b ed s/ in h a b it an t s t y p e, atte m p tin g to ad ap t th em t o th e h e alt h o b j e c t i ves estab lish ed by th e SROSS. It was an a tt em p t to m ove fro m a n a d m in istra t i ve typ e of logic to on e of op p ortu n ity orien ted by con -t ra c-t ua liza-t io n (Gu e r s - Gu ilh o -t, 1997) an d -t o b uild a “n ew p ub lic h ealth”, n ot on ly the re p ro-d u c tion of t h e cen tra l m o ro-d el, ro-d em an ro-d in g th e adap tation of in terven tion in strum en ts con sis-ten t with local n e ed s (He n ra rd , 1996). The law p rovid es for “th e n eed for eva l u a t i o n , resp ect for p a t i e n t s’ r i g h t s , an d t h e con cep t of u n ive r s a l h ealth care . Eva l u a t i o n , an im p ortan t yet u n de-fin ed con cep t, h as becom e t hrou gh th is law a lead in g chan nel for h ealth care re g u l a t i o n , m a n -agem en t an d plan n in g in Fra n c e”(Weill, 1995). A st ud y by Mic h el et al. (1997) give s con c re t e e xam p les o f th e p roc ess of e valua tion of p fession al p ractices an d a quality assuran ce p ro-g ram evolvin ro-g tow a rds a rero-gion al h ealth p olicy. In order to im plem en t this n ew sector ial eva l u-a tion p olicy, th e legislu-at io n p rovid es for th e c reation of two n ew m ech an ism s:
1) Region al Com m ittee s for the Eva l u a t i o n of Med ical an d Hosp ital Ca re (Cre m e s ) .
H A RTZ, Z. M . A.
2 3 8
Cad. Saúde Pública, Rio d e Jane iro , 15(2):229-259, abr-jun, 1999
region al p re f e c t s, com m ission ed to p rovide th e n e c e s s a ry m ethodological su p p ort to th e local l e ve l, wh ere ea ch p u b lic o r p ri vate h o sp it al sh ou ld evalu ate its activity in delive rin g quality c a re. Th ese com m ittees, n o t b ein g p erm a n e n t o rgan ization s an d with n o h um an re s o u rces or b u d gets o f th eir own , leave d ou b ts a s t o th eir e f f i c a c y. An d em rem ain s as su p p ort for the en -t i re sys-tem , sin c e i-t is in c h ar ge o f va l i d a -t i n g t h e m e th od s u sed in the p lan n in g p ro c e s s (Weill, 1995).
2) Bu re a u for Evalu ation in th e Hosp ital De-p a rtm en t of the Mi n i s t ry of He a l t h .
Co m m ission ed to fu n ction as a con su ltin g b ody for issues related to th e n ew h ospital m is-sion of evaluatin g care, this b road re s p o n s i b i l i-ty in vo l ves d efin in g ad eq u ate an d we l l - a d a p t-e d m t-et ho d s fo r t-eva lu atin g st-e r vict-es a s p ar t o f p o l i c i e s, u n d er th e ob jectives of p u b lic h ealth an d system p erf o rm an ce at th e local, re g i o n a l , a n d n a tion a l le ve l s. In d escr ib in g this n ew s t ru c t u re, Weill (1995) was con cer n ed with th e fact th at its staff was lim ited at th e tim e to on e p u b lic health p h ysician as a p erm an en t m em -b e r. Th e u n it’s m a in goal in th e first two -ye a r p e r iod was to con secrate the m i s e - e n - p l a c eof a q u ality assu ra n ce p o licy, wit h a c tu al com p li-a n c e by th e h osp itli-a l system , in wh ic h evli-a l u li- at io n was se en as on e of ath e fu n d a m en at al in -s t ru m e n t -s. A -sp ec ial fed eral b u d get a lloc at ed in 1994 as an in cen tive to hosp itals th at join ed t h r ou gh p rojects for th e d evelop m en t of m ea-s u reea-s in qu alité et secu rité san itaire is viewed as an in d icator of th e success of this stra t e g y.
The agen cy cu rren tly has a staff of four fu ll-tim e p rofession als an d a techn ical board ( m a d e u p of p hysic ian s a n d n u r se s) with a sm all weekly tim e allotm en t, allowin g for th e im p le-m en tation of th e bu reau an d its in itial p ro p o s-al. An exam p le of its activity was the u tilization o f “t racer c on d ition s” fo r q u a lit y eva l u a t i o n , a p p lied so as t o p ri o ri t i ze h e alt h p r o b l e m s a m on gst va r ious p op ulation gro up s an d re o r-g a n i ze h ealth care re s o u rc e s, in corp orated in to a p articip an t m e th od ology in the form of a c o n -f é ren ce d e con sen su s(sim ilar to th e p ro c e d u re s adop ted by An d em ) wh ich p roved to b e a facil-itatin g factor for th e h ealth re f o rm re g i o n a l i z a-tion p ro c e s s.
Und er the o rd on n an ces of 1996, th e Re g i o n -al Hosp it-al Agen cies (accordin g to p ublic h e a l t h o b j e c t i ves) are directly respon sible for con tra ct u a liza ction of goa ls, e va lu action of resu lcts o b -tain ed by va riou s establishm en ts in accord a n c e with th e gu id elin es o f th eSch èm as Région au x d es Organ isat ion s San itaire s( S ROSS) an d th e Nation al Agen cy for Ac c red itation an d Eva l u a-tion (ANAES, form erly An dem ). Au t h o ri z a t i o n s
for in itial an d on -goin g fun ction in g of serv i c e s re m ain u n d e r t h e b u rea u , wh ich is also in c h a rge of c on stru c tin g t h e eva lu a tion m od e l for m on itorin g th e set of n ew m easu res as p rvailin g p u b lic health p olicy. On e q u estion d e-m an din g an ie-m e-m ediate an swer an d su e-m e-m ari z-in g th is leve l’s con cer n is h ow to evalu at e t h e n ew ter ri t o rial organ ization of h ealth care.
Th e re g u l a t o ry m ac h in er y for sector ial in -stitu tio n alization was ap p en d e d in 1992 with th e Act on RMOs -R é f é ren ces Médicales Op p o s a-b l e s. These m an datory m edical gu id elin es we re d rafted by the An dem for out-p atien t c a re, b ols t e rin g th e logic of a m ed icin e baolsed on olscien -tific evid en ce (Po u vo u rv i l l e, 1997). Th e use of fin a n cial in c en tives a n d a r ticu lation wit h the p h a rm aceu tical in dustr y at th e tim e th e guide-lin es we re d r afted in d ic at es t h e fact ors th at we re to favor t h e ir im p lem en ta tion , to b e re -n e wed p er m a-n e-n tly b ased o -n co-n tri b u t i o -n s f rom eva l u a t i ve re s e a rch , b u t t h e att em p t to t ran sp o se th is to th e h osp it al a rea a p p ea re d im p ossib le (Du rieu x & Ra vaud , 1997).
With th e d ec re e on th e m ission o f d e ce n -t ra l i zed ser vices an d given -th e law’s im p era -t i ve w o rd in g, t he DRASS (Di rectio n s Région a les d’ Action Sa n itaire et Sociale) we re to b e in -vo l ved in th e evaluation of p u blic p olicies, an d each echelon , in coord in ation with other in stitu tion al p a r t n e r s, was to focu s on t he eva l u a -tion of p ro g ram s an d ac-tion s (Catin c h i, 1995). Th e sta te wa s t o b e th e “f o re m a n” o f eva l u a -tion , an d alth ou gh it was difficult to determ i n e wh ether on ly on e level (like th e Com ité tech n i-qu e région al et in terd é p a rt e m e n t a l) cou ld com -m an d, coord i n a t e, an d/ or p erf o r-m eva l u at i o n s, it was u p to th e state to en sure th e coexisten ce of in sp ection - c o n tr ol an d in sp ection - as-e valu at io n a n d th as-e n as-e gotia tas-e d a n d va l i d a t as-e d n o rm s an d m ethodologies. An oth er Fren ch au -th or (Ge f f r oy, 1994) h ad a lread y id en tified -th e com bin ation of evalu ation an d regu lation (dis-tin gu ish in g b e tween b u t n ot op p osin g the d o-m ain s of ve rification an d evalu ation , where the la tter allowe d for th e valid ity of re f e ren c es for regu la tion ) as the on ly a p p roa ch cap ab le of recon cilin g ethics, q uality of care, an d econ o m -i c s. Ac c o rd -in g to Schaetzel & Ma rch an d (1996), wh o an alyzed som e o f the se e xp er im en tal re -gion al p ro j e c t s, orien ted by PSAS (Pro g ra m m a-tion St ratégiq u e d es Aca-tion s d e San té), eva l u atio n b e ga n to a ct u ally d r aw on th e p art i c i p a -tion of local p laye r s, givin g it an “u n a vo i d a b l e c h a ra c t e r”. The followin g p hen om en a we re al-ready obser ved:
Cad . Saúd e Públic a, Rio d e Jane iro , 15(2):229-259, ab r-jun, 1999
• e m e rgen ce of p ro g ram s orien ted by a p op u lation ap p roac h, in a b reak with th e p re p o n d e ran t con cern s of th e organ ization an d m an -agem ent of h ealth care stru c t u res an d activities; • a f f i rm ation of a com m on d esire on th e p art of th e DGS (Di rection Gén érale de San té Pu b l i-qu e) an d t h e EN SP (École N ation ale d e San t é Pu b l i q u e )of th e Mi n i s t ry of Health that eva l u atio n b e c on te m p la ted system at ic ally in t ra i n -in g career p rofession als; an d
• ra ised a waren e ss of a co n text su b jec t to b u d get con stra i n t s, in wh ich it is n ecessar y to b e a ccou n tab le a n d argu e for re s o u rce alloc a-tion .
Exam p les o f th e ro le th at eva lu a tion can p lay in the region alization p rocess are cove re d in th e work of Abballeia & Jo u rdain (1996), with th e em ergen cy m ed ical ser vices in Bo u rg o n n e an d th e u tiliza tio n of q u a lity tracer s for p lan -n i-n g th e SROSS i-n Lorrai-n e (Ga ri-n et al. 1995). T h e Bu re a u for Health Evaluation an d Ec on om ics of the DGS, established ion 1993 b ut oon -ly actu al-ly im p lem en ted in 1994, is in ch arge of defin in g th e p olicy objectives for evaluation of m ed ic al p ra c t i c e s, with th e b road er c h allen ge of in stitution alizin g th e legal, fin an cial, an d or-gan ization al asp ects of evaluation in the h ealth field ( Weill, 1995). It fe ll t o th is m in ist er i a l u n it to c on clu d e th e re g u l a t o ry fra m e w o rk for h ealth re f o rm in relation to th e gen era l i z a t i o n an d exp an sion of eva lu ation m ech an ism s d e-velop ed exp erim en tally by th e Bu reau for Ho s-p ita l Ad m i n i s t rat ion . With a sta ff of ju st o n e p ub lic h ealth m ed ical in sp ector an d two econ -o m i s t s, with a b ac k-u p sec re t a r iat, it b e ca m e clear th at it sough t to m ain tain this n on execu -t i ve p rofile vis-à-vis evalua-tion , -the ove rall exe-cution of wh ich was en tru sted to the ANAES. It is in terest in g t o h igh light th a t th e b u reau was c o n c e rn ed with m ain tain in g the c lear d istin c-tion b e tween in sp ec c-tion a n d evalu a tio n , d sp ite t h e d isco u rse o f th e a ud itin g b o d ies b e-in g orien ted e-in crease-in gly tow a rd s evalu ation of re s u l t s. Il l u s t ratin g this issue, th e p h ysician s in c h a rge of accreditation we re n ot in sp ectors bu t visitin g p h ysic ian s, a ctin g d irectly in h osp ita l c a re. Wh at rem ain s to be d ecid ed , wh ere irre g-u l a rities are fog-u n d , is how th ey articg-u late with th e Pu b lic In sp ection area.
Th e r ela tio n sh ip b et wee n d ifferen t b od ies i n vo l ved in h ealth evalu ation can b e illustra t e d by th e Na tion al Pro g ram for th e Pre ven t ion of Breast Ca n c e r, wh ose op eration al re s p o n s i b i l i-ty an d fu n din g com e from th e n a tion al h ealth i n s u ra n ce syste m (CN AM TS). The Ca h ier d es Ch arges (im p le m e n tatio n gu id e) was d ra f t e d un der th e coordin ation of th e An d em an d stan d a rd i zed by th e Na tio n al Pilo t Pro g ra m Co m
-m it tee o f th e D GS, t o orien t th e -m o n it ori n g an d evalu ation of th e p ro g ra m ’s im p act by th e C N A M TS with the sam e p art n e r s h i p s. In short , th e ro le o f th e b u reau a p p ears to c on c en tra t e essen tially on issu es p rop er to in stitu tion aliza-tion , i.e.: to in trodu ce an d m ain tain eva l u a t i o n at th e c en ter of tech n ical/ p olicy d ecision s de-p en d in g o n th e st at e a de-p de-p ar atu s a n d n ot on ly with th e role of en cou ragin g or facilitatin g un -d e r t h e aegis o f p ro fe ssio n al b o -d ies (wit h th e e xc ep tion o f ou t-p at ie n t evalu a tio n or m é d e-cin e de ville). Its role as catalyst was illu stra t e d , for exa m p le, b y th e sp e cial issu e on l ’ É va l u a-tion et San té p u b lish ed by th eHau t Com ité de la San té Pu b l i q u e, the ed itor of wh ich was th e head of th e b u reau , an d th e re g u l a t o ry d ecre e s (Ord o n n a n c e s )in th e h e alt h ca re syste m re-f o r m th a t m ad e p ossib le th e c rea tion ore-f th e ANAES (Nation al Agen cy for Ac c red itation an d Evalu ation in He a l t h ) .
Th e n ew laws o r Ord o n n a n c e s , th e m a in th em es of wh ich in vo l ved th e m aîtrise m éd ica-lisé of m ed ical care exp en ses (96-345) a n d the re f o r m of p u b lic a n d p r i vat e h osp ita liza tion (96-346), su bstan tially altered the pu blic h ealth an d health in su ran ce cod e. Th ey are con sisten t with th e stan ce taken by Ge f f roy (1994): follow-in g th e failu re of acc ou n tfollow-in g co st re d u c t i o n , on ly regulation b ased on evaluation would p ro-vid e legitim ac y for cost red u c tio n , b e yo n d a m e re ration in g to allow for an in crease in q ual-ity o f ca re. Goin g b eyon d th e lim its o f t re a t-m en t p ractices an d p rotocols to en cot-m p ass or-gan ization al p ractices focusin g on the solution o f p rob lem s at th e p op u lat ion leve l, th e m a in t h ru sts are the followin g:
• p lan n in g foc u sed on p ri o rit ies d e fin ed year ly in n ation al a n d region al hea lth con fe r-e n c r-e s, du ly backr-ed by th r-e Hau t Com ité de San -té Pu b l i q u e ;
• ad ap ta tion o f in it ia l m ed ica l t ra in in g a n d i n c e n t i ves for con tin uin g m edical edu cation ; • p rom o tion o f e xp erim e n tal co ord i n a t i o n m od els for th e out-p atien t an d in -hospital care system s over a five - year p erio d , un d er th e re-sp on sib ility of region al agen cies;
• t ra n s f o rm ation of An d em in to AN AES, ex-e c u t i vex-e ex-eva lu ation u n it p ex-e r sex-e coo rd in a tin g a n ation al an d local n etwork of exp ert s, am on gst oth er th in gs to articu late the activity of th e two m i n i s t e rial evalu ation un its.
H A RTZ, Z. M. A.
2 4 0
Cad. Saúde Pública, Rio de Janeiro , 15(2):229-259, abr-jun, 1999
et p rivés qu e da n s le cad re d e l’é xe rcice libera l , le d év elop p em en t d e l’évalu ation d es soin s et d es p ratiqu es profession n elles et m ettre en oeu -v re la p ro c é d u re d ’ a c c r é d i t a t i o n . . .”. No te th e a g e n c y ’s ab ility to acq u ire good s a n d p ro p e rt y an d to alloca te, from its own b u dget, “s u bve n -tion s,prêts ou avan cés à des person n es p u b l i q u e s ou p rivées qu i réalisen t d es étu d es, re c h e rc h e s , t ra va u x ou équ ip em en ts con cou ra n t à l’a pp li-cation d e ses m ission s”.In ad dition it can coop -e rat-e with in d ividu als an d corp oration s -eith -er Fren ch or foreign an d “n ottam en t avec d es é t a-blissem en ts d’ e n s e i g n e m e n t , de re c h e rch e ou de san té q u i on t d es m ission s id en tiqu es ou com -p l e m e n t a i res d es sien n es”.
With re g a rd to evalu ation , the agen cy’s an -n u al a-n d p luria-n -n ual p ro g ram was to focu s es-p ecially on the ees-p id em iological es-p ro f i l e, in clud-in g h ealth p r o b lem s an d t h eir re s p e c t i ve r i s k f a c t o r s, evolution of available tech n ologies (d e la préven tion à la réa n im ation), iatrogen ic ac-c i d e n t s, hosp ital in feac-ction s, an d m aterials an d e q u ip m en t still n o t valid ate d in t h e h e alt h fie ld . Ad m i n i s t ra t i ve a n d Scien t ific Co u n c i l s we re to share tech n ical an d fin an cial re s p o n s i-b i l i t i e s. Th e scien tific i-board h as a sin gle chair-p er son a n d two schair-p ecific sec tion s (eva l u a t i o n an d accred itation ) wh ose m em b ers are n ot re-m u n e ra te d exc ep t fo r f ra is d e d ep la cem en t et d e séjou r. A co llege o f a ccre d it at io n an d a re-seau n at ion al et loca l d ’ ex p e rt sp a r ticip a te in t h e ANAES m ission s. Mem b ers o f th e c ollege m ay b e re m u n e rate d , while t he word in g d o es n ot m en tion th is p oin t for th e re s e a u.
By way of con clu d in g th is br ief d escri p t i o n of sectorial in itiatives in h ealth -related eva l u a-t io n , we sh o u ld h igh ligh a-t a-t h a a-t ala-th ou gh in Fran ce th ere is still n o cer tified (a g r é é) p ro f e s-sion al m ilieu for evalu a tio n of p ro g ra m s a n d p olic ies (CSE, 1996), a n im p or t an t eva l u a t i o n m a rket is op en in g u p to re s e a rc h e r s. T h u s, p ri-vate con sultin g firm s specializin g in eva l u a t i o n o f he alt h serv i c e s, m ed ica l tec h n olo gies, a n d h osp ita l m a n agem en t, like the CNEH (Ce n t re Na tion a l d e l’Équ ip em en t Ho s p i t a l i e r, a sem i-p ublic organ ization u n til 1990), “establish d a t a-b a s e s, aud it h osp itals, an d re p o rt m ed ical p ro-jects for establishm en ts m ade legal by th e 1991 l a w” (Weill, 1995). An ot h er in d ica tor is fro m the graduate studies train in g sector, where m o re t ha n 20 cou rses a re listed in the An n u a i re d es Form ation s à l’Évalu ation M éd ica le en Fra n c e
(An dem ), 1997.
A rticles by Fr o s s a rd & Jo u rd ain (1997) an d Pa ra y ra (1997) id e n tify a p r om isin g scen a ri o for the p ractice of collective lear n in g by re g i o n -al p layers at a m om en t wh en cost-effective n e s s c ri t e ria are b ein g re c o n c e p t u a l i zed an d in fo
r-m a tion syste r-m s re s t ru c t u red , exercisin g n ew regu lation m od els that en h an ce th e local deci-sion - m akin g p ro c e s s. Th is d yn a m ism va l u e s p lan n in g an d valid ation of th eoretical m o dels at t h e lo c al level, sin ce “les fon ct ion n a ire s n e d o i ven t p a s s’ i n t e rd i re d e th éoriser” ( Ba sset & Ha u ry, 1995).
Be f o re co n textualizin g th e Fren ch tren d in the in tern ation al scen ario I sh ou ld re i t e rate m y ju stification for ha vin g u sed an in tegrated ap -p roach to the evalu ation of -p ro g ram s an d -p oli-c i e s, in som e oli-cases in volvin g m edioli-cal oli-care p ra oli- c-tic es a n d tec h n olo gie s in t rin sically re lated to th em . Alth ou gh I agree th a t th ese d im en sion s can b e treated sep ara t e l y, wh en ever an alyzin g th e effe ctiven ess of p u blic h ealth in terve n t i o n s a system ic c on cep tual fra m e w o rk is n ecessary to u n d e rstan d t h ese d im en sion s a s o b se rva -tion un its n ece ssa rily lin ked to th e eva l u a t i o n p ro c e s s. T h u s, a ch an ge in a p op ulation’s sta te of h ealth c an n ot b e lim ited to th at of a n in d i-vid u al, for wh om an eva lu at ion of th e clin ica l efficacy of m ed ical p ractices an d tech n ologies m a y b e su fficie n t. It re q u i re s o rg a n i zed p ro -g ram m atic action , d em an din -g collective ch oices p rop er to th e field of p u blic p olicioices. Me a n w h i l e, th e im p lem en ta tion of a p ro g ra m d e -ri ves from th e evolu tion of availab le p ra c t i c e s, whic h in tu rn are in flu en c ed by th e d eve l o p -m en t of -m ed ica l tec h n ologies, org a n i z a t i o n a l s t ru c t u re s, an d p olitical p ri o r ities fo rm u l a t e d fo r a c olle ct ive b od y (Ha r tz et al., 1997). Ac-cordin g to Schraiber (1997), “the p ro g ra m / p l a n” in re sp o n se to tech n ically (i.e., ep id em io logi-cally) de fin ed h ealth n eeds is p art of th e p u blic p o lic y field , a t lea st in Brazil. In th e c ase of Fra n c e, th is relatio n sh ip ca n b e illu strat ed b y th e evalu at io n o f th e b re ast c an c er sc re e n i n g p ro g ram fo rm ulate d as a gove rn m en t p olicy, i n volvin g as cru cial issu es for collective in ter -ven tion the stren gth en in g of p rofession al org a-n izatioa-n of rad iologists aa-n d th e m u ltip licatioa-n of m ed ical tests u sin g typ es of equ ip m en t with va ryin g d egrees of p e rf o rm a n c e (Grem y et al., 1 9 9 5 ) .
The international context
Cad . Saúde Pública, Rio d e Jane iro , 15(2):229-259, ab r-jun, 1999
syste m (Co n t a n d rio p o u los et al., 1997). Re-cou rse to evaluation is ju stified as an essen tial p ractice for th e ration aliza tion of m ed ica l ac -tivity an d d ecisio n s co n c ern in g re s o u rce allo-cation (Po u vo u rville & Mi n v i e l l e, 1995).
Co n s i d e rin g the im p ortan ce of cost con tro l in m ed ical care an d th e world wid e cr isis in so-cial secur ity system s, we are n ot su rp rised th at th e health sector was am on g the first to b en efit f rom the PPBS logic, as in th e case of th e RAND ( Re s e a rch an d De velop m en t) Co r p o ration , th e resu lt of an Am er ican strategic re s e a rch p ro j e c t d u rin g World War II, wh ich in 1968 laun ch ed its Hea lth Scien ce Pro g ra m (still o n e o f th e m ost i m p o rt an t in ve stm en ts in civilia n re s e a rc h ) wh en Me d i c a re an d Med ica id we re c re a t e d ( Gerbau d, 1995).
The h ealt h se ct or is a lso o n e of th e m o st i m p o rtan t on es in th e PE M D - G AO, to th e poin t of h avin g p erf o rm ed stud ies com m ission ed by the AHCPR, created by th e US Co n g ress in 1989 to an a lyze t h e e ffective n e ss of m e d ic al in t er-ven tion s as p u blic p olicy (GAO, 1992). Th e GAO led a st u d y o n evalu at io n s p erf o r m ed in th e 19881992 fiscal p er iod, a sort of m eta eva l u a -tion o n th e wo rk of th e va r io u s levels of th e Pu b lic Hea lth Se r vic es Pro g ra m Eva l u a t i o n , c on c lu d in g th at t h e eva lu a tion s h ad b e en in su ffic ien t a s a so urce of in fo r m a tio n for Co n -g re s s. An oth er m ore recen t re p o rt (GAO, 1995) d ea ls wit h a n evalu at io n of t h e d ec en tra l i z a-tion of gra n t s, c o n clu d in g in fa vo r o f gre a t e r flexib ility in th e freq u en cy of re p o rts re q u i re d of the States for th em to con cen trate on re s u l t s. Co n g ress on ly b e co m e s m ore p re s c ri p t i ve in c ases in vo lvin g in ad eq ua te in for m a tion sys-tem s on fu n d ed p ro g ra m s.
Ac c o rd in g to Myers (1996), wh ile th e Un i t-ed States an d Can ad a h a ve t h eir sp ec ific it ies, th ey h ave evo l ved u n d er th e sam e Co n t i n u o u s Qu alit y Im p rovem en t m od el, m ea n in g th a t e va lu ato rs in co r p o ra te scien tifica lly cre d i b l e in d icators re flec tin g p atien t sa tisfa ctio n . T h i s an d other ch an ges in eva l u a t i ve practices s p e a k in favor of p lu rality as th e rallyin g cry for in sti-tu t ion alizatio n m od els. Th e Pro g ra m m e d ’ Ac-tion Com m u n au taire p ou r les En f a n t s( PAC E ) , with an an n u al b u dget of U$33 m illion (10% of wh ich is a lloca ted for eva lu at ion ), ad op ted in th e n ew Can ad ian evalu a tion p o licy, is a good e xam p le of organ izin g a n ation al p r ê t - à - p o rt e r
an d a p r ovin cial su r m esu re( Tu rgeo n , 1997). Pe t t i g rew (1996), in En glan d, ob ser ves great in -t e res-t b o-th in exp erim en -tal m odels, ex-tra p o l a-tion of c lin ical tri a l s, a n d th e Am eric an t h eo-ries of “e m p owe rm en t an d d em ocratic e va l u a-t i o n” in sp irin g evalu a a-to rs e n ga ge d a a-t a-th e re-gion al or local level.
A cu rren t p ro g ra m in Qu éb ec , Sim ad (Se r-vices In ten sifs d e M ain t ien à Do m i c i l e), illu s-t r as-tes s-th e issue of p rop er us-tilizas-tion o f re s u l s-t s of evaluation for th e (re ) f o rm ulation of sector i-al p olicies, in ad d ition to showin g the com p le-m e n t a ry n atu re of an in itial (q uan titative) su r-vey from 1983, in d icatin g th e ap p a ren t failu re of a p re vious p ro g ram focusin g on the eld erly, an d a sub seq u en t q u a lit at ive c ase st u d y su ggestin g that the m ain p rob lem was th at of con -cep tu alization an d th e degree of im p lem en ta-t ion . Cob ata-t off (1996) su m m ar i ze d ta-t he lata-t ta-ter’s c r itica l reaction to th e form e r: “h ow cou ld th e au th or su p p ose ...t h at p erson s liv in g at h om e w ith re l a t i vely seve re disabilities ...m igh t be in -flu en ced by sh ort -term a n d sp orad ic serv i c e s w h ich a m ou n t on ave rage to less then on e hou r p er w eek?”Th e q u estion also su ggests th e am -b igu ity of the term “h om e ca re” as u sed -by th e p ro g ram , m ean in g either m ed ical an d/ or h o m e s e rv i c e s. Th e Sim ad p roved t o b e a n ew in ter ven tion m od ality, evalu ated in a p ilot stu d y in c o r p o ratin g gap s in p rio r kn ow l e d g e, d em on -s t ratin g th at the “p ro g ra m evalu ation m eth od s a re on ly b etter in the sen se th at th ey are b etter a d ap ted to p olicy-m ak in g situ ation s th at th ey a re attem p tin g to in flu en ce”( Cobatoff, 1996).
The fact th at on e defen d s th e im p ortan ce of ad ju stin g re s e a rch lin es to p lan n ed utilization o f re su lts d o es n o t m ea n th at th e re s e a rc h m ean s n othin g m ore th an p recise an d im m ed i-ate ap p lication . On th e con tra ry, eva l u a t i ve re-s e a rch re-sh ou ld b e e n cou r aged on an o n -goin g b a s i s, sin ce its “t i m i n g” can n ot b e d eterm i n e d b y the u rgen cy of th e decision . The m ain objec-t i ve of objec-t h e Na objec-tion al Healobjec-th Fo r u m cre aobjec-t ed recen tly by th e Can ad ian gove rn m en t with p ar -ticip ation by exp er ts from th e field is to u n d er-sta n d h ow re s e a rc h ca n h elp th e gove rn m e n t d e ve lo p c on se n su se s o n t h e m e an s t o m ain -tain the system’s efficien cy while resp ectin g l e s tem p s d e la re c h e rch e ( C h a m p a g n e, 1996). A p at h way wor th exp lo rin g wa s suggeste d by Po u vo u rville (1992), i.e., to org a n i ze a re g u l a r f l ow of in for m ation fr om th e p rod u c tion o f k n owled ge gen era ted by eva lu ation s in su ch a way as to b e available to the Mi n i s t ry of He a l t h a n d other age n cies resp on sib le for related ac-t i v i ac-t i e s.
