REVISTA
BRASILEIRA
DE
ANESTESIOLOGIA
Official Publication of the Brazilian Society of Anesthesiologywww.sba.com.br
SCIENTIFIC
ARTICLE
Anesthesiology
Journal
club
assessment
by
means
of
semantic
changes
Joaquim
Edson
Vieira
a,b,∗,
Marcelo
Luís
Abramides
Torres
a,
Regina
Albanese
Pose
b,
José
Otávio
Costa
Auler
Junior
aaDepartmentofSurgery,FaculdadedeMedicina,UniversidadedeSãoPaulo,SãoPaulo,SP,Brazil
bCenterforDevelopmentofMedicalEducationProfessorEduardoMarcondes,FaculdadedeMedicina,UniversidadedeSãoPaulo,
SãoPaulo,SP,Brazil
Received9September2013;accepted20November2013 Availableonline8January2014
KEYWORDS
Education; Medical; Graduate; Anesthesiology; Self-assessment; Programevaluation; Educational measurement; Periodicalsastopic
Abstract
Backgroundandobjectives: theinteractiveapproachofajournalclubhasbeendescribedin themedicaleducationliterature.Theaimofthisinvestigationistopresentanassessmentof journalclubasatooltoaddressthequestionwhetherresidentsreadmoreandcritically. Methods:thisstudyreportstheperformanceofmedicalresidentsinanesthesiologyfromthe ClinicsHospital---UniversityofSãoPauloMedicalSchool.Allmedicalresidentswereinvitedto answerfivequestionsderivedfromdiscussedpapers.Theanswersheetconsistedofan affir-mativestatementwithaLikerttypescale(totallydisagree---disagree---notsure---agree---totally agree),eachrelatedtooneofthechosenarticles.Theresultswereevaluatedbymeansofitem analysis---difficultyindexanddiscriminationpower.
Results:residentsfilledonehundredandseventythreeevaluationsinthemonthsofDecember 2011(n=51),July2012(n=66)andDecember2012(n=56).Thefirstexampresentedallitems withstraightstatement,secondandthirdexamspresentedmixeditems.Separating‘‘totally agree’’from‘‘agree’’increasedthedifficultyindices,butdidnotimprovethediscrimination power.
Conclusions:theuseofajournalclubassessmentwithstraightandinvertedstatementsand bymeansoffivepointsscaleforagreementhasbeenshowntoincreaseitsitemdifficultyand discriminationpower.Thismayreflectinvolvementeitherwiththereadingorthediscussion duringthejournalmeeting.
©2013SociedadeBrasileiradeAnestesiologia.PublishedbyElsevier EditoraLtda.Allrights reserved.
∗Correspondingauthor.
E-mail:[email protected](J.E.Vieira).
PALAVRAS-CHAVE
Educac¸ão; Médico; Graduac¸ão; Anestesiologia; Autoavaliac¸ão; Avaliac¸ãodo programa; Avaliac¸ão educacional;
Periódicoscomotema
Avaliac¸ãodoclubederevistadeanestesiologiapormeiodemudanc¸assemânticas
Resumo
Justificativaeobjetivos: Aabordageminterativadeumclubederevistafoidescritana liter-aturadeensinomédico.Oobjetivodestainvestigac¸ãoéapresentarumaavaliac¸ãodoclubede revistacomoumaferramentaparadiscutiraquestãodaleituracríticaentreosresidentes. Métodos: EsteestudorelataodesempenhodosmédicosresidentesdeanestesiologiadoHospital das Clínicas---Faculdade deMedicina daUniversidadede SãoPaulo. Todososmédicos resi-dentesforamconvidadosaresponderacincoperguntasderivadasdeartigosdiscutidos.Afolha derespostaconsistiaemumadeclarac¸ãoafirmativacomumaescaladotipoLikert(discordo totalmente, discordo,não tenhocerteza,concordo,concordototalmente),cadadeclarac¸ão relacionadaaumdosartigosescolhidos.Osresultadosforamavaliadospormeiodaanálisede itens---índicededificuldadeepoderdediscriminac¸ão.
Resultados: Osresidentescompletaram173avaliac¸õesnosmesesdedezembrode2011(n=51), julho de 2012 (n=66) e dezembro de 2012 (n=56). O primeiro teste apresentou todos os itenscomdeclarac¸ãodiretaeosegundoeterceirotestesapresentaramitensmistos.Separar ‘‘concordototalmente’’de‘‘concordo’’aumentouosíndicesdedificuldade,masnãomelhorou opoderdediscriminac¸ão.
Conclusões: Ousodeumaavaliac¸ãodocluberevistacomdeclarac¸õesdiretaseinversasecom ousodeumaescaladecincopontosparaacordomostrouaumentaradificuldadedoiteme opoderdediscriminac¸ão.Issopoderefletiroenvolvimentocomaleituraoucomadiscussão duranteasreuniõesdoclube.
©2013SociedadeBrasileira deAnestesiologia.PublicadoporElsevierEditoraLtda.Todosos direitosreservados.
Introduction
The interactive approach of a journal club (JC) has been described in the medical education literature. As stated before,the majorgoal for most clubs is to teachcritical appraisalskills. Inaddition,earlycharacteristicsstillwork forhighattendance:mandatoryattendance,availabilityof food,andperceivedimportancebytheprogramdirector.1In
addition,providingcreditsforattendancehasbeen associ-atedwithincreasedparticipation.2Ithasbeendescribedas
powerfulmotivatorofcriticalhouse-staffreadingbehavior,3
andindeedasameanstodevelopacurriculumin epidemi-ology,biostatisticsandexperimentaldesign.4
Journalclubshavebeenanintegralpartofhealthcare education, medical residency in general and in general surgery training and it is an educational resource that can help a residency program to meet core competency requirements.5,6 Asurveywiththe Associationof Program
Directors in Surgery showed that the typical journal club meetsonceinamonthanddiscussthreetosixarticles.It isinterestingtonotice,though,thatfewprogramsreported performinganysystematicorformalevaluation.7
Notwith-standing,thosewhodosousedobjectiveassessmentwith pretestandposttestexaminations,8measurementofJC
sat-isfactionwithastructuredchecklistforarticlereview,9and
theuseofcriticalappraisaltests.
Lately,theuseofinternetdidnotsurpasstheJCmodel withinpersonmoderationforsurgicalresidents.10
Consider-ingthesuccessfulJC,itseemsreasonabletousechecklists, make explicit the learning objectives and provide a for-malmeetingstructureandprocess,notmuchdifferentfrom thefeaturespointedoutpreviouslybyAlguire.1,11 Itisalso
noticeablethatajournalclubshouldfocusoncurrent,real
patient’sproblemofmostinteresttothegroup,12aswellas
toprovideexperiencesliketheuseofthe‘CriticalAppraisal Tool’(CAT)softwaretohelpresidentsstreamlinetheir pre-sentations leaving the majority of journal club time for discussion.13
Considering that the medical literature continues to expand and physicians must keep up withthe amount of informationavailable, theJC providesaplacewhere spe-cialists openly review with residents the literature while alsopromoteanambiencewheretoteachtoevaluatethe scientificmethodscritically.14
Theaimofthisinvestigationistopresentanassessment ofJC asa tooltoaddress thequestion whetherresidents readmoreandcritically,beyondtheJCpresentation.
Methods
Thisstudyreportstheperformanceofmedicalresidentsin anesthesiologyfromtheClinicsHospital---UniversityofSão PauloMedicalSchool(HCFMUSP),SãoPaulo,Brazil,during threeevaluationsofJournalClubactivities,December,2011 throughDecember2012.Theseassessmentsarepartoftheir formalevaluationsystemtoachieveanesthesiology profes-sionalcertificationfromHCFMUSPResidencyProgram. The studyispartofacontinuingprojectthatsurveysthequality oflifeofmedicalresidentsintheHCFMUSPandapprovedby theEthicalReviewBoardofUniversityofSãoPauloMedical School(protocol9017).
Table1 Thefirstexampresenteditems/questionswithstraightstatements.
Indices Agree+totallyagree Totallyagree
Discrimination Difficulty Discrimination Difficulty
Roquillyetal.onhydrocortisoneaftertrauma(JAMA 2011;305:1201---9)
−0.24 0.82 0.12 0.06
Fast---ASACommitteeonStandardsandPractice Parameter(Anesthesiology2011;114:495---511)
0.35 0.65 0.71 0.35
Ingrandeetal.onbodyweightandpropofol(Anesth Analg2011;113:57---62)
0.29 0.79 0.82 0.41
Kahokehretal.onropivacaineintraperitoneal(AnnSurg 2011;254:28---38)
0.65 0.44 0.24 0.12
Saaletal.onanesthesiasatisfaction(BrJAnaesth 2011;107:703---9)
0.24 0.65 0.29 0.15
IndexDifficulty:smallerthepercentage,moredifficultistheitem;IndexDiscrimination:highermeansbetterdiscrimination.
(normalObstetriccare)andtheCentralInstitutethat com-prises medical specialties surgeries like Plastic Surgery, VascularSurgery,ThoracicSurgery,HeadandNeckSurgery, Urology, Gynecology and Obstetric (pathologic), Neuro-surgery,Ear,Nose,andThroatSurgery,andOphthalmology and Emergency Surgery. This complex of institutes per-forms30,000surgeriesyearly.Besidesactivitiesinsurgical centers,residentsstay ondutydaily at thepostoperative Anesthesia Intensive Care Unit. They receive theoretical informationbasedontworegularweeklyactivities:lectures androtationalmeetings---complications,intensivecareand publishedarticlesdiscussion(journalclubs).
The journal club comprises a monthly meeting where residents previously prepare and present an article in the anesthesiology fieldrecently published, chosen by an instructor of anesthesiology (JEV). The way to choose an articlereliedmostlyontwodecisions:either anew guide-lineor anissue relatedtothepracticeofanesthesia. The majority of papers wereoriginal investigation and one or twoat everysemesterwasaguideline. Asecond resident colleagueisresponsiblefor criticizingandhighlightingthe keypointsofthepresentedinvestigation.
Afterfiveformalmeetingsandat least10articles pre-sented, all medical residents were invited to answer 5 questionsderived from some of thoseread and discussed papers.Theanswersheetconsistedofanaffirmative state-mentwithaLikerttypescale(totallydisagree---disagree---not sure---agree---totallyagree),each relatedtoone of chosen articles.An item wasassembled as a straight answer (to be agreed upon) or as an inverted one (to be disagreed upon).
Theresultswereevaluatedbymeansofitemanalysisin ordertoachievedifficultyindexanddiscriminationpower of everyitem/question. It can tellwhether theitem was tooeasyortoohardandhowwellitdiscriminatedbetween highandlow scorersonthetest. Briefly,totheitem diffi-culty,selectonethirdofexamswiththehigherscoresand thesamenumberwiththelowerscores.Sumthenumberof theseexamsandsumthenumberamongtheseexamsthat selectedtherightanswerforeachitem.Todividethesum ofrightanswersbythetotalofthoseexamswithhigherand lowerscoreswillprovidetheitemdifficulty.Smallerthis per-centage,moredifficultistheitem.Theitemdiscrimination istheresultofsubtractingthenumberofrightanswersin
thelowerscoregroupfromtherightanswersinthehigher group. This result is divided by the average of those one thirdofexamswiththehigherscoresandthesamenumber withthelowerscores.15
Results
Residentsfilledoutonehundredandseventythree evalua-tions in the months of December 2011 (n=51),July 2012 (n=66) and December 2012 (n=56), when they received theirendofsemesterassessments.Thefirstexam (Decem-ber/2011) presented all items with straight statements. Consideringseparately‘‘totally agree’’madethe discrim-inationanddifficultyindicestoimprove(onlyonebecame lessdiscriminant,item4)(Table1).
The second exam (July/2012) was presented in three forms: all items with straight statements to be chosen, two first items with straight plus two final withinverted statements, and the third form with the first two items withinvertedstatementsplustwofinalwithstraightones. The presentation of these mixed up items resulted in higherdifficultyindex,althoughhavereducedthe discrim-ination power, evenslightly. To separate ‘‘totally agree’’ from ‘‘agree’’ increased the difficulty indices, but did notimprove thediscriminationpowertoall thequestions (Table2).Thethirdexam(December/2012)waspresentedin twoforms:allitemswithstraightorallitemswithinverted statements. This presentation resulted inhigher difficulty index,andbetterdiscriminationpower.Toseparate‘‘totally agree’’from‘‘agree’’increasedthedifficultyindices,but didnotimprovethediscriminationpowertoallthequestions (Table3).
Discussion
The use of a journal club assessment with straight and inverted statementsand bymeans offive pointsscalefor agreementhasbeenshowntoincreaseitsitemdifficultyand discriminationpower. This mayreflect involvementeither withthereadingorthediscussionduringthejournal meet-ing.
Table2 Thesecondexamusedallitemsstraight,twofirstitemsstraightplustwofinalinverted,andfirsttwoitemsinverted plustwofinalstraight.
Agree+Totallyagree Totallyagree
Indices Discrimination Difficulty Discrimination Difficulty
Gaszynskietal.onTOFwithsugammadex(BritJ Anaesth2012;108(2):236)
0.50 0.63 0.88 0.44
Moonetal.onAnti-emeticeffectofondansetron(BritJ Anaesth2012;108(3):417---22)
0.50 0.50 0.50 0.25
Stumpneretal.onDesflurane-inducedpost-conditioning (BritJAnaesth2012;108(4):594---601)
0.75 0.63 0.63 0.31
Nakasujietal.onHypotensionfromspinalanesthesiain patientsmorethan80years(JClinAnesth2012;24: 201---6)
0.63 0.69 0.50 0.25
Radkeetal.onSpontaneousbreathingduringgeneral anesthesia(Anesthesiology2012;116(6):1227---34)
0.38 0.44 0.13 0.06
IndexDifficulty:Smallerthispercentage,moredifficultistheitem;IndexDiscrimination:Highermeansbetterdiscrimination.
beincorporatedintothemedicaleducationalcurriculum.16
The organization of these meetings followed previously pointedcharacteristicsassociatedwithlongandhigh atten-dance: mandatory attendance and fewer house staff, althoughtherewasnoprovisionoffood.17
Although thislearning activityhas been applied world-wide and arranged according to previously described characteristics, regular surveys usually assess goals and monitor satisfaction of attendees. As much positive and reassuringastheycanbe,suchevaluationsdidnotaddress the objectives, amongothers, of critical appraisalof the presentedliterature.18 Thepresentreportsuggeststheuse
of exams with specific questions retrieving the literature presentedandwithafive-pointagreementscaleasa strat-egytomeasureresidents’participationandtheretrievalof meaningfullearning.Thisapproachisdistinctfrommultiple choicetestsorevenwrittenessays,consideringtheinterest inpreservingauthorship---citingtheauthors’findings,and providingresponsechoicesthatshouldbeagreed-disagreed upon.
Multiplechoiceexaminationsarecomprisedofquestions whichusually,butnotalwayshavecorrectanswers.These
are snapshots at one point in time of a small subset of memory, from which evaluators attempt to make predic-tionsabouttheoverallbodyofknowledgethatastudentor residenthasachievedat thetimetheytakethe examina-tion.Theexpedientinthisinvestigationofcitingthemain contentofapublishedliterature,andeventuallyreversing itsfindings,althoughelicitingmemory,mediatesattention
asthemajorgeneral variablethroughwhich motivational factorsinfluencemeaningfullearning.19
Theexamswerealsoarrangedwithallitemspresenting straight or reversing statements, without the use of neg-atively worded items. This approach works as the stem remainsasdirectlywordeditemandtheresponseoptions remain the same but arein a different order.This strat-egycouldguardagainstmedicalresidentsacquiescenceand responsesetbehaviors, whichmeansagreeing/disagreeing without reasoning.20 The results computerized ‘‘totally
agree’’separatelyfromapreviousdegreeinaLikertscale andshowedimprovements of difficultyand discrimination indices. Taken together, the use of mixed items and a Likertscalefor agreement,thestrategiesof reading, par-ticipating inthe discussion,or both can bedifferentiated
Table3 Thethirdexampresentedeitherallitems/questionswithstraightstatementsorwithinvertedstatements.
Indices Agree+totallyagree Totallyagree
Discrimination Difficulty Discrimination Difficulty
Angstetal.onalfentanilandhereditary(Anesthesiology 2012;117:22)
0.56 0.50 0.33 0.17
vanGuliketal.onremifentanilonchronicpain(BrJ Anaesth2012;109:616)
−0.11 0.72 0.67 0.33
Aldenkorttetal.onPEEPandobesity(BrJAnaesth 2012;109:493)
0.22 0.67 0.56 0.28
Jacobetal.oncerebralmetabolism(Anesthesiology 2012;117:1062)
0.67 0.67 0.56 0.28
Walkeretal.onspinalanesthesiaonneonates(Anesth Analg2012;115)
0.67 0.67 0.33 0.17
fromonlywatchingtheJCpresentation.Thosemoreactive in JC may be prompted to choose the highest Likert degree in contrast to those who decided for a lower engagement.
In conclusion, thisinvestigation proposes toassess the retrievalofJC sessionscitingthe originalarticle, directly wordedor ina differentorder,bymeans ofaLikert scale ofagreement.Thisapproachincreaseditemdifficultyand discriminationindices.
Conflicts
of
interest
Theauthorsdeclarenoconflictsofinterest.
References
1.AlguirePC.Areviewofjournalclubsinpostgraduatemedical education.JGenInternMed.1998;13:347---53.
2.HinksonCR,KaurN,SipesMW,etal.Impactofoffering continu-ingrespiratorycareeducationcredithoursonstaffparticipation inarespiratorycarejournalclub.RespirCare.2011;56:303---5.
3.Linzer M, Brown JT, Frazier LM, et al. Impact of a medi-cal journal club on house-staff reading habits, knowledge, andcriticalappraisalskills.Arandomizedcontroltrial.JAMA. 1988;260:2537---41.
4.LetterieGS,MorgensternLS.Thejournalclubteachingcritical evaluationofclinicalliteratureinanevidence-based environ-ment.JReprodMed.2000;45:299---304.
5.HarrisJ,KearleyK,HeneghanC,etal.Arejournalclubs effec-tiveinsupportingevidence-baseddecisionmaking?Asystematic reviewBEMEGuideNo.16.MedTeach.2011;33:9---23.
6.ShiffletteV,MitchellC,MangramA,etal.Currentapproaches tojournalclubbygeneralsurgeryprogramswithinthe South-westernsurgicalcongress.JSurgEduc.2012;69:162---6.
7.Crank-PattonA,FisherJB,ToedterLJ.Theroleofthejournal clubinsurgicalresidencyprograms:asurveyofAPDSprogram directors.CurrSurg.2001;58:101---4.
8.KellyAM,CroninP.Settingup,maintainingandevaluatingan evidencebasedradiologyjournalclub:theUniversityof Michi-ganexperience.AcadRadiol.2010;17:1073---8.
9.BursteinJL,HollanderJE,BarlasD.Enhancingthevalueof jour-nalclub:useofastructuredreviewinstrument.AmJEmerg Med.1996;14:561---3.
10.McLeodRS,MacRae HM,McKenzieME,et al.EvidenceBased ReviewsinSurgerySteeringCommittee.Amoderatedjournal clubismoreeffectivethananInternetjournalclubin teach-ingcriticalappraisalskills:resultsofamulticenterrandomized controlledtrial.JAmCollSurg.2010;211:769---76.
11.Lee AG,Boldt HC, Golnik KC, et al. Usingthe JournalClub to teach and assess competence in practice-based learning andimprovement:aliteraturereviewandrecommendationfor implementation.SurvOphthalmol.2005;50:542---8.
12.MisraUK,KalitaJ,NairPP.Traditionaljournalclub:acontinuing problem.JAssocPhysiciansIndia.2007;55:343---6.
13.HartzellJD,VeerappanGR,PosleyK,etal.Residentrunjournal club:amodelbasedontheadultlearningtheory.MedTeach. 2009;31:e156---61.
14.Valentini RP, Daniels SR. The journal club. Postgrad Med J. 1997;73:81---5.
15.Gronlund NE. Constructing achievement tests prentice-hall. NewJersey:EnglewoodCliffs;1968.
16.LinzerM.Thejournalclubmedicaleducation:overonehundred yearsofunrecordedhistory.PostgradMedJ.1987;63:475---8.
17.SidorovJ.Howareinternalmedicineresidencyjournalclubs organizedand whatmakesthemsuccessful?ArchInternMed. 1995;155:1193---7.
18.DirschlDR,Tornetta3rdP,BhandariM.Designing,conducting, andevaluatingjournalclubsinorthopaedicsurgery.ClinOrthop RelatRes.2003;413:146---57.
19.Ausubel DP. The acquisition and retention of knowledge: a cognitiveview.Dordrecht,TheNetherlands:KluwerAcademic Publishers;2000.
