Validation of Two US Risk Scores for Percutaneous Coronary Intervention in a Single-Center Portuguese Population of Patients with Acute Coronary Syndrome

www.revportcardiol.org

Revista

Portuguesa

de

Cardiologia

Portuguese

Journal

of

Cardiology

ORIGINAL

ARTICLE

Validation

of

two

US

risk

scores

for

percutaneous

coronary

intervention

in

a

single-center

Portuguese

population

of

patients

with

acute

coronary

syndrome

Ana

T.

Timóteo

_,

_André

_V.

_Monteiro

_,

_Guilherme

_Portugal

_,

_Pedro

_Teixeira

_,

Helena

Aidos

_,

_Maria

_L.

_Ferreira

_,

_Rui

_C.

_Ferreira

a_{CardiologyDepartment,SantaMartaHospital,CentroHospitalardeLisboaCentral,Lisbon,Portugal} b_{Minalytics,AdvancedSolutionsforDataMiningandAnalytics,Lisbon,Portugal}

c_{InstitutodeTelecomunicac¸ões,InstitutoSuperiorTécnico,Lisbon,Portugal}

Received4May2015;accepted13September2015 Availableonline1February2016

KEYWORDS Riskstratification; Scores; Acutecoronary syndromes; Coronaryangioplasty Abstract

Introduction:NewscoreshavebeendevelopedandvalidatedintheUSforin-hospitalmortality riskstratificationinpatientsundergoingcoronaryangioplasty:theNationalCardiovascularData Registry(NCDR)riskscoreandtheMayoClinicRiskScore(MCRS).Wesoughttovalidatethese scores inaEuropean populationwithacute coronarysyndrome(ACS)andtocompare their predictiveaccuracywiththatoftheGRACEriskscore.

Methods:Inasingle-centerACSregistryofpatientsundergoingcoronaryangioplasty,weused theareaunderthereceiveroperatingcharacteristiccurve(AUC),agraphicalrepresentation of observed vs. expected mortality, andnet reclassification improvement (NRI)/integrated discriminationimprovement(IDI)analysistocomparethescores.

Results:Atotalof2148consecutivepatientswereincluded,meanage63years(SD13),74% maleand71%withST-segmentelevationACS.In-hospitalmortalitywas4.5%.TheGRACEscore showedthebestAUC(0.94,95%CI0.91---0.96)comparedwithNCDR(0.87,95%CI0.83---0.91, p=0.0003)andMCRS(0.85,95%CI0.81---0.90,p=0.0003).Inmodelcalibrationanalysis,GRACE showedthebestpredictivepower.WithGRACE,patientsweremoreoftencorrectlyclassified thanwithMCRS(NRI78.7,95%CI59.6---97.7;IDI0.136,95%CI0.073---0.199)orNCDR(NRI79.2, 95%CI60.2---98.2;IDI0.148,95%CI0.087---0.209).

Conclusion: TheNCDRandMayoClinicriskscoresareusefulforriskstratificationofin-hospital mortalityinaEuropeanpopulationofpatientswithACSundergoingcoronaryangioplasty. How-ever,theGRACEscoreisstilltobepreferred.

© 2016SociedadePortuguesade Cardiologia.Publishedby ElsevierEspaña,S.L.U.Allrights reserved.

∗_{Correspondingauthor.}

E-mailaddress:anatimoteo@yahoo.com(A.T.Timóteo).

http://dx.doi.org/10.1016/j.repc.2015.09.018

PALAVRAS-CHAVE Estratificac¸ão derisco; Scores; Síndromescoronárias agudas; Angioplastia coronária

Validac¸ãodedoisscoresderiscoamericanosparaaintervenc¸ãocoronária percutâneanumestudounicêntricodapopulac¸ãoportuguesaparadoentes comsíndromecoronáriaaguda

Resumo

Introduc¸ão:ForamvalidadosnovosscoresnosEUparaestratificac¸ãoderiscodemortalidade hospitalarem doentessubmetidos aangioplastiacoronária:daNationalCardiovascular Data Registry(NCDR)edaMayoClinic(MC).Procurámosvalidarestesscoresnumapopulac¸ãoEuropeia comSíndrome CoronáriaAguda (SCA)ecomparar asuaacuidade preditiva comoscore de GRACE.

Métodos: RegistodeSCAdeumúnicocentrodedoentessubmetidosaangioplastiacoronária. Utilizaram-seascurvasReceiverOperatingCharacteristics(ROC)eaAreaUnderCurve(AUC), amortalidadeobservadaeesperadaeaanálisedoNetReclassificationIndex(NRI)/Integrated DiscriminationImprovement(IDI).

Resultados: Foramincluídos2148doentesconsecutivos.Idademédiade63(DP13)anos,74% dosexomasculinoe71%comSCAcomelevac¸ãoST.Amortalidadehospitalarfoide4,5%.Oscore

GRACEfoioquemostroumelhorAUC(0,94,IC95%0,91---0,96)comparativamentecomoNCDR (0,87,IC95%0,83--- 0,91,p=0,0003)eoMC(0,85,IC95%0,81---0,90,p=0,0003).Naanáliseda calibrac¸ão,oGRACEmostrouomelhorpoderpreditivo.ComoscoreGRACE,osdoentesforam mais corretamenteclassificados comparativamentecom odaMayo Clinic(NRI78,7, IC 95% 59,6---97,7;IDI0,136,IC95%0,073---0,199)eNCDR(NRI79,2,IC95%60,2---98,2;IDI0,148, IC95%0,087---0,209).

Conclusão:OsscoresNCDReMCsãoúteisnaestratificac¸ãoderiscoparamortalidadehospitalar numapopulac¸ãoeuropeiadedoentescomSCAsubmetidosaangioplastiacoronária.Contudo, oscoreGRACEcontinuaaseroideal.

©2016SociedadePortuguesadeCardiologia.PublicadoporElsevierEspaña,S.L.U.Todosos direitosreservados.

Introduction

Riskstratification isessentialinthemanagementof acute coronarysyndromes(ACS),particularlyinnon-ST-elevation ACS. ACS patients are a heterogeneous group of patients

in terms of risk, which has important implications for

managementstrategies.Earlyriskstratificationistherefore essential.

The Global Registry of AcuteCoronary Events (GRACE) scoreisthemostwidelyusedriskscoreduetoitshigh pre-dictiveaccuracy.1_{TwogroupsintheUShavedevelopedmore}

recentrisk scores for stratificationof patients undergoing coronaryangioplasty,includingpatientswithACS.2,3

WesoughttovalidatetheseUSriskscoresinaEuropean populationofpatientswithACSandtocomparetheir pre-dictiveaccuracywiththatoftheclassicalGRACEriskscore.

Methods

Allconsecutiveadultpatients(aged≥18years)includedin a single-center registry on ACS between January 1, 2005 andOctober 31, 2013 wereeligible. This isa continuous, prospective and observational registry. Inclusion criteria were a history of chest pain at rest or other symptoms suggestive of an ACS within 24 hours before admission associated with new or presumed new ECG changes (sig-nificant ST-T wave changes or left bundle branch block) and/ordynamicchangesinlevelsofbiomarkersof myocar-dialnecrosis.ST-elevationmyocardialinfarction(STEMI)was

definedaspersistent (>20 minutes)ST-segmentelevation. Allothercaseswereconsiderednon-ST-elevationACS.

Datawere recordedinadatabase thatincluded demo-graphic, clinical and patient managementcharacteristics, as well as hospital outcome. Hypertension, diabetes and hyperlipidemiaweredefinedaseitherpreviouslyknownor onspecific therapy. Patientswere classifiedas smokersif theyhadsmokedduringtheprevioussixmonthsandwere self-reported.

Decisions regarding patient management strategy, includingreferralforcoronaryangiographyandmyocardial revascularization, either by percutaneous coronary inter-vention(PCI)orbycoronaryarterybypassgrafting(CABG), werelefttothediscretionoftheattendingphysician.Only patientsundergoingPCIwereeligibleforthepresentstudy. For each patient a score was retrospectively assigned according to the National Cardiovascular Data Registry (NCDR) risk score, the Mayo Clinic risk score (MCRS) and theGlobalRegistryofAcuteCoronaryEvents(GRACE) risk score.1---3

Theprimaryendpointwasall-causemortalityduringthe indexhospitalization.

Other outcomes were also recorded in the study pop-ulation: stroke/transient ischemic attack (TIA) and major bleeding. Stroke/TIA was defined as the presence of new neurological symptoms with signs of ischemia or bleeding on computed tomography or magnetic reso-nance imaging. Major bleeding was defined according to the Global Use of Strategies to Open Occluded Coro-nary Arteries (GUSTO) criteria asintracranial bleedingor

bleeding that caused hemodynamic compromise requiring intervention.4

Informed consentwasobtainedfromeachpatient.The studyprotocolconformstotheethicalguidelinesofthe1975 DeclarationofHelsinkiandwasapprovedbytheinstitution’s humanresearchcommittee.

Statisticalanalysis

Categoricalvariablesarereportedaspercentagesand con-tinuousvariablesasmeans(standarddeviation[SD]).

Concerningthevariablesincludedintheriskscores,we foundmissingdatain6.6%ofpatientsforheartrate,6.3% forsystolicbloodpressureand3.7%forcreatinineon admis-sion in the study population. Qualitative left ventricular ejection fractionwas also missingin 8.3%. A missing val-uesanalysiswasperformed,whichshowedthatthesevalues weremissingcompletely atrandom, which wasconfirmed byLittle’sMCARtest(p=0.137).Amultipleimputation tech-nique basedonaMarkov chainMonteCarloapproachwas usedtoestimate themissingvalues for each ofthe miss-ing data points in the study, including death. A primary modelincludedimputedvariables,withasecondarymodel that excluded patients with missing variables to deter-mine the consistency of the general findings with each approach.Theimputationofmissingvaluesdidnotchange themeansofthevariables.

Goodness of fit for each model was assessed with the Hosmer-Lemeshowtest.Inthistest,ahigh(non-significant) p-valueindicatesthatthemodelisperformingwellandhas a good fit. Since we aimed to compare different models usingthesamedataandpredictingthesameoutcome,we also analyzed and compared pseudo R-squared statistics. A higher pseudo R-squared value indicates which model better predicts the outcome. Discriminative ability was assessed by the area under the ROC (receiver operating characteristic) curve (AUC). The AUC provides a measure ofthemodel’sabilitytodiscriminatebetweensubjectswho experiencetheoutcomeofinterestandthosewhodonot. TocomparetheAUCofeachof thesemodels,themethod describedbyDeLongetal.wasused.5_{Calibrationwastested}

withacomparisonoftheobservedfrequenciesofpatients withthe event of interest withthe expectedfrequencies basedonthevaluesoftheestimatedprobabilitiesobtained bythelogisticregressionmodels.

Continuous net reclassification improvement (NRI) and integrated discrimination improvement (IDI) indices were also calculated. The net proportion of patients whodied (with events) withhigherprobabilities ofdeath (NRIevents) andofpatientswhodidnotdie(withoutevents)withlower probabilitiesofdeath(NRInonevents)werecalculated consid-ering eachgroup ofmodels.NRItotal isthe sumofNRIevents andNRInoneventsandquantifiesthecorrectnessofupwardand downwardreclassificationormovement ofpredicted prob-abilities asa result of usinga different risk score.6 _IDIis

a measure of the improvement in prediction and may be viewedasthedifferencebetweenimprovementinaverage sensitivityandaverage1-specificity.6

IBMSPSSStatisticssoftware(version21,Armonk,NY)was used for all statistical analyses. Model comparisons were made using the R Project for Statistical Computing open

sourceprogramming language.7---9 _{Allstatistical testswere}

two-sidedwithavalueof0.05forstatisticalsignificance.

Results

Atotalof2148patientswereincludedinthepresentstudy. Baselinecharacteristics are detailedin Table 1. Risk fac-torsand previouscardiovascular diseasedistributionwere similartootherstudiesandACSregistries.Therewasa pre-dominanceofSTEMI.Treatmentandoutcomearepresented inTable2.There were96in-hospitaldeaths (4.5%)inthe studysample.

Table1 Baselinecharacteristicsofthestudypopulation. Overall(n=2148) Age(years) 62(13) Malegender(%) 74.1 Riskfactors(%) Hypertension 62.8 Hyperlipidemia 50.0 Diabetes 23.7 Smoking 41.9 Previoushistory(%) MI 12.5 PCI 9.7 CABG 2.7 PAD 2.8 Stroke/TIA 4.6 Initialpresentation Heartrate(bpm) 78(19) SBP(mmHg) 135(28) Killipclass>1(%) 9.7 STEMI(%) 70.9 Creatinine(mg/dl) 1.0(0.6)

CABG: coronaryarterybypassgrafting;MI: myocardial infarc-tion; PAD: peripheral arterial disease; PCI: percutaneous coronaryintervention;SBP:systolicbloodpressure;STEMI: ST-elevationmyocardialinfarction;TIA:transientischemicattack.

Table2 Treatmentandcomplications.

Overall(n=2148) Treatment(%) Aspirin 99.4 Clopidogrel 97.9 ACEI/ARB 87.9 Beta-blocker 84.5 Statin 94.6 PCI 100.0 CABG 0.2 Complications(%) Stroke 1.1 Mechanicalcomplication 2.3 Majorbleeding 1.1

ACEI:angiotensin-convertingenzymeinhibitor;ARB:angiotensin receptorblocker;CABG: coronaryarterybypassgrafting; PCI: percutaneouscoronaryintervention.

1.0 1.0 1-Specificty Sensitivity 0.8 0.8 NCDR MCRS GRACE 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0

Figure1 Areaunderthecurveforthethreemodels.GRACE: GlobalRegistryofAcuteCoronaryEventsriskscore;MCRS:Mayo Clinicriskscore;NCDR:NationalCardiovascularDataRegistry riskscore.

TheMCRSshowedadequategoodnessoffit(p=0.337)and discriminative accuracy (AUC 0.85, 95% confidence inter-val[CI] 0.81---0.90) (Figure 1). Calibration was also good, exceptforpatientswithhighscores,inwhompredicted mor-talitywaslowerthan that observed(Figure2). The NCDR riskscorealsopresentedadequategoodnessoffit(p=0.474) andgooddiscrimination(AUC0.87,95%CI0.83---0.91). Cal-ibrationwasalmost perfect.Finally, theGRACE riskscore also showed adequate goodness of fit (p=0.482) and the highest discriminative ability (AUC 0.94, 95% 0.91---0.96) with excellent calibration. Comparing the GRACE risk score with the other scores, GRACE enabled significantly betterreclassificationofpatientsintheirriskclass,witha NRI>75%(Tables3and4).PseudoR-squaredstatistics con-firmedthatGRACEisthebestpredictiveriskscore:GRACE R2_=0.497;NCDRR2_=0.337;MCRSR2_=0.339.

Discussion

Risk stratification is an important step in the manage-ment of ACS patients, who are a heterogeneous group in term of characteristics and risk, particularly the sub-group with non-ST-elevation ACS. It is thus essential to

Table 3 Statistics for model improvement (comparingGRACEandMayoClinicriskscores).

ContinuousNRI(%) NRIevents 27.1(8.3---45.9) NRInonevents 51.6(48.6---54.6) NRItotal 78.7(59.6---97.7) IDIstatistics 0.136(0.073---0.199) AUC(95%CI) MCRS 0.85(0.81---0.90)

GRACEriskscore 0.94(0.91---0.96)

Difference(p) <0.001

Goodnessoffit(MCRS)a _0.337

Goodnessoffit(GRACEriskscore)a _0.482

95%confidenceintervalsareshowninbrackets.

AUC:areaunderthereceiveroperatorcharacteristiccurve;IDI: integrateddiscriminationimprovement;MCRS:MayoClinicrisk score;NRI:netreclassificationimprovement.

a _{Hosmer-Lemeshowgoodnessoffittest(p-value).}

Table 4 Statistics for model improvement (comparing

GRACE and National Cardiovascular Data Registry risk

scores). ContinuousNRI(%) NRIevents 26.0(7.3---44.8) NRInonevents 53.2(50.2---56.1) NRItotal 79.2(60.2---98.2) IDIstatistics 0.148(0.087---0.209) AUC(95%CI) NCDRriskscore 0.87(0.83---0.91)

GRACEriskscore 0.94(0.91---0.96)

Difference(p) <0.001

Goodnessoffit(NCDRriskscore)a _0.474

Goodnessoffit(GRACEriskscore)a _0.482

95%confidenceintervalsareshowninbrackets.

AUC: area under the receiver operator characteristic curve; CI:confidenceinterval;IDI:integrateddiscrimination improve-ment; NCDR: National Cardiovascular DataRegistry; NRI: net reclassificationimprovement.

a _{Hosmer-Lemeshowgoodnessoffittest(p-value).}

performearlyriskstratificationtooptimizedecision-making for appropriate management of these patients, including thebesttimeforrevascularizationandtheideal combina-tionofantithrombotictherapies.Althoughthesetreatments

0.500 0.500 0.500 0.800 0.900 0.900 0.800 0.800 0.700 0.700 0.600 0.600 0.500 0.500 0.400 0.400 0.300 0.300 0.200 0.200 0.100 0.100 0.000 0.000 0.800 0.700 0.700 0.600 0.600 0.500 0.450 0.450 0.400 0.400 0.400 0.400 0.350 0.350 0.300 0.300 0.300 0.300 0.250 0.250

Predicted Predicted Predicted

GRACE risk score NCDR risk score

Mayo Clinic risk score

Observed0.200 Observed Observed

0.200 0.200 0.200 0.150 0.150 0.100 0.100 0.100 0.100 0.050 0.050 0.000 0.000 0.000 0.000

Figure 2 Calibration of the three scores (plots of predicted vs. observed probability of death for the three scores). NCDR:NationalCardiovascularDataRegistry.

improveoutcome,theycanincreasebothcomplicationsand costs. For this reason, informed decisions are important, andforthisriskscoresarehelpfultools.

SeveralriskstratificationscoresforACShavebeen pub-lished sincethe 1990s.The first was the TIMI risk scores for STEMIandunstable angina/non-STEMI.10,11 _{These were}

developedfromcohortsofpatientsincludedinrandomized clinicaltrials(RCTs).ThelaterPURSUITriskscorewasalso based onan RCT, which can limit itsapplication anduse becausereal-worldcohortshavedifferentcharacteristics.12

The GRACE risk score1 _{was published in 2003, based}

on the GRACE multicenter ACS registry database that includedthewholespectrumofACS,reflectingareal-world cohortofpatients.Duetoitshighpredictiveaccuracyitis nowadaysthemostwidelyusedriskscoreinACS.13

Morerecently,boththeMayoClinicandtheNational Car-diovascularDataRegistrydevelopedriskscoresforpatients undergoing coronary angioplasty.2,3 _{Both scores included}

patientswithACS.AlthoughtheseUSscoresweredeveloped fromPCIregistries,thevariablesselectedweremainly clini-calinnature;noangiographicdatawasusedforriskscoring. Thus,thevariables selectedfor thesenewriskscores are very similar to those usedin the GRACE risk score. Both scoreshave beenvalidatedinUS populations,14 _{but notin}

European populations, particularlyin the context of ACS, andsowesoughttovalidatebothriskscoresandtocompare themwiththeclassicalGRACEriskscore.

Allthescoresshowedmoderatepredictiveaccuracyfor in-hospital all-causemortality. However,the GRACE score hadthebestperformance,withsignificantlybetter discrim-inativeaccuracyasshownbyROCcurveanalysis.SinceROC curveanalysisisconsidered insufficienttocompare differ-ent prediction models, it is recommended to perform a reclassification study of one model compared to another. Patientreclassification wasbetter forGRACE accordingto theirobservedrisk,thatis,patientswitheventsweremore oftencorrectlyreclassifiedtoahigherrisklevelandpatients withouteventstoalowerrisklevel.

Thereareseveralpossibleexplanationsfortheseresults. Firstly,theUS scores werenotdeveloped fromcohortsof patients with ACS but from larger registries of patients undergoingPCI.Inthesestudygroups, only25---40%of the patientshad ACS.2,3,14 _{This explains severaldifferent}

clin-ical characteristics of the patients in both US registries comparedtoourpopulationofpatientswithACS:our pop-ulationhasfewerfemales,fewerdiabeticpatientsandless previous history of myocardial infarction, PCI, peripheral arterialdiseaseandcerebrovasculardisease.Therewasalso a significant proportion of patients with STEMI, which is alsoanimportantdifferencecomparedtoothercohortsof ACS patients. In addition, the scores were developed for medium-term mortality (30-day and 1-year) and we vali-datedthemforshort-termmortality(in-hospital).

Limitations

Ourstudyhassomelimitations.Althoughourstudy popula-tionwasobtained fromaclinical registryasfor theMayo Clinic,NCDRandGRACEriskscores,theobservationaland non-randomizednatureofthestudymightlimitour conclu-sions,particularlyforotherEuropeancohortswithdifferent

characteristics.OnlypatientsundergoingPCIwereincluded inthe study cohort.Thus ourresults donotapply toACS patients who were not revascularized. Furthermore, our studypopulationissomewhatdifferentfromothercohorts, particularlyitspredominanceofSTEMIpatients(70.9%),and thus our results cannot be directly extrapolated to other cohorts.

Conclusions

InalargeEuropeancohortofpatientswithACS,theNCDR score and the MCRS are useful for in-hospital mortality riskstratificationinpatientsundergoingPCI.However,the GRACErisk score hasthe bestpredictive accuracy and as suchitshouldremainthepreferredriskstratificationscore.

Conflicts

of

interest

Theauthorshavenoconflictsofinteresttodeclare.

References

1.Granger CB, Goldberg RJ, Dabbous O, et al. Predictors of hospital mortality in the Global Registry of Acute Coronary Events.ArchInternMed.2003;163:2345---53.

2.PetersonED,DaiD,DeLongER,etal.,onbehalfoftheNCDR RegistryParticipants.Contemporary mortalityriskprediction forpercutaneouscoronaryintervention.Resultsfrom588,398 proceduresintheNationalCardiovascularDataRegistry.JAm CollCardiol.2010;55:1923---32.

3.SinghM,RihalCS,LennonRJ,etal.Bedsideestimatesofrisk frompercutaneouscoronaryinterventions:thenewMayoClinic riskscores.MayoClinProc.2007;82:701---8.

4.Sabatine MS, Morrow DA, Giugliano RP, et al. Association ofhemoglobinlevelswithclinicaloutcomesinacutecoronary syndromes.Circulation.2005;111:2042---9.

5.DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areasundertwoormorecorrelatedreceiveroperating char-acteristic curves: a non-parametric approach. Biometrics. 1988;44:837---45.

6.PencinaMJ,D’AgostinoRB,SteyerbergEW.Extensions ofnet reclassification improvement calculations tomeasure useful-nessofnewbiomarkers.StatMed.2011;30:11---21.

7.R Development Core Team. R: a language and environ-ment for statistical computing. Vienna, Austria: R Foun-dation for Statistical Computing; 2008, ISBN 3-900051-07-0 http://www.R-project.org

8.Kundu S, Aulchencko YS, van Duijn CM, et al. PredictABLE: an R packagefor the assessmentof risk prediction models. EurJEpidemiol.2011;26:261---4.

9.Robin X, Turck N, Hainard A, et al. pROC: an open-source packageforRandS+toanalyzeandcompareROCcurves.BMC Biometrics.2011;12:77.

10.AntmanEM,CohenM,BernikPJLM,etal.TheTIMIriskscore forunstableangina/non-STelevationmyocardialMI:amethod for prognostication and therapeutic decision making.JAMA. 2000;284:835---42.

11.MorrowDA,AntmanEM,CharlesworthA,etal.TIMIriskscorefor ST-elevationmyocardialinfarction:aconvenient,bedside, clin-icalscoreforriskassessmentatpresentation:anintravenous nPAfortreatmentofinfarctingmyocardiumearlyIItrial sub-study.Circulation.2000;102:2031---7.

12.Boersma E, Pieper KS, Steyerberg EW, et al., for the PURSUITInvestigators.Predictorsofoutcomeinpatientswith acutecoronarysyndromeswithoutpersistentST-segment ele-vation. Results from aninternational trialof 9461 patients. Circulation.2000;101:2557---67.

13.de Araujo Gonc¸alves P, Ferreira J, Aguiar C, et al. TIMI, PURSUIT,andGRACEriskscores:sustainedprognosticvalueand

interaction withrevascularization in NSTE-ACS. EurHeart J. 2005;26:865---72.

14.Singh M, Peterson ED, Milford-Beland S, et al. Validation of the Mayo Clinic risk score for in-hospital mortality afterpercutaneous coronaryinterventionsusingtheNational Cardiovascular Data Registry. Circ Cardiovasc Intervent. 2008;1:36---44.