A Importância da aplicação dos
Scores e do conceito do “Heart
Team” para a cirurgia cardíaca.
Mauro Paes Leme
[email protected]
F.W.Taylor
(Administração Científica)
1856 - 1915
•
é preciso estudar cientificamente o
trabalho como objeto de pesquisa séria;
•
é preciso desenhar as ferramentas e os
processos de trabalho para a dimensão
humana;
•
é preciso preparar as pessoas para usar as
ferramentas da forma mais eficiente que se
conhece.
Benchmarking
(Xerox)
•
Processo de comparação do desempenho
entre dois ou mais sistemas;
•
Método sistemático de procurar os melhores
processos e idéias inovadoras que conduzem
a um desempenho superior (Christipher E.
Bogan);
•
Curiosidade histórica: marcas (“marks”)
deixadas nos bancos de descanço(“bench”),
indicando a distância de Roma, capital do
Império.
Benchmarking
Modelos de Predição
•
processo contínuo;
•
procura fornecer informações valiosas;
•
processo de aprendizagem;
•
permite análise dos processos;
•
ferramenta de gestão, aplicável aos processos da
organização (alocação de recursos).
A experiência clínica individual e a habilidade
do profissional que emprega esses
instrumentos é indispensável para uma
correta
interpretação
dos resultados.
•
Meta-análise recente verificou-se que o EuroSCORE e o Parsonnet apresentam melhor performance em
termos de discriminação, acurácia e calibração dentre 14 modelos de predição de risco para permanência
prolongada em UTI após cirurgia cardiovascular, apesar de ambos terem sido originalmente desenvolvidos
para predizer mortalidade.
•
Apesar disso, o EuroSCORE é o mais utilizado internacionalmente, inclusive no Brasil e, de maneira geral,
tem se mostrado eficiente mesmo quando aplicado a populações não europeias,
apesar de apresentar
limitações.
Editorial
Modelos de Predição de Risco: são eles Realmente Necessários?
Risk Prediction Models: Are they Really Necessary?
Domingo M. Braile1,2,3, Rosangela Monteiro3,4, Ricardo Brandau3, Fabio B. Jatene4
Faculdade Estadual de Medicina de São José Rio Preto1, São José do Rio Preto, SP; Faculdade de Ciências Médicas Unicamp2, Campinas, SP;
Escritório Editorial da Revista Brasileira de Cirurgia Cardiovascular3, São José do Rio Preto, SP; Instituto do Coração (InCor) HC-FMUSP4, São
Paulo, SP - Brasil
Av. Juscelino K. Oliveira, 1505 - Jardim Tarraf I - 15091-450 - São José do Rio Preto, SP - Brasil
E-mail: [email protected]
Artigo recebido em 15/11/10, revisado recebido em 15/11/10, aceito em 18/11/10.
Palavras-chave
Procedimentos cirúrgicos cardíacos/mortalidade, medição de risco/métodos, prognóstico.
Os modelos de predição de risco têm ocupado cada vez mais espaço nas publicações científicas e também no dia a dia de profissionais e instituições médico-hospitalares1-9.
Criados inicialmente com o objetivo de analisar a probabilidade de complicações e óbitos de pacientes submetidos a intervenções, esses modelos de predição possibilitam realizar um balanço dos riscos e benefícios do procedimento. Apesar de nenhum sistema de predição ser suficientemente abrangente para estimar o resultado específico para cada paciente, a estratificação de risco possibilita a pacientes e médicos conhecerem o provável risco de complicações ou óbito para o grupo de indivíduos com perfil de risco similar, submetidos ao procedimento proposto, colaborando na tomada de decisões.
Além disso, esses modelos multivariados de avaliação de risco têm sido aplicados na comparação do desempenho de instituições ou mesmo de profissionais individualmente, configurando como uma forma objetiva de mensurar a qualidade dos serviços de saúde, e auxiliando na adequação da alocação de recursos. Ainda que sejam alvo de muitas críticas, os modelos de avaliação de risco, obviamente, são superiores à comparação de números absolutos, tais como taxas de mortalidade, na avaliação de desempenho de grupos ou hospitais.
Em sua maioria, os sistemas de predição desenvolvidos em cirurgia cardíaca foram desenvolvidos a partir de grandes populações de pacientes, resultantes, muitas vezes, de estudos multicêntricos. A partir desses dados são estabelecidos escores de risco baseados nos fatores identificados como preditores de óbito ou complicações.
O fato é que, desde o primeiro escore de risco a se tornar amplamente conhecido - o índice Parsonnet, na década
de 80 do século passado -, uma grande variedade desses instrumentos tem sido proposta, dentre eles o Cleveland Clinic score, French score, Pons score, Ontario Province score, o Society of Thoracic Surgery (STS) Scoring System, o EuroSCORE e o Bernstein-Parsonnet.
Embora não exista um modelo de estratificação de risco ideal, esse deveria reunir as seguintes características: facilidade de implementação, objetividade, acurácia na predição da mortalidade observada e ter uso difundido.
Em meta-análise recentemente publicada10, os autores
verificaram que os EuroSCORE e o Parsonnet apresentam melhor performance em termos de discriminação, acurácia e calibração dentre 14 modelos de predição de risco para permanência prolongada em UTI após cirurgia cardiovascular, apesar de ambos terem sido originalmente desenvolvidos para predizer mortalidade.
Diversos centros passaram a aplicar o EuroSCORE, entretanto surgiram resultados discrepantes entre a mortalidade esperada e a observada, especialmente em pacientes de alto risco. Parolari e cols11,12, dentre outros autores, apontam que
o EuroSCORE superestima a mortalidade.
Em decorrência dos avanços no cuidado perioperatório em cirurgia cardiovascular, muitos pacientes que morreriam no período em que o EuroSCORE e Parsonnet foram criados, agora sobreviveriam, mas ainda teriam alta probabilidade de desenvolver complicações. Assim, considerando-se que as técnicas cirúrgicas e de cuidado pós-operatório estão em constante evolução, bem como o perfil dos doentes, os escores devem ser dinâmicos e submetidos a constante atualização. O STS score é atualizado quase que anualmente, enquanto o EuroSCORE somente agora, mais de 10 anos após sua proposição, está sendo submetido a sua primeira revisão. Vários autores defendem que o EuroSCORE está ultrapassado e que os resultados da cirurgia melhoraram significativamente na última década, especialmente entre os idosos13-15. Apesar disso, o EuroSCORE é o mais utilizado
internacionalmente, inclusive no Brasil16-28 e, de maneira
geral, tem se mostrado eficiente mesmo quando aplicado a populações não europeias, apesar de apresentar limitações.
Dentro desse contexto, o desenvolvimento de um escore nacional, com base no perfil de risco dos pacientes submetidos à cirurgia cardíaca no Brasil e refletindo verdadeiramente a nossa prática clínica, possivelmente será o próximo passo.
Muitos grupos ainda apresentam barreiras na adoção de escores de risco na prática clínica diária, dentre outros motivos, por preferirem empregar as informações clínicas de
677
Cirurgia Cardíaca - “Scores” de Risco
•
Parsonnet Score
•
Cleveland Clinic Score (1986)
•
STS Score
(2007, atualmente 4.4 milhões de pacientes)
•
EuroScore/EuroScore II
(aditivo e logístico)
•
The Mayo Clinic Risk Score (APTC e CRM)
•
Ontario Province Risk Score
•
Bernstein-Parsonet Score
•
Ambler Score (cirurgia valvar, UK)
STS Score
2007
•
Desenvolvido para avaliar 7 tipos de
operações ou combinações de
cirurgias específicas;
•
Calcula o risco de mortalidade,
permanência hospitalar, AVC,
ventilação prolongada, mediastinite,
insuficiência renal e reoperação.
I. Introduction
The Society of Thoracic Surgeons’ risk models predict the risk of operative mortality and
morbidity after adult cardiac surgery on the basis of patient demographic and clinical
variables. The models are primarily used to adjust for case mix when comparing outcomes
across institutions with different patient populations. Such comparisons are provided in the
Database reports received by STS Database participants. The STS models are also used by
physicians and patients as tools for understanding the possible risks of surgery. As these
risks are solely statistical estimates, they should be supplemented by the professional
judgment of the patients’ healthcare provider, particularly their cardiac surgeon.
This overview is provided as background to help users of the online STS risk calculator
understand and interpret the results. Throughout this document, variable short names are
used frequently. Detailed information on the STS variables, including variable short names
and clinical definitions can be found at the STS website -
http://www.sts.org
under the STS
National Database tab, Data Managers Section. Brief definitions are also available by
clicking the “definitions” link on the risk calculator web page.
II. Surgical Procedures
The STS currently has three risk models: CABG, Valve, and Valve+CABG. The models
apply to seven specific surgical procedure classifications:
CABG model
1. Isolated Coronary Artery Bypass
(CABG Only)
Valve model
2. Isolated Aortic Valve Replacement
(AV Replace)
3. Isolated Mitral Valve Replacement
(MV Replace)
4. Isolated Mitral Valve Repair
(MV Repair)
Valve+CABG model
5. Aortic Valve Replacement + CABG
(AV Replace + CABG)
6. Mitral Valve Replacement + CABG
(MV Replace + CABG)
7. Mitral Valve Repair + CABG
(MV Repair + CABG)
See Table 3 below for detailed definitions of these procedure classifications.
NOTE: A predicted risk value will NOT be calculated for any procedure that does not fall into
one of these precisely defined categories.
STS Adult Cardiac Surgery Database Risk Model Variables – Data Version 2.61
STS Score
Separado em 3 grandes coortes com mais de
100 mil pacientes cada
I. Introduction
The Society of Thoracic Surgeons’ risk models predict the risk of operative mortality and
morbidity after adult cardiac surgery on the basis of patient demographic and clinical
variables. The models are primarily used to adjust for case mix when comparing outcomes
across institutions with different patient populations. Such comparisons are provided in the
Database reports received by STS Database participants. The STS models are also used by
physicians and patients as tools for understanding the possible risks of surgery. As these
risks are solely statistical estimates, they should be supplemented by the professional
judgment of the patients’ healthcare provider, particularly their cardiac surgeon.
This overview is provided as background to help users of the online STS risk calculator
understand and interpret the results. Throughout this document, variable short names are
used frequently. Detailed information on the STS variables, including variable short names
and clinical definitions can be found at the STS website -
http://www.sts.org
under the STS
National Database tab, Data Managers Section. Brief definitions are also available by
clicking the “definitions” link on the risk calculator web page.
II. Surgical Procedures
The STS currently has three risk models: CABG, Valve, and Valve+CABG. The models
apply to seven specific surgical procedure classifications:
CABG model
1. Isolated Coronary Artery Bypass
(CABG Only)
Valve model
2. Isolated Aortic Valve Replacement
(AV Replace)
3. Isolated Mitral Valve Replacement
(MV Replace)
4. Isolated Mitral Valve Repair
(MV Repair)
Valve+CABG model
5. Aortic Valve Replacement + CABG
(AV Replace + CABG)
6. Mitral Valve Replacement + CABG
(MV Replace + CABG)
7. Mitral Valve Repair + CABG
(MV Repair + CABG)
See Table 3 below for detailed definitions of these procedure classifications.
NOTE: A predicted risk value will NOT be calculated for any procedure that does not fall into
one of these precisely defined categories.
STS Adult Cardiac Surgery Database Risk Model Variables – Data Version 2.61
III. About the Current Models
The current models were developed during the fall of 2007 using STS Adult Cardiac Surgery
Database records for surgical procedures taking place between January 1, 2002 –
December 31, 2006. Risk models were developed for the nine endpoints defined in Table 1:
Table 1. Definition of STS Risk Model Outcomes
Endpoint
Description
Operative Mortality
STS v2.61 Sequence number 3050 (MtOpD):
Operative mortality includes both (1) all deaths occurring during the hospitalization
in which the operation was performed, even if after 30 days; and (2) those deaths
occurring after discharge from the hospital, but within 30 days of the procedure
unless the cause of death is clearly unrelated to the operation.
Permanent Stroke
STS v2.61 Sequence number 2830 (CNStrokP):
Postoperative stroke (i.e., any confirmed neurological deficit of abrupt onset caused
by a disturbance in cerebral blood supply) that did not resolve within 24 hours.
Renal Failure
STS v2.61 Sequence number 2890 (CRenFail):
Acute or worsening renal failure resulting in one or more of the following:
1. Increase of serum creatinine to > 2.0, and 2x most recent preoperative creatinine
level.
2. A new requirement for dialysis postoperatively.
Prolonged
Ventilation
> 24 hours
STS v2.61 Sequence number 2860 (CPVntLng):
Prolonged pulmonary ventilator > 24 hours.
Include (but not limited to) causes such as ARDS, pulmonary edema, and/or any
patient requiring mechanical ventilation > 24 hours postoperatively.
Deep Sternal
Wound Infection
STS v2.61 Sequence number 2780 (CIStDeep):
Deep sternal infection, within 30 days postoperatively, involving muscle, bone,
and/or mediastinum REQUIRING OPERATIVE INTERVENTION.
Must have ALL of the following conditions:
1. Wound opened with excision of tissue (I&D) or re-exploration of mediastinum
2. Positive culture
3. Treatment with antibiotics.
Reoperation
for any reason
STS v2.61 Sequence numbers 2720 (CopReBld), 2730 (COpReVlv), 2740
(COpReGft), 2750 (COpReOth), 2760 (COpReNon):
Reoperation for bleeding/tamponade, valvular dysfunction, graft occlusion, other
cardiac reason, or non-cardiac reason
Major Morbidity or
Operative Mortality
A composite endpoint defined as any of the outcomes listed in the first six rows of
this table.
Short Stay:
PLOS < 6 days *
Discharged alive and within 5 days of surgery
Long Stay:
PLOS >14 days
Failure to be discharged within 14 days of surgery
*NOTE: The definition of the short patient length-of-stay endpoint differs from previous
versions of the STS risk model. In the new definition, patients must be discharged alive in
order to receive credit for a PLOS < 6 days.
STS Adult Cardiac Surgery Database Risk Model Variables – Data Version 2.61
Os modelos preditivos
do STS Score foram
desenvolvidos em 2007
contemplando os 9
“endpoints” ou
desfechos (18-110
anos).
I. Introduction
The Society of Thoracic Surgeons’ risk models predict the risk of operative mortality and
morbidity after adult cardiac surgery on the basis of patient demographic and clinical
variables. The models are primarily used to adjust for case mix when comparing outcomes
across institutions with different patient populations. Such comparisons are provided in the
Database reports received by STS Database participants. The STS models are also used by
physicians and patients as tools for understanding the possible risks of surgery. As these
risks are solely statistical estimates, they should be supplemented by the professional
judgment of the patients’ healthcare provider, particularly their cardiac surgeon.
This overview is provided as background to help users of the online STS risk calculator
understand and interpret the results. Throughout this document, variable short names are
used frequently. Detailed information on the STS variables, including variable short names
and clinical definitions can be found at the STS website -
http://www.sts.org
under the STS
National Database tab, Data Managers Section. Brief definitions are also available by
clicking the “definitions” link on the risk calculator web page.
II. Surgical Procedures
The STS currently has three risk models: CABG, Valve, and Valve+CABG. The models
apply to seven specific surgical procedure classifications:
CABG model
1. Isolated Coronary Artery Bypass
(CABG Only)
Valve model
2. Isolated Aortic Valve Replacement
(AV Replace)
3. Isolated Mitral Valve Replacement
(MV Replace)
4. Isolated Mitral Valve Repair
(MV Repair)
Valve+CABG model
5. Aortic Valve Replacement + CABG
(AV Replace + CABG)
6. Mitral Valve Replacement + CABG
(MV Replace + CABG)
7. Mitral Valve Repair + CABG
(MV Repair + CABG)
See Table 3 below for detailed definitions of these procedure classifications.
NOTE: A predicted risk value will NOT be calculated for any procedure that does not fall into
one of these precisely defined categories.
STS Adult Cardiac Surgery Database Risk Model Variables – Data Version 2.61
STS adult cardiac
database,
version 2.73. July 1, 2011
•
Doença hepática
•
Irradiação prévia
•
Necessidade de oxigênio
•
Aorta em porcelana
•
Fragilidade “Frailty”
Hipoalbuminemia e estado
nutricional ?
Malignidade ?
sábado, 18 de agosto de 12Resultados em cirurgia cardiovascular: oportunidade
para rediscutir o atendimento médico e cardiológico no
sistema público de saúde do país.
“Mortality related to cardiac surgery in Brazil, 2000-2003 (JTCVS 2008)”.
Editorial, RBCCV 2008
•
interpretação dos dados (70% das cirurgias, DATASUS);
•
traduz a competência das instituições e do sistema;
•
a utilização dos Scores de risco possibilita a correção dos
resultados de gravidade,
se adequadamente
calibrados.
•
Brasil - 350 cir/milhão hab/ano - $ 290 (7,6% PIB)
•
Europa - 900 cir/milhão hab/ano
•
EUA - 2000 cir/milhão hab/ano - $2.725 (15,2%, PIB)
•
Argentina - $ 380 (8,9% PIB)
•
Resultados precisam ser interpretados além da competência individual:
suporte organizacional, estado evolutivo da doença, diagnóstico
acurado, cuidados intensivos pré e pós-operatórios, treinamento em
todas as áreas.
Resultados em cirurgia cardiovascular: oportunidade
para rediscutir o atendimento médico e cardiológico no
sistema público de saúde do país.
Walter J. GOMES, José Teles de MENDONÇA, Domingo M. BRAILE
RBCCV 2007
•
Foram coletados dados de 19 mil pacientes adultos operados
consecutivamente em 128 centros de 8 países europeus;
•
Analisados 68 fatores de risco pré-op e 29 fatores operatórios;
•
A relação entre os diversos fatores de risco - análise univariada e
regressão logística. Identificados 17 fatores de risco “reais” (foram
atribuidos “Scores”), classificando em três grupos de risco: baixo
(Euroscore 0-2), médio (Euroscore 3-5) e alto (Euroscore ≥6);
•
Observado a definição de cada fator de risco;
•
Calibração mensurada comparando a mortalidade observada com a
esperada (teste de bondade de ajuste de Homer-Lemeshow);
•
Acurácia, capacidade do modelo discriminar os pacientes que foram a
óbito dos sobreviventes, foi avaliada através da estatística-C ou propoção
das previsões concordantes (0,5 a 1,0), também denominada de área sob
a curva ROC.
O EuroSCORE como preditor de mortalidade em
cirurgia cardíaca. A construção do método de
predição.
Editorial
The EuroSCORE — 10 years later. Time to change?
Keywords: Aortic valve replacement; Risk-score; Euroscore
In this issue of the journal, Di Giammarco et al., from
Chieti, Italy
[1]
, report their evaluation of the performance
of the EuroSCORE calculator in the prediction of the 30-day
outcome after isolated aortic valve replacement (AVR), ‘in
order to assess its absolute reliability and usefulness as
selection criteria to percutaneous aortic valve implantation
(PAVI).’ With this aim, they carried out a retrospective
statistical analysis on 379 of their patients consecutively
submitted to isolated AVR in the past 10 years of surgical
activity. Their observed mortality was 5.2%, significantly
lower than the 9.4% expected mortality by the logistic
EuroSCORE. The discrepancy was particularly significant in
the latter 5-year period. They conclude that the ‘EuroSCORE
appears to be a not validable model in absolute and relative
risk prediction for isolated AVR. On this basis its use in
selecting candidates to PAVI should be carefully weighted.’
The emergence of PAVI has resulted in an unprecedented use
of the EuroSCORE to predict the results of conventional aortic
valve replacement, especially in the high-risk groups, which
would, thus, become the candidates for PAVI. Large series with
mean logistic scores above 20% suddenly appeared, where they
were not suspected before. Most surgeons had not seen so many
patients considered too high a risk for surgery. I certainly had
not. This, in turn, triggered many retrospective analyses of
series with patients who were recently submitted to AVR,
which, invariably, found that the EuroSCORE overestimates the
risk of isolated AVR
[2,3]
. This is the main message of the article
by Di Giammarco et al., but only this year there have been
another half a dozen papers with identical conclusions.
The group from Bad Oeynhausen, Germany, is about to
publish one based on 2757 patients submitted to AVR recently
in their centre
[4]
. Similarly, concluding that ‘patient
selection for interventional AVR cannot be based on the
EuroSCORE, because it lacks discrimination and
centre-specific calibration.’ These authors thus recommend
‘indi-vidual, surgical judgement that weighs institutional
exper-tise of high risk patients against possible reduction of
mortality by using interventional techniques.’ Although not
usually recognised, we all know that expertise, hence the
results thereof, vary from group to group and, within these,
from surgeon to surgeon. No risk score can cover all these
differences. Similarly, an analysis of a subgroup of 6305
patients submitted to isolated AVR registry of the German
Society of Thoracic and Cardiovascular Surgery from 2006 and
2007 revealed an overall hospital mortality of 3.9% whereas
the logistic EuroSCORE predicted 7.3%, which supports ‘the
substantial lack of predictive value of the EuroSCORE
[5]
.’
Some other articles also recently published were directed at
validating the EuroSCORE in parts of the world other than
Europe and their conclusions are that the EuroSCORE does not
apply there
[6—9]
. However, I believe, it also does not apply in
Europe. For three main reasons: firstly, the EuroSCORE is
already outdated, as it was developed from data on patients
operated on almost a decade and a half ago, and the results of
surgery have improved significantly since, especially in the
elderly. Secondly, because the data originated from only eight
European countries and, from each one of these, only few
centres contributed. As with any type of statistical analysis, it
generated mean values which may even not serve all these
centres. As stated recently
[10]
, ‘the logistic EuroSCORE risk
stratification system was developed and validated within the
European population. There should be caution in the utilisation
of any particular risk stratification system outside the countries
of origins, and it is important to carefully evaluate the validity
of such system amongst foreign population,’ which means that
it may even not be applicable to all European countries.
Thirdly, and most important, the EuroSCORE was especially
developed for cardiac surgery in general, especially for
coronary re-vascularisation procedures, the majority of data
belonging to this group of patients, and not specifically for AVR.
Hence, it is now generally recognised that the EuroSCORE
does not reflect current results of cardiac surgery, including
coronary re-vascularisation, but especially AVR. Curiously, the
additive model of the EuroSCORE, initially published in 1999
[11]
, was modified in 2003 by the introduction of the logistic
model
[12]
, which is now the most widely used, but whose
values are even higher than those of the additive score.
Furthermore, it is important to stress that the model would not
apply, in any case, in any situation, before validation, to each
particular surgical group. We, in Coimbra, have recognised
that and have, recently, developed our own scores for
prediction of the risks of mortality and morbidity after
coronary surgery in our population and are about to do the
same for valve surgery. Others have gone in a similar path.
www.elsevier.com/locate/ejcts
European Journal of Cardio-thoracic Surgery 37 (2010) 253—254
1010-7940/$ — see front matter # 2009 European Association for Cardio-Thoracic Surgery. Published by Elsevier B.V. All rights reserved.
doi:
10.1016/j.ejcts.2009.08.017
‘EuroSCORE appears to be a not validable model in absolute and relative
risk prediction for isolated AVR. On this basis its use in selecting
candidates to PAVI should be carefully weighted.’
.
The group from Bad Oeynhausen, Germany, is about to publish one study
based on 2757 patients subm Similarly, concluding that ‘patient selection
for interventional AVR cannot be based on the EuroSCORE, because it
lacks discrimination and centre- specific calibration.’
The additive model of the EuroSCORE, initially published in 1999, was
modified in 2003 by the introduction of the logistic model, which is now
the most widely used, but whose values are even higher than those of
the additive score.Furthermore,
it is important to stress that the model
would not apply, in any case, in any situation, before validation, to each
particular surgical group.
Manuel J. Antunes*
Cardiothoracic Surgery, University Hospitals,
Coimbra.
Portugal
•
Controle de qualidade em cirurgia é necessário para que se obtenha
de uma operação planejada o resultado esperado, em concordância
com o conhecimento médico atual.
•
O primeiro passo para o controle de qualidade em cirurgia é a
configuração de uma base de dados.
•
Os hospitais precisam ter os dados dos procedimentos nele
realizados, para promover melhora dos resultados cirúrgicos, corrigir
defeitos técnicos e estruturais e diminuir gastos desnecessários.
•
As sociedades médicas são guardiãs das especialidades e por meio de
análise de dados podem localizar erros e corrigi-los.
•
O controle de qualidade em cirurgia cardíaca depende da equipe
cirúrgica, da unidade hospitalar e do sistema de saúde.
Controle de qualidade em cirurgia
cardiovascular: um paradigma a ser atingido.
Quality control in cardiovascular surgery: a new paradigm
Rev Bras Cir Cardiovasc, 2007
•
Operative risk was calculated using the STS Predicted Risk of Mortality, the EuroSCORE logistic and
additive algorithms, and the Ambler Risk Score.
•
CONCLUSION:
The STS Predicted Risk of Mortality most accurately predicted peri- operative and
long-term mortality for the highest risk patients having aortic valve replacement.
Reliability of risk algorithms in predicting early and late operative outcomes in high-risk
patients undergoing aortic valve replacement.
The Journal of Thoracic and Cardiovascular Surgery 2008Objective: Risk algorithms were used to identify a high-risk population for trans- catheter aortic valve
implantation instead of standard aortic valve replacement in patients with aortic stenosis. We evaluated
the efficacy of these methods for predicting outcomes in high-risk patients undergoing aortic valve
replacement.
Mean predicted operative mortality for the STS-PROM
high-risk group was 13.31% compared with 50.87%,
14.04%, and 19.03% for the LES, AES, and ARS high-risk
cohorts, respectively (
Table 3
).
Measured operative mortality was 18.75% (12/64) in the
STS-PROM high-risk cohort, 15.63% (10/64) and 11.90%
(10/84) in the LES and AES groups, respectively, and
13.40% (13/97) in the ARS group. The observed/expected
ratios calculated for perioperative mortality were 0.31, 0.85,
and 0.71 in the LES, AES, and ARS cohorts, respectively.
The STS-PROM group had an observed/expected mortality
ratio of 1.41. Longer-term follow-up showed that 45.3%
(29/64) of the patients identified by the STS-PROM as
being at high risk died within the follow-up period. Within
the LES and AES groups, 29.7% (19/64) and 33.3% (28/
84), respectively, of the patients identified as being at high
risk died during the follow-up period. Only 26.8% (26/97)
of the ARS patients were noted as having died. The
con-TABLE 2. Preoperative demographics of high-risk patients identified by various risk algorithms
STS-PROM (64) % LES (64) % AES (84) % ARS (97) % P value
Age (y)
79.5 ! 8.9
79.3 ! 6.3
78.6 ! 7.4
77.3 ! 7.7
.22
Ejection fraction
45.7 ! 14.6
43.0 ! 14.1
43.7 ! 14.2
41.1 ! 14.8
.31
Female gender
54.7
45.3
47.6
46.4
.70
Diabetes
42.2
26.6
31.0
41.2
.13
Renal failure
26.6
20.3
19.1
10.3
.062
Renal failure requiring dialysis
13.2
8.8
9.8
7.3
.85
Cerebrovascular accident
22.2
38.1
28.9
13.5
.004
Chronic lung disease
23.1
12.5
15.8
7.7
.82
Peripheral vascular disease
34.4
37.5
35.7
15.5
.004
Cerebrovascular disease
29.7
40.6
38.1
18.6
.008
Previous CABG
44.4
57.1
51.8
41.7
.22
Myocardial infarction
18.8
25.0
23.8
16.5
.49
Congestive heart failure
75.0
67.2
64.3
65.0
.51
Cardiogenic shock
6.3
4.7
3.6
3.1
.78
Arrhythmia
35.9
32.8
32.1
42.3
.48
NYHA class (III-IV)
71.9
60.9
63.1
64.6
.33
Status
Elective
31.3
53.1
56.0
43.3
.075
Emergency
1.6
1.6
1.2
3.1
Urgent
67.2
45.3
42.9
53.6
Intra-aortic balloon pump
14.1
17.2
16.7
11.5
.70
STS-PROM, The Society of Thoracic Surgeons Predicted Risk of Mortality; LES, EuroSCORE logistic; AES, EuroSCORE additive; ARS, Ambler Risk Score; CABG, coronary artery bypass grafting; NYHA, New York Heart Association.
TABLE 3. Risk calculations and mortality rates for four risk algorithms
All patients (638) STS-PROM LES AES ARS
Overall mean/minimum–maximum
4.26% 0.91–46.82
13.21% 1.51–93.32
7.46% 2–22
6.99% 0.4–39.7
High-risk groups
STS-PROM (n " 64)
LES (n " 64)
AES (n " 84)
ARS (n " 97)
90th percentile risk break point
8.38%
33.47%
12%
14.3%
Expected (predicted) mortality in top 10%
(mean/ minimum–maximum)
13.31% 8.38%–46.8% 50.87% 33.47%–93.32% 14.04% 12.0%–22.0% 19.03% 14.3%–39.7%
Observed perioperative mortality
18.75% (12/64)
15.63% (10/64)
11.90% (10/84)
13.40% (13/97)
Observed/expected (perioperative)
mortality
1.41
0.31
0.85
0.71
Observed late mortality
45.3% (29/64)
29.7% (19/64)
33.3% (28/84)
26.8% (26/97)
Observed overall mortality
64.06% (41/64)
45.31% (29/64)
45.24% (38/84)
40.21% (39/97)
Kaplan–Meier survival estimate at 5 y
38.5% # 6.7%
53.7% # 7.2%
49.8% # 6.6%
55.1% # 6.2%
Overall mortality odds ratio,
8.06
3.27
3.45
2.76
High/low risk group
(4.63–14.01)
(1.92–5.57)
(2.14–5.57)
(1.75–4.36)
STS-PROM, The Society of Thoracic Surgeons Predicted Risk of Mortality; LES, EuroSCORE logistic; AES, EuroSCORE additive; ARS, Ambler Risk Score; CABG, coronary artery bypass grafting; NYHA, New York Heart Association.
Dewey et al
Evolving Technology
The Journal of Thoracic and Cardiovascular Surgery
●Volume 135, Number 1 183
ET
•
Variáveis disponíveis, entretanto
ignoradas, posteriormente observou-se
estarem associadas à mortalidade.
•
Variáveis raras em alguns subgrupos de
pacientes.
•
Variáveis que representam subgrupos
totalmente ausentes do conjunto de
pacientes sobre os quais os modelos
foram desenvolvidos.
Evolving
Technology
Reliability of risk algorithms in predicting early and late
operative outcomes in high-risk patients undergoing aortic
valve replacement
Todd M. Dewey, MDa, David Brown, MDa, William H. Ryan, MDa, Morley A. Herbert, PhDb, Syma L. Prince, RNa,
and Michael J. Mack, MDa
Objective: Risk algorithms were used to identify a high-risk population for
trans-catheter aortic valve implantation instead of standard aortic valve replacement in patients with aortic stenosis. We evaluated the efficacy of these methods for predicting outcomes in high-risk patients undergoing aortic valve replacement.
Methods: Data were collected on 638 patients identified as having isolated aortic
valve replacement between January 1, 1998 and December 31, 2006, using The Society of Thoracic Surgeons (STS) database. Long-term survival was determined from the Social Security Death Index or family contact. Operative risk was calcu-lated using the STS Predicted Risk of Mortality, the EuroSCORE logistic and additive algorithms, and the Ambler Risk Score. Patients at or above the 90th percentile of risk (8.38% for STS, 33.47% for logistic, 12% for additive, 14.3% for Ambler) were identified as high risk. We then compared actual with predicted mortality and each algorithm’s ability to identify patients with the worst long-term survival.
Results: Operative mortality was 24 of 638 (3.76%). An additional 121 (19.0%)
patients died during the follow-up study period (mean 4.2 ! 2.7 years). Overall mortality was 145 of 638 (22.7%). Expected versus observed mortality for the high-risk group by algorithm was 13.3% versus 18.8% for STS, 50.9% versus 15.6% for logistic, 14.0% versus 11.9% for additive, and 19.0% versus 13.4% by Ambler. Long-term mortality, per high-risk group, was 64.1% in the STS Predicted Risk of Mortality, 45.3% in the logistic, 45.2% in the additive, and 40.2% in Ambler Risk Score. Logistic regression showed that the STS algorithm was the most sensitive in defining the patients most at risk for long-term mortality.
Conclusion: The STS Predicted Risk of Mortality most accurately predicted
peri-operative and long-term mortality for the highest risk patients having aortic valve replacement.
A
ortic stenosis (AS) is the most common valvular abnormality encountered in the United States, with a reported incidence of approximately 5 of every 10,000 adults.1The prevalence of AS is expected to markedly increase asthe US population ages, with aortic valve calcification and stenosis affecting the health of larger numbers of patients.2-4
From the Cardiopulmonary Research Sci-ence and Technology Institute (CRSTI)a
and the Medical City Dallas Hospitalb,
Dal-las, Tex.
Read at the Eighty-seventh Annual Meeting of The American Association for Thoracic Surgery, Washington, DC, May 5-9, 2007. Received for publication May 4, 2007; re-visions received Aug 29, 2007; accepted for publication Sept 12, 2007.
Address for reprints: Todd M. Dewey, MD, 7777 Forest Ln, Suite A323, Dallas, TX 75230 (E-mail: [email protected]). J Thorac Cardiovasc Surg 2008;135:180-7 0022-5223/$34.00
Copyright © 2008 by The American Asso-ciation for Thoracic Surgery
doi:10.1016/j.jtcvs.2007.09.011
180 The Journal of Thoracic and Cardiovascular Surgery●January 2008
ET
Evolving
Technology
Reliability of risk algorithms in predicting early and late
operative outcomes in high-risk patients undergoing aortic
valve replacement
Todd M. Dewey, MDa, David Brown, MDa, William H. Ryan, MDa, Morley A. Herbert, PhDb, Syma L. Prince, RNa,
and Michael J. Mack, MDa
Objective: Risk algorithms were used to identify a high-risk population for
trans-catheter aortic valve implantation instead of standard aortic valve replacement in patients with aortic stenosis. We evaluated the efficacy of these methods for predicting outcomes in high-risk patients undergoing aortic valve replacement.
Methods: Data were collected on 638 patients identified as having isolated aortic
valve replacement between January 1, 1998 and December 31, 2006, using The Society of Thoracic Surgeons (STS) database. Long-term survival was determined from the Social Security Death Index or family contact. Operative risk was calcu-lated using the STS Predicted Risk of Mortality, the EuroSCORE logistic and additive algorithms, and the Ambler Risk Score. Patients at or above the 90th percentile of risk (8.38% for STS, 33.47% for logistic, 12% for additive, 14.3% for Ambler) were identified as high risk. We then compared actual with predicted mortality and each algorithm’s ability to identify patients with the worst long-term survival.
Results: Operative mortality was 24 of 638 (3.76%). An additional 121 (19.0%)
patients died during the follow-up study period (mean 4.2 ! 2.7 years). Overall mortality was 145 of 638 (22.7%). Expected versus observed mortality for the high-risk group by algorithm was 13.3% versus 18.8% for STS, 50.9% versus 15.6% for logistic, 14.0% versus 11.9% for additive, and 19.0% versus 13.4% by Ambler. Long-term mortality, per high-risk group, was 64.1% in the STS Predicted Risk of Mortality, 45.3% in the logistic, 45.2% in the additive, and 40.2% in Ambler Risk Score. Logistic regression showed that the STS algorithm was the most sensitive in defining the patients most at risk for long-term mortality.
Conclusion: The STS Predicted Risk of Mortality most accurately predicted
peri-operative and long-term mortality for the highest risk patients having aortic valve replacement.
A
ortic stenosis (AS) is the most common valvular abnormality encountered in the United States, with a reported incidence of approximately 5 of every 10,000 adults.1The prevalence of AS is expected to markedly increase asthe US population ages, with aortic valve calcification and stenosis affecting the health of larger numbers of patients.2-4
From the Cardiopulmonary Research Sci-ence and Technology Institute (CRSTI)a
and the Medical City Dallas Hospitalb,
Dal-las, Tex.
Read at the Eighty-seventh Annual Meeting of The American Association for Thoracic Surgery, Washington, DC, May 5-9, 2007. Received for publication May 4, 2007; re-visions received Aug 29, 2007; accepted for publication Sept 12, 2007.
Address for reprints: Todd M. Dewey, MD, 7777 Forest Ln, Suite A323, Dallas, TX 75230 (E-mail: [email protected]). J Thorac Cardiovasc Surg 2008;135:180-7 0022-5223/$34.00
Copyright © 2008 by The American Asso-ciation for Thoracic Surgery
doi:10.1016/j.jtcvs.2007.09.011
180 The Journal of Thoracic and Cardiovascular Surgery●January 2008
ET
EuroSCORE II, illum qui est gravitates magni observe*
Paul Sergeant*, Bart Meuris and Matteo Pettinari
Department of Cardiac Surgery, Gasthuisberg University Hospital, Leuven, Belgium
* Corresponding author. Department of Cardiac Surgery, Gasthuisberg University Hospital, Herestreet 40, 3000 Leuven, Belgium. Tel: 344219; fax: +32-16-344616; e-mail: [email protected] (P. Sergeant).
Keywords: Risk assessment • EuroSCORE • Cardiac surgery • Mortality
*‘Illum qui est gravitates magni observe!’ (‘Pay careful attention to that which is of great importance!’)
The authors of the EuroSCORE I project have submitted a follow-up project called EuroSCORE II [1]. In this project, they intend to improve the discrimination and calibration of their first mathem-atical model (additive and logistic versions).
THE EUROSCORE I
A Google search for EuroSCORE identifies at least 108 000 refer-ences and more than 1300 formal citations. The first versions have indeed been used and misused without an in-depth under-standing of their limitations.
EuroSCORE I has been used as a quality monitoring or com-parison tool. The quality of care is largely dissociated with the early risk. Indeed, the quality of care in cardiac surgery proce-dures has to use extended observation intervals years beyond the observation interval of the EuroSCORE models. In addition, the quality of care involves a whole series of criteria as there are appropriate diagnostic systems, waiting times, early risks, late benefits, resources used …
EuroSCORE I has been used for differential therapy or informed consent forms to express the early risk to patient and society. This use was similarly inappropriate since the risk inter-val should be based on the observation of the hazard function and will vary for each pathology, each event, major variability in the procedure and in post-procedural approaches. EuroSCORE I therefore needed to use an observation interval including the observation intervals mandatory for all major pathologies and procedural approaches, included in its reference database. It failed to do so and therefore always presented an incorrect and incomplete depiction of this early risk. The dramatic misuse of the EuroSCORE for the TAVI (Transcatheter Aortic Valve Implantation) market expansion was also a mistake. There has never been any information about the density at this high-end of the risk spectrum of the original reference database, and even less information about discriminatory and calibration power within this zone of risk.
A similar misuse has been the application of EuroSCORE I for early non-lethal events as renal failure or length of stay. Of course, there were some predictive values and some ROC values in amalgamating the more usual risk factors (even using
an irrelevant coefficient), but this misuse rejected the complete notion of outcome analysis, its rules and limitations.
ANALYTIC PROCESS
Early risk is a rare event and there were a number of possible strategies [2] available for the development of EuroSCORE II. They are listed in Table 1. The authors have decided to repeat the previously used statistical forecasting and have added some judgment adjustment. The essence of the method used is the basis of the knowledge, often called the reference class (the database).
The science of risk prediction forces us to focus on three lim-itations of this approach and evaluate this manuscript versus those criteria.
The sparsity in the reference class improves with the actuality, the size and the richness of the database, most of all in the density of the extremes of variability. In the presence of a spars-ity in the reference class, statistical forecasting becomes unreli-able. This unreliability is slightly improved through judgment adjustment.
Secondly, the inappropriateness of the reference class makes the statistical forecasting unreliable. An inappropriate reference does not include the essential variability or the appropriate outcome.
Thirdly, the inappropriate statistical model (e.g. oversimplified models, poor calibration etc.) destabilizes completely statistical forecasting. There is mixed evidence about judgement adjusting the model. The accuracy of a mathematical model evaluates cali-bration, discrimination and a combination of both. The calibra-tion describes how well the predicted probabilities agree with the actual observed risk. The Hosmer–Lemeshow statistic com-pares proportions but is most certainly imperfect. A non-significant Hosmer–Lemeshow test means that there is no evi-dence of bad calibration, it does not mean that there is good calibration. The discrimination describes how well a model sepa-rates black from white. The methods used are the sensitivity (true positive), the specificity (true negative), the positive predict-ive value (PPV), the negatpredict-ive predictpredict-ive value (NPV), the misclas-sified, the ROC [3] (receiving operating characteristics) and the C-statistic (concordance index). The combination of both calibra-tion and discriminacalibra-tion is best described using the likelihood sta-tistics, the R2[4] and Brier [5] scores.
© The Author 2012. Published by Oxford University Press on behalf of the European Association for Cardio-Thoracic Surgery. All rights reserved.
ED
ITO
R
IA
LS
European Journal of Cardio-Thoracic Surgery 41 (2012) 729–731
EDITORIAL
doi:10.1093/ejcts/ezs057
*‘Illum qui est gravitates magni observe!’ (‘Pay careful attention
to that which is of great importance!’)
THE EUROSCORE II REFERENCE DATABASE
The reference class used for EuroSCORE II is an extremely large
database of 24 385 records (patients), originating from
volun-teering units. The project managers need to be applauded for
the collection and connection of so many records. There has
been no external validation of this dataset, not even of a
random sample. The quality is therefore dependent on the
repeated testimony of the units responsible. This testimony is
devalued by the observation of double, triple and quadruple
submissions of the same record.
The domain studied is adult cardiac surgery and the project
managers have chosen a common predictive model covering the
complete domain. This is a philosophical and pragmatic decision
that has both benefits and limitations. The system is indeed
ap-plicable to a complete unit of adult surgery, but the variables' list
loses specificity. Indeed different variables play a role, possibly
with a different coefficient, in different pathologies or surgical
therapies. Echocardiographic data, as an example, play a
domin-ant role in valve surgery but are possibly less importdomin-ant in
cor-onary surgery. Patients in cardiogenic shock or cardiopulmcor-onary
resuscitation, as in aortic dissections, coronary bypass surgery or
endocarditis, demand a completely different list of variables [
6
]
never encountered in traditional scoring systems. The project
managers could have responded to their philosophical decision
by including, for all patients, a list of variables from different
sub-domains of adult cardiac surgery that would possibly only play a
role in some of them. They decided not to respond and thereby
reduced the quality of their global reference class versus the
outcome event.
The selection and the format of the collected variability assure
the richness of the reference class. The variable list has been
minimally improved with some additional variables, but
consid-erable variability remains excluded (quality of life, frailty, mental
reserves etc.). This is most certainly a missed opportunity that
could have revolutionised cardiac surgery.
One of the major limitations of EuroSCORE I was the
parsi-moniously dichotomous (yes/no) registration of variability, even
with the availability of validated, possibly transformed,
continu-ous presentations. Except for renal function, it is unclear from
the manuscript if all continuous variability is registered in a
continuous format since the final model repeats the use of
di-chotomous presentations for this variability. For example, by
not presenting pulmonary function in the format of vital
cap-acity (or % of normal) or one-second value (or % of normal)
and ventricular function in the format of ejection fraction,
end-diastolic pressure, end-systolic/end-diastolic volumes, a repeated
op-portunity has been lost to enrich the reference dataset.
Continuous variables could indeed easily be transformed in a
search for an optimal relation between the outcome event and
the available variability. In addition, risk is never residual in the
average value of a variable but in the density at the outliers. As
an example, to possibly allow body mass index (BMI) to enter
into a final model with a correct transformation and coefficient,
the reference dataset needs sufficient patients with a BMI <20,
or <15 or with a BMI >35, >40, >45, >50. This density
informa-tion is not transparent in this manuscript; therefore, any
vari-able selection or elimination cannot be discussed and
evaluated.
The manuscript does not give sufficient information about
missing values, since the authors classify variables as
‘compul-sory’ and ‘non-compul‘compul-sory’ after the analysis. They identify
missing values in compulsory variables as important and
not-important in non-compulsory variables. They also give the
impression of not having taken action to improve the missing
data or impute the missing data using any of the available
methods.
THE EUROSCORE OUTCOME EVENT
EuroSCORE II has the ambition of predicting early mortality, so
the analysed outcome event is of the utmost importance for the
quality of the reference class. The authors have correctly chosen
the most discrete and serious event: mortality. Early mortality
after adult cardiac surgery extends for coronary bypass patients
[
7
] up to 3 months and even further for valve patients in
follow-up. The authors, have therefore correctly chosen the 90-day
ob-servation interval as a primary outcome variable. The reference
database has 55.4% missing information for this appropriate
interval, and even for the secondary non-appropriate end-point
of 30-day 43.4% of the information is missing. This information
destabilizes the reader and raises a series of questions. The 160
participating units only needed to follow-up on average 140
patients each. To complete a follow-up of a patient, an average
of three contacts per patient are needed and an average time of
<1 h per patient [
8
]. The authors have consequently redefined
their outcome interval into what they have as available
informa-tion, namely the biased hospital stay in the primary hospital.
This is incomplete, biased and inadmissible.
It is indicative of a failure of the volunteering units and their
unit’s responsibilities in their participation of this project. A
pos-sible ethical problem in cardiac surgery is magnified to society
through this project.
These and the previous observations classify the reference
class as sparse and inappropriate for forecasting rare events.
Further reading of this manuscript places the reader and society
at risk of false interpretation. Therefore, this model should not
be used, as such, for quality monitoring or comparison, for
dif-ferential therapy or informed consent and most certainly not for
the public reporting of medical performance.
IMPROVEMENTS IN CALIBRATION AND
DISCRIMINATION
Appropriate statistical methods were used on this sparse
refer-ence class; we therefore remain reluctant to discuss the
infer-ences proposed. The authors have been able to recalibrate
Table 1: Methods for prediction of rare events
Statistical forecasting
Expert judgement
Structured judgemental decomposition
Structured analogies
Statistical forecasting with judgemental intervention or adjustment
Delphi
Prediction markets
Scenario planning
P. Sergeant et al. / European Journal of Cardio-Thoracic Surgery
730
•
EuroScore I, período de observação de diferentes doenças e
tratamentos;
•
Não há informação densa no espectro de variáveis extremas;
•
Hosmer-Lemeshow compara proporções e testa se há
evidência de má calibração;
•
As variáveis devem ter pesos diferentes (coeficientes) in
doenças diferentes e operações;
•
Ainda não inclui variáveis importantes como fragilidade;
•
Variáveis contínuas não são claras, exceto função renal.
Disfunção pulmonar (sim) x (não), por exemplo.
•
“Missing information” 43,4% follow-up de 30 dias.
EuroSCORE II, illum qui est gravitates magni observe*
Paul Sergeant*, Bart Meuris and Matteo Pettinari
Department of Cardiac Surgery, Gasthuisberg University Hospital, Leuven, Belgium
* Corresponding author. Department of Cardiac Surgery, Gasthuisberg University Hospital, Herestreet 40, 3000 Leuven, Belgium. Tel: 344219; fax: +32-16-344616; e-mail: [email protected] (P. Sergeant).
Keywords: Risk assessment • EuroSCORE • Cardiac surgery • Mortality
*‘Illum qui est gravitates magni observe!’ (‘Pay careful attention to that which is of great importance!’)
The authors of the EuroSCORE I project have submitted a follow-up project called EuroSCORE II [1]. In this project, they intend to improve the discrimination and calibration of their first mathem-atical model (additive and logistic versions).
THE EUROSCORE I
A Google search for EuroSCORE identifies at least 108 000 refer-ences and more than 1300 formal citations. The first versions have indeed been used and misused without an in-depth under-standing of their limitations.
EuroSCORE I has been used as a quality monitoring or com-parison tool. The quality of care is largely dissociated with the early risk. Indeed, the quality of care in cardiac surgery proce-dures has to use extended observation intervals years beyond the observation interval of the EuroSCORE models. In addition, the quality of care involves a whole series of criteria as there are appropriate diagnostic systems, waiting times, early risks, late benefits, resources used …
EuroSCORE I has been used for differential therapy or informed consent forms to express the early risk to patient and society. This use was similarly inappropriate since the risk inter-val should be based on the observation of the hazard function and will vary for each pathology, each event, major variability in the procedure and in post-procedural approaches. EuroSCORE I therefore needed to use an observation interval including the observation intervals mandatory for all major pathologies and procedural approaches, included in its reference database. It failed to do so and therefore always presented an incorrect and incomplete depiction of this early risk. The dramatic misuse of the EuroSCORE for the TAVI (Transcatheter Aortic Valve Implantation) market expansion was also a mistake. There has never been any information about the density at this high-end of the risk spectrum of the original reference database, and even less information about discriminatory and calibration power within this zone of risk.
A similar misuse has been the application of EuroSCORE I for early non-lethal events as renal failure or length of stay. Of course, there were some predictive values and some ROC values in amalgamating the more usual risk factors (even using
an irrelevant coefficient), but this misuse rejected the complete notion of outcome analysis, its rules and limitations.
ANALYTIC PROCESS
Early risk is a rare event and there were a number of possible strategies [2] available for the development of EuroSCORE II. They are listed in Table 1. The authors have decided to repeat the previously used statistical forecasting and have added some judgment adjustment. The essence of the method used is the basis of the knowledge, often called the reference class (the database).
The science of risk prediction forces us to focus on three lim-itations of this approach and evaluate this manuscript versus those criteria.
The sparsity in the reference class improves with the actuality, the size and the richness of the database, most of all in the density of the extremes of variability. In the presence of a spars-ity in the reference class, statistical forecasting becomes unreli-able. This unreliability is slightly improved through judgment adjustment.
Secondly, the inappropriateness of the reference class makes the statistical forecasting unreliable. An inappropriate reference does not include the essential variability or the appropriate outcome.
Thirdly, the inappropriate statistical model (e.g. oversimplified models, poor calibration etc.) destabilizes completely statistical forecasting. There is mixed evidence about judgement adjusting the model. The accuracy of a mathematical model evaluates cali-bration, discrimination and a combination of both. The calibra-tion describes how well the predicted probabilities agree with the actual observed risk. The Hosmer–Lemeshow statistic com-pares proportions but is most certainly imperfect. A non-significant Hosmer–Lemeshow test means that there is no evi-dence of bad calibration, it does not mean that there is good calibration. The discrimination describes how well a model sepa-rates black from white. The methods used are the sensitivity (true positive), the specificity (true negative), the positive predict-ive value (PPV), the negatpredict-ive predictpredict-ive value (NPV), the misclas-sified, the ROC [3] (receiving operating characteristics) and the C-statistic (concordance index). The combination of both calibra-tion and discriminacalibra-tion is best described using the likelihood sta-tistics, the R2[4] and Brier [5] scores.
© The Author 2012. Published by Oxford University Press on behalf of the European Association for Cardio-Thoracic Surgery. All rights reserved.
ED
ITO
R
IA
LS
European Journal of Cardio-Thoracic Surgery 41 (2012) 729–731
EDITORIAL
doi:10.1093/ejcts/ezs057