components, uptake, and effectiveness of quality improvement toolkits

(1)

S Y S T E M A T I C R E V I E W Open Access

Spread tools: a systematic review of

components, uptake, and effectiveness of quality improvement toolkits

Susanne Hempel

^1,2*

, Claire O ’ Hanlon

³

, Yee Wei Lim

⁴

, Margie Danz

^1,2

, Jody Larkin

⁵

and Lisa Rubenstein

³

Abstract

Background: The objective was to conduct a systematic review of toolkit evaluations intended to spread

interventions to improve healthcare quality. We aimed to determine the components, uptake, and effectiveness of publicly available toolkits.

Methods: We searched PubMed, CINAHL, and the Web of Science from 2005 to May 2018 for evaluations of publicly available toolkits, used a forward search of known toolkits, screened references, and contacted topic experts. Two independent reviewers screened publications for inclusion. One reviewer abstracted data and appraised the studies, checked by a second reviewer; reviewers resolved disagreements through discussion.

Findings, summarized in comprehensive evidence tables and narrative synthesis addressed the uptake and utility, procedural and organizational outcomes, provider outcomes, and patient outcomes.

Results: In total, 77 studies evaluating 72 toolkits met inclusion criteria. Toolkits addressed a variety of quality improvement approaches and focused on clinical topics such as weight management, fall prevention, vaccination, hospital-acquired infections, pain management, and patient safety. Most toolkits included introductory and implementation material (e.g., research summaries) and healthcare provider tools (e.g., care plans), and two-thirds included material for patients (e.g., information leaflets). Pre-post studies were most common (55%); 10% were single hospital evaluations and the number of participating staff ranged from 17 to 704. Uptake data were limited and toolkit uptake was highly variable. Studies generally indicated high satisfaction with toolkits, but the perceived usefulness of individual tools varied. Across studies, 57% reported on adherence to clinical procedures and toolkit effects were positive. Provider data were reported in 40% of studies but were primarily self-reported changes. Only 29% reported patient data and, overall, results from robust study designs are missing from the evidence base.

Conclusions: The review documents publicly available toolkits and their components. Available uptake data are limited but indicate variability. High satisfaction with toolkits can be achieved but the usefulness of individual tools may vary. The existing evidence base on the effectiveness of toolkits remains limited. While emerging evidence indicates positive effects on clinical processes, more research on toolkit value and what affects it is needed, including linking toolkits to objective provider behavior measures and patient outcomes.

Trial Registration: PROSPERO registration number: PROSPERO 2014:CRD42014013930.

Keywords: Spread, Diffusion of innovation, Quality improvement, Toolkit, Implementation

© The Author(s). 2019Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

* Correspondence:susanne_hempel@rand.org

1Southern California Evidence-based Practice Center, RAND Corporation, Santa Monica, USA

2Southern California Evidence Review Center, University of Southern California, Los Angeles, USA

Full list of author information is available at the end of the article

(2)

Background

Diffusion of innovations is a complex process. While re- search studies continue to show successful interventions to improve healthcare, their dissemination is slow [1–3].

Implementations of proof of concept studies and adop- tion of interventions shown to be effective in research studies into routine clinical practice is delayed or not achieved at all.

In recent years, a number of organizations have devel- oped “toolkits” for healthcare quality improvement [4].

Toolkits are resource and tool collections designed to fa- cilitate spread across settings and organizations and to ease the uptake and implementation of interventions or intervention bundles and practices. They are a resource for documentation of interventions, for implementation of successful interventions, and for scaling up initiatives developed in pilot or demonstration sites into large-scale rollouts. Toolkits may include a variety of materials use- ful to organizations to help introduce an intervention, practical tools to help incorporate best practices into routine care such as pocket cards for healthcare pro- viders, or patient education materials. There is currently no definition of nor standard approach to toolkit con- tents or formats.

A variety of healthcare research agencies publish toolkits. The US Agency for Healthcare Research and Quality (AHRQ) alone has published a large number, on topics ranging from allergy and immunologic care to urologic care. The AHRQ Healthcare Innovations Ex- change website has tracked the development of tools or toolkits to improve quality and reduce disparities (web- site maintenance ended in 2017). Users may browse the resources online or download them free of charge. Little is known, however, about uptake of published toolkits.

While exact copying of the intervention is possible, a process of re-invention in the new context is also likely to occur. Re-invention may change the intervention to some extent during the diffusion process as it transitions from the developer to the adopter, with or without the help of a toolkit [5], potentially resulting in decreased but still significant effort for toolkit adaptation [6]. To date, we know very little about successful components that may be useful across toolkits, about the toolkit adoption process, or about what makes toolkits easier or harder to adopt.

Furthermore, little is known about the effectiveness of published toolkits. A scoping review describing toolkits assembled for individual research projects concluded that the toolkits often did not specify the evidence base from which they draw and their effectiveness as a know- ledge translation strategy was rarely assessed [1, 7]. The effectiveness of a toolkit is likely to depend on its quality, the effectiveness of the intervention, and the set- ting characteristics. However, for published toolkits, an

additional consideration is apparent. Toolkits applied in new settings may not be as effective as seen in the original implementation of the intervention bundle that led to the development of the toolkit. Potential reasons include diminished healthcare provider motivation, re- duced staff buy-in, or other aspects of low readiness (e.g., healthcare providers were not instrumental in initi- ating and shaping the interventions).

Our objective was to conduct a systematic review on the spread of interventions intended to improve health- care quality through toolkits. This systematic review aims to determine the following key questions:

Key question 1: What are the components of published quality improvement toolkits?

Key question 2: What is the uptake and utility of published quality improvement toolkits?

Key question 3: What is the effectiveness of published quality improvement toolkits?

The review explores the types of tools included in toolkits, measures and results that describe the uptake and utility, and the effectiveness of published toolkits to inform users and developers of toolkits.

Methods

We registered in PROSPERO, registration number PROSPERO 2014:CRD42014013930. The reporting fol- lows the PRISMA guidelines (see Additional file 1).

Searches

We searched the databases PubMed, CINAHL, and Web of Science for evaluations of toolkits in May 2018. The PubMed search strategy is given in full in Additional file 2. The strategy searched for the term “toolkit” in the title, abstract, keywords, or full text of the publication (Web of Science only). We did not limit the search to publications using the MeSH term “diffusion of innovation” because the pilot search strategy showed that known toolkit evaluations were not systematically tagged with this term. We limited to English-language citations published since 2005 to identify current toolkits readily applicable to US settings.

In addition, we searched resources from nine organiza- tions dedicated to healthcare improvement to find pub- lished toolkits: AHRQ, World Health Organization (WHO), Institute for Healthcare Improvement [IHI], Robert Wood Johnson Foundation [RWJF], Association of perioperative Registered Nurses [AORN], Emergency Care Research Institute [ECRI], Centers for Disease Control and Prevention (CDC), Centers for Medicare and Medicaid Services (CMS), and Department of Veterans Affairs (VA). We also screened the category

“QualityTool” in AHRQ’s database of innovations. A

(3)

“forward search” identified any publication that had cited the titles of the toolkits we located. We screened included studies and relevant reviews and contacted content ex- perts to identify additional relevant publications.

Study inclusion and exclusion criteria

Two independent reviewers screened titles and abstracts to avoid errors and bias. We obtained publications deemed as potentially relevant by at least one reviewer as full text. Full text publications had to meet the out- lined criteria to be eligible for inclusion in the review.

Discrepancies were resolved through discussion in the review team. In the absence of a universally agreed definition of a toolkit, the project team developed the outlined working definition.

Participants and condition being studied: Publications evaluating toolkits in healthcare delivery organizations were eligible. The review was not limited to toolkits targeting specific clinical conditions, but toolkits had to be aimed at healthcare. Toolkits aimed primarily at other than healthcare provider professions (e.g., policy makers in non-healthcare delivery settings), or aimed at students not yet involved in healthcare delivery (e.g., nursing students) were excluded. Toolkits only aimed at patients, such as patient education material or patient self-management programs, were excluded.

Intervention and toolkit definition: Studies evaluating the use of toolkits designed to aid healthcare delivery organizational were eligible. A

“ toolkit ” was defined as an intervention package, or set of tools. Toolkits had to be aimed at quality improvement (an effort to change/improve the clinical structure, process, and/or outcomes of care by means of an organizational or structural change) [8] of healthcare; toolkits to increase research capacity or workforce issues were excluded. Test batteries, image processing protocols, or computer software termed “toolkit” were not eligible. Toolkits had to be either publicly or commercially available.

Comparator/study design: Studies evaluating the use of existing toolkits were eligible. Studies supporting the development of toolkits and reporting on earlier versions rather than the currently available toolkits were excluded. Controlled and uncontrolled studies with historic (e.g., pre-post studies) or concurrent comparators (e.g., randomized controlled trials, RCTs) were eligible. Comparators could include active controls (a different intervention) or passive controls (e.g., status before the introduction of the toolkit).

Outcome: Publications reporting on patient, provider, or organizational findings were eligible.

Studies had to report on structured evaluations (e.g., surveys); informal or anecdotal evaluation

statements were not sufficient.

Timing: To capture current and relevant toolkits developed in accordance with current standards and applicable material, evaluated toolkits must have been published in 2005 or more recently, or be still available.

Setting: Implementations of toolkits were included regardless of the setting, but the original toolkits had to be aimed at quality improvement in health care.

Toolkits developed for other than healthcare delivery organizations such as school settings or laboratories as well as toolkits primarily focusing on health system improvements in conflict zones or disrupted healthcare systems were excluded.

We consolidated publications reporting on the same sample of participants. Evaluations published in academic journals as well as gray literature (conference abstracts, dissertations) were eligible. The literature flow diagram is shown in Fig. 1.

Potential effect modifiers and reasons for heterogeneity

The review included a large number of study designs and study outcomes to allow a comprehensive overview of the available evidence on toolkits. In particular, the study design (e.g., comparative studies, post-only study) and the study outcomes (e.g., feasibility, patient health outcome) were sources of heterogeneity across studies.

Data extraction strategy

One reviewer abstracted and a second experienced systematic reviewer checked the data; disagreements were resolved by discussion. We determined categories based on the initial review of publications and used a piloted-tested data extraction form to ensure standard- ized data abstraction.

We extracted the toolkit name, the developing organization, the general area of application, the toolkit components, and type of availability (publicly or com- mercially). In addition, information on the evaluation—

including study design, participants, setting, and additional non-toolkit components—were extracted.

We documented the uptake and adherence to toolkit

components (e.g., number of downloaded toolkits); util-

ity and feasibility; healthcare provider measures includ-

ing knowledge, attitudes, and barriers; procedural,

structural, and organizational changes (e.g., number of

ordered tests); and patient outcomes including patient

health outcomes and patient-reported satisfaction. We

added effectiveness results from the development phase

of the toolkit where available.

(4)

Study quality assessment

We used the Quality Improvement Minimum Quality Criteria Set (QI-MQCS) to assess studies [9]. The QI- MQCS is a 16-item scale designed for critical appraisal of quality improvement intervention publications; the domains are described in Additional file 2. The synthesis for the primary outcome integrates the appraisal finding;

results for all included studies are documented in Additional file 2.

Data synthesis and presentation

We documented the included studies in an evidence table (with supporting tables in the appendix) and sum- marized evaluation results in a narrative synthesis. Given the diversity of the identified studies, the quality of evidence assessment was limited to assessing inconsist- ency in study results across studies and study limitations of identified studies. The synthesis followed the key questions. Key question 1 was organized by the devel- oped framework of components. Key question 2 was or- ganized by outcome category: uptake and utility. Key question 3 was organized by provider outcomes, proced- ure/organizational results, and patient outcomes. The primary outcome of the review was patient health out- comes. The synthesis differentiated evidence from stud- ies with concurrent and with historic comparator. For each toolkit, the evaluation of the intervention spread (i.e., using an available toolkit to disseminate practices and tools included in the toolkit) was also contrasted

with initial results obtained in the organization where the toolkit had been first developed (where information was available).

Results

Review statistics

The electronic search for “toolkit” publications and a forward search for 156 specific toolkits (see Additional file 2) published by AHRQ, CMS, WHO, IHI, RWJF, AORN, ECRI, CDC, VA, or on the AHRQ Innovation Exchange identified 5209 citations. We obtained 661 citations as full text articles; of these, 77 studies were identified that met inclusion criteria (Fig. 1).

Study characteristics

Four included evaluations of groups randomized to an intervention or a control condition. Six studies provided a comparison to concurrent (non-randomized) control groups that did not participate in toolkit implementation.

Forty-two studies presented pre- and post-intervention data for at least one outcome but did not include a con- current comparator to account for secular trends inde- pendent of the intervention. Twenty-five studies reported only post-intervention data and provided no comparison to the status before or without the toolkit. Assessment methods and reported details varied widely and included online and written staff surveys, administrative data, medical chart review data, and web statistics.

Records identified through database searching

(n = 5117)

Additional records identified through other sources (Specific toolkit search,

pre-toolkit studies n = 92)

Records screened after duplicates removed (n = 5,209)

Records excluded n = 4,548 (no potential quality improvement toolkit

publication)

Full-text articles assessed for eligibility (n = 661)

Full-text articles excluded, with reasons Intervention n = 302

Outcome n = 221 Duplicate n = 12 Publication year n = 2 Retained as background

(more information on included studies or pertinent

review) n= 47 Studies included in the

review (n = 77)

Fig. 1Literature flow diagram

(5)

The range of healthcare organizations involved in the evaluation varied widely from single hospital evaluations (10%) to studies with data on 325 institutions; and 22%

of studies, often those that reported on web download statistics, did not report on the number of institutions.

The number of participating staff members, often healthcare providers asked to use tools contained in the toolkit in clinical practice, ranged from 17 to 704, but the number of participants was only reported in 47% of studies. Of those studies reporting patient data, 59% re- ported the number of patients the data were based on;

the number varied and ranged from 43 to 337,630.

Sixty-nine percent of included evaluations described elements in addition to the toolkit such as workshops and presentations to introduce the toolkit or the inter- vention promoted in the toolkit. The developer of the toolkit was part of the evaluation of the toolkit in more than half of the included studies (59%); toolkits were evaluated by independent study groups in 27% of studies (14% unclear).

Most evaluations were conducted in the USA (75%);

other countries contributing to the study pool were Canada, the UK, Australia, Mongolia, and an inter- national evaluation with multiple countries. In 34% of studies, the evaluation setting was a hospital; in 32%, toolkits were evaluated in primary care facilities; other organizations included community health centers, ambu- latory care clinics, long-term care facility, specialty clinics (e.g., multiple sclerosis clinic), a hospice, and in some cases the characteristics were not reported.

The details of the included studies are shown in the evidence table (Table 1).

Quality assessment

As a critical appraisal tool, the QI-MQCS targets the in- formational quality of QI studies and informs decisions about applicability of results to other settings. The num- ber of criteria met per study ranged from 3 to 14 (mean 9.78, SD 3.04). Since the objective of this systematic review was to assess the spread of QI interventions through the use of toolkits, 100% of included publica- tions/studies addressed Spread and described the ability of the intervention to be replicated in other settings.

In addition, for ten of the 16 domains, more than 50%

of the included publications met the minimum QI- MQCS criteria. The top five described aspects related to study initiation and included Organization motivation (description of the organization reason, problem, or mo- tivation for the intervention, 93%); Intervention rationale (description of the rationale linking the intervention to the effects, 88%); Intervention (description of the pro- cesses, strategies, content, and means of achieving the effects associated with the intervention and considered to be permanent as opposed to activities considered to

be temporary for the purpose of introducing the inter- vention, 70%); Implementation (description of the ap- proach to designing and/or introducing the intervention, 81%); and Data sources (documentation of how data were obtained and whether the primary outcome was defined, 82%). The other five domains, for which more than 50% of studies met minimum QI-MQCS criteria, included Organizational characteristics (description of setting demographics and basic characteristics, 68%);

Timing (clear outline of the timeline for intervention im- plementation and evaluation so that follow-up time can be assessed, 60%); Adherence/fidelity (level of compli- ance with the intervention, 57%); Organizational readi- ness (description of QI culture and resources available for the intervention, 64%); and Limitations (outline of limitations and the quality of the interpretation of findings, 68%).

The five domains, for which less than 50% of studies met minimum QI-MQCS criteria, addressed evaluation of results and included Study design (documentation of the evaluation approach with respect to study design, 36%); Comparator (description of the control condition against which the intervention was evaluated, 26%);

Health outcomes (inclusion of patient health outcomes in the evaluation, 17%); Penetration/reach (reporting of the proportion of eligible units that participated in the intervention, 29%); and Sustainability (information on the potential for maintaining or sustaining the interven- tion with or without additional resources, 40%).

Key question 1: what are common elements of quality improvement toolkits?

The evaluated toolkits addressed a variety of quality improvement approaches. Most focused on a specific clinical topic rather than general healthcare provider be- haviors. Seven toolkits addressed weight management;

four toolkits evaluated in five studies addressed fall pre-

vention; three, emergency preparedness; three each pa-

tient safety and three perinatal care; and two (evaluated

in three studies) were aimed at vaccination. We identi-

fied two toolkits each addressing the topics asthma man-

agement, cancer screening, elective delivery, health

literacy, hospital-acquired infections, hospital readmis-

sion, medical errors, mental health, pain management,

screening, smoking cessation, and substance use. The

other toolkits addressed antimicrobial stewardship, aut-

ism communication, brain injury symptom management,

cancer care, cardiac care, care quality, clinical decision

making for critical care, depression care, diabetes care,

end of life care, geriatric care, heart failure, hepatitis C

care, kidney disease care, medication management,

multiple sclerosis symptom management, newborn

screening, nursing best practices, obstetric care, parental

education, pediatric preventive care, psychotherapy

(6)

Table1Evidencetable StudyID Country Topica

Setting Nhospitals Nproviders Npatients Studydesign

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient Ashiru-Oredope2016[56] UK Antimicrobialstewardship (AMS)

Primarycareandacutetrusts #hospitals:211cinical commissioninggroups (primarycare),146acutetrusts #providers:NR #patients:NR Post-only

TreatAntibioticsResponsibly,Guidance, Education,Tools(TARGET)(primarycare) andStartSmartThenFocus(SSTF)(trusts) Implementation:TARGET:Guidance (local/nationalantibiotictreatment recommendations),suggestedantibiotic practiceaudits.SSTF:Writtenmaterials Provider:TARGET:Educationalmaterials andtoolsforproviderstosharewith patients.SSTF:Examplesofaudittools, reviewstickers,anddrugcharts Patient:Patientleaflets(TreatingYour Infection),resourcesforclinical andwaitingareas Otherintervention:None

Uptake:Ofthe82respondinggroups,60%hadformallyor informallyreviewedTARGETand13%hadanactionplanto implementAMSinterventionsrecommendedbyTARGET. Fiftypercenthadimplementeduseofthepatient informationleaflet,59%wereusingtheTARGETeducational presentation,58%promotedTARGETduringpractice prescribingvisits,46%promotedtheuseofTARGETtoGPs foruseinCPD/revalidation.Groupsthathadreviewed TARGETweremorelikelytohaveimplementedanaction plan(OR8.68,CI1.06,71.48,p=.044),andmorelikelytohave implementedtheuseofthepatientinformationleaflet(OR 4.38,CI1.70,11.27,p=.002).Of100respondingtrusts,87% hadreviewednationalAMStoolkits,and46%had implementedanactionplantodelivernationaltoolkitAMS interventions.Acutetruststhathadundertakenareviewof SSTFweremorelikelytohaveimplementedanactionplan (OR3.33,CI1.00,11.06,p=.050). Feasibility:NR Providers:NR Procedures:Fewgroupshadimplementedsuggestedaudits orcollateddatainprevalencesurveys;however,69%groups hadusedlocalantibioticauditswithinthepast2years.The mostfrequentrecommendedauditsintrust-widepoint prevalencesurveysincluded:adherencetoguidelinesof dose,routeandduration(82%);clinicalindicationand treatmentdurationdocumentedondrugchart(82%);and intravenoustooralswitchat48h(49%).Otheraudits,suchas reviewofprescription/evidenceofdocumentingdecisionat 48handtimetofirstdoseinseveresepsis,wereless commonlyimplemented(42%and40%ofacutetrusts). Patients:NR Jones2017[69] UK Antimicrobialstewardship (AMS)

Primarycare #hospitals:56 #providers:269(surveys), 24(focusgroups),29(interviews) #patients:NA Post-only

TARGET(TreatAntibioticsResponsibly; Guidance,Education,Tools)AntibioticsToolkit Implementation:seeAshiru-Oredope,2016 Provider:seeAshiru-Oredope,2016 Patient:seeAshiru-Oredope,2016 Otherintervention:1hworkshop coveringAMR,guidance,howto optimizeantibioticprescribing,use ofresourcesintheToolkit,reflection ontheirownantibioticprescribing dataandsomeactionplanning. Workshopparticipantscompleted afive-pointLikertscaleeval

Uptake:AroundhalfofGPsreportedusingtheTARGET resourcestovaryingdegrees. Feasibility:MostGPstaffandstakeholdersdescribedthe TARGETAntibioticsToolkitasausefulresource,which addressedtheirownprescribingbehaviorandpatient expectations.Theyfeltthatitcomplementedexistingefforts andwasrelevanttoallpracticestaffindevelopinga consistentapproachtopatientenquiriesaboutantimicrobials. Providers:88%respondedthattheworkshophelpedthem understandhowtooptimizeantimicrobialprescribing,88% respondedthattheworkshophelpedthemtounderstand whyresponsibleantimicrobialprescribingisimportant.All stakeholderswerepromotingitsuse. Procedures:Theself-assessmentchecklistisakeyresource thatcanbeusedformonitoringbutwasinfrequently mentionedbyparticipants. Patients:NR

(7)

Table1Evidencetable(Continued) StudyID Country Topica

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient Bender2011[19] US Asthmamanagement

Primarycare #hospitals:58 #providers:372 #patients:~15,508 Pre-post

ColoradoAsthmaToolkit Implementation:Trainingmanual Provider:Spirometer Patient:Toolkitforpatients,peak flowmeter,guideforself-management support,telephoneoutreachenrollment form,asthmaactionplan,"understandingasthma"booklet, othereducationalmaterials Otherintervention:On-sitetraining/coaching,optional interactivevoiceresponseprogramforpatients

Uptake:98%ofpracticesofferedthetoolkittookitup; medianactionplanuseincreasedfrom0%to20%after coaching(53%ofpracticesincreased) Utility:NR Providers:NR Procedures:Patientswithasthmausinginhaled corticosteroidsincreasedfrommedian25to50%(p<.0001), patientswithanactionplan0to20%(p<.001),useof spirometry0to40%(p<.0001);costoftrainingeachpractice was$4194 Patients:NR Taylor2017[87];Tapp2017 [108] US Asthmamanagement

Community-basedpractices #hospitals:6 #providers:NR #patients:92 CT

SharedDecisionMaking(SDM)Toolkit(asthmamanagement) Implementation:Asthmahalfdayclinicflow,initialand follow-upscripts Provider:Spirometrytechnique,documentationtemplate, follow-uppatientinformationsheet,controllerreliever posters,blankcontroldial,treatmentgoalsandmedication preferencesform,priorauthorizationforms Patient:Factsaboutasthma,allergyinformation,smoking cessationinformation,severityandcontroldials,medication options,medicationplanner,generaltypesofasthma medications,howtouseyourinhalerhandouts,asthmadiary Otherintervention:Afamilymedicineproviderwhowasalso partoftheresearchteamfacilitatedandtailoredthe implementationofthetoolkittotheindividualneedsofeach practice.Therollouttypicallyconsistedofhour-longweekly sessionswithpracticeprovidersandstaff.Adoptionofthe SDMinterventionwasreinforcedwithmonthlymeetings withrepresentativesfromparticipatingpracticesanda refreshertrainingattheendofyearone.

Uptake:NR Feasibility:NR Providers:NR Procedures:Patientsurveysasking,“Whomadethetreatment decisiontoday?”werecollectedfrom319oftheSDMtoolkit visits.87%reportedashareddecisionwasmadebetween patientandproviderwith6%indicatingthepatientmostly madethedecision,74%indicatingitwassharedequallyand 7%indicatingtheprovidermostlymadethedecision. Patients:Thetoolkitinterventionwasassociatedwithhigher qualityoflifescoreswhencomparedtousualcare(MD0.9; CI0.4–1.4).Modelsassessingtheimpactonindividualquality oflifedomains(activities,emotions,symptoms)revealed similarresults.Similarly,modelsexaminingdifferencesin asthmacontrolshowedlowerasthmacontrolproblemsfor childrenintheSDMtoolkitinterventioncomparedtousual care(MD−0.9;CI−1.6–0.2). Nicolaidis2016[79] US Autismcommunication

Primarycare #hospitals:NR #providers:51 #patients:259 Pre-post

AcademicAutismSpectrumPartnershipinResearchand Education(AASPIRE)Healthcaretoolkit Implementation:Autisminformation/diagnosis/referrals,legal andethicalinformation,othereducationalresources Provider:AutismHealthcareAccommodationsTool(AHAT), checklistsandworksheets Patient:Personalizedaccommodationsreport,patient information,checklists,worksheets Otherintervention:Patientsrecruited,sentAHATtopatient's PCPwithacoverletter,informationsheetaboutthestudy,

Uptake:NR Feasibility:MostPCPsrateditasmoderatelyorveryuseful andindicatedthattheywouldrecommendittotheir patients.MostanswersfromPCPstoopen-endedquestions werepositiveandprovidedexamplesofthetool'sutility. However,severalPCPsnotedthattheyalreadyweredoing whatwasrecommendedinthereport,andtwoPCPsfeltthat theydidnothavetimetoimplementaccommodations. Providers:NR Procedures:NR Patients:Almostallautisticparticipantsandsupportersfelt thattheAHATandthetoolkitwereeasytouse,important, anduseful.Over90%saidtheywouldrecommendthetoolkit toafriendortheirhealthcareprovider.Thetotalnumberof barriersencounteredbypatientsdecreasedsignificantly (p<.001).Participants’self-efficacyinnavigatingthe healthcaresystemalsoincreased(p=.02).Participants

(8)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient describedthetoolkitasgivingthemameanstoclarifyand communicatetheirneeds.Manyparticipantsfeltthatthe toolkitvalidatedtheirexperienceandempoweredthemto self-advocatemoreeffectively.Participantsalsooftengave examplesofhowthetoolkitimprovedtheirself-efficacy, especiallybyhelpingthemprepareforvisits.Most participantswereenthusiasticabouthowtheAHATreport mightaffecttheirPCPs'behavior.Amongthe43patients whosawtheirPCPwithinthe1-monthfollow-upperiod, satisfactionwithPCPcommunicationimprovedsignificantly (p=.03).Participantsdescribedmanyconcretepositive changesinprovidersortheirstaff.Severalparticipantsvoiced frustrationthattheirprovidersdidnotreadtheAHATreport ordidnotmakeanychangesbasedonthereport. Chrisman2011[18] US Braininjurysymptom management

Primarycare #hospitals:NR #providers:414 #patients:0 RCT HeadsUp:BrainInjuryinYourPractice Implementation:Bookletdescribingdiagnosisand management,CDwithadditionalresources,onlineresources suchaspostersandcustamizablehandoutsavailable Provider:Palmcard,AcuteConcussionEvaluation(ACE)form Patient:PatientinformationinEnglishandSpanish Otherintervention:NR

Uptake:NR Utility:NR Providers:Nodifferenceingeneralconcussionknowledge betweeninterventionandcontrolgroups;intervention physicianslesslikelytorecommendnextdayreturntoplay afterconcussion(OR0.31,CI0.12,0.76) Procedures:NR Patients:NR Latsko2015[72] US Cancercare

Hospital #hospitals:NR #providers:165 #patients:NR Post-only TreatingMyelodysplasticSyndrome(MDS)Toolkit Implementation:Summaryofthepatientsurvey,counseling guide Provider:Diagnosticspectrumreferencecard,mobileapp Patient:Patientinformationsheets,freepatientbookletorder form Otherintervention:None

Uptake:24%ofrespondingoncologynursesreportedhaving atoolkit. Feasibility:NR Providers:Comparisonoftheresponsesofnursesin possessionofthetoolkit:ForMDSEducation,oncology nursesreportedoneducationatthefollowingtimes:During thediagnosticworkup(53vs41%);ateachfollow-up appointment(75vs51%);priortoachangeintreatment(85 vs74%);atcompletionoftreatment(53vs33%)andatthe timeofdiseaseprogression(78vs68%).Forsideeffect education,respondentsreported:priortoadministrationof eachtreatment(93vs86%);ateachtreatmentappointment (75vs70%);atthecompletionofeachtreatment(50vs. 39%);andduringfollow-upcallsaftertreatment(45vs40%). Procedures:NR Patients:NR Gulati2015[67] UK Cancerscreening

Primarycare #hospitals:3374practicesthatusedtoolkit betweenMarch2012andJune2013 #providers:8163users,1002-1007survey responders,276GPsmakingreferrals #patients:NR Post-only GeneralPractitioner(GP)SkinCancerToolkit Implementation:NR Provider:Onlineresourcesincludingreferralguidelines,real- lifecasehistories,accreditedquiz Patient:NR Otherintervention:Promotedthoughregularemail notificationstoGPsandonthehosthomepage Uptake:Accessedby8163GPs12586timesviewing127036 pageviewsandspendingamediantimeof4mineach (mean5.37min)over255days.1/3usersusedthesitemore thanonce;therewere6%moreuniqueusersand23%more uniqueusersthanthenextmostpopulareducation campaignonDoctorsNetUK(DNUK)in2012.Clinicalcase studieswereusedmostcommonly(3082timesby2587 users),followedbythereferraldecisionaid(2935times)and thelesionrecognitionresource(2215times).

(9)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient Feasibility:FocusgrouspprincipalthemesincludedthatGPs consideredthemselvestobegatekeepersforreferring suspiciouslesions,butwereconcernedaboutavoiding unnecessaryreferrals;existingresourceswereuseful,butprior dermatologyteachinghadbeenpoor;thetoolkitwasuseful fordiagnosingskinlesionandallayingpatientanxiety,but didnotinfluencereferralrates.Criticismsofthetoolkit includedlackofinformationandimagesforborderlineand atypicalskinlesions.Suggestedimprovementstothetoolkit includedmoreclinicalimages,onlineforumsfordiscussing uncertaincasesandazoomfunctionforclinicalimages. Providers:Reportedconfidenceinrecognizingdifferentskin lesionssuspiciousofskincancerwaslowerin2013compared to2011(p<0.01)buttherewerenodifferencesbetween toolkitusersandnon-users(p=0.798).Confidencein knowledgeoftheappropriatereferralpathwaysformalignant skinlesionswashigherinrespondentswhousedthetoolkit (p<0.01).Therewasnodifferenceinperceivedinformation andtrainingaboutskincancerrecognition(p=0.786). However,morerespondentswhohadusedandrecalled usingthetoolkitsaidtheyhadreceivedadequatetraining andinformationcomparedtonon-users(p<0.05).The proportionofappropriatereferralsincreasedfrom21to32% (p<0.0001). Procedures:Althoughtheabsolutenumbersofurgentskin cancerreferralsandmelanomaandnon-melanoma diagnosesincreased,therewerenosignificantchangesinthe numberofurgentGPreferralsforsuspectedskincancer(p< 0.001),diagnosesofmelanoma(p<0.001)ordiagnosesof non-melanomaskincancer(p<0.001)betweenthetoolkit userandnon-usergroups. Patients:NR Spruce2012[17] US Cancerscreening

Primarycare #hospitals:NR #providers:30outof106invited #patients:NR Post-only

ImprovingColonCancerScreeninginNevadawithaPrimary CareToolkit Implementation:Samplechartaudits,trackingsheets, decisionaids,officestrategiestoimprovescreening, resources Provider:Phonescripts,careflowsheets,recommendations, algorithm Patient:Reminderandresultsletters,postcards Otherintervention:NR

Uptake:NR Utility:Allparticipantswereverysatisfiedorsatisfiedwiththe overallusefulnessofthetoolkit,97%wereverysatisfiedwith theeducationalcontent,allwereverysatisfiedthattheinfo waspresentedclearly,83%wereverysatisfiedwithoffice strategiestoimprovescreening,allwereverysatisfiedwith algorithmsandtools Providers:95%statedtheywouldincreasepracticeuseof fecalimmunochemicaltestsforthosepatientswhoarenot eligiblefororrefuseacolonoscopyandwouldimplement manyofthetoolkitrecommendations Procedures:NR Patients:NR Adsett2014[41] AustraliaNR #hospitals:NRHEART(HeartEducationAssessmentRehabilitationToolkit) OnlineUptake:NR Utility:Ratingsofthetoolkitonafivepointscale(5=strongly

(10)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient Cardiaccare#providers:340 #patients:NR Post-only Implementation:Literature,references,links,videos,glossary Provider:Assessmenttools,clinicalcalculators(e.g.BMI), educationalmaterialsandtipsforinitiatingpatienteducation Patient:Self-helpresources Otherintervention:NR agree)were>4on8surveyitemsrelatedtocontent accuracy,easeofuse,andrelevancetopractice Providers:NR Procedures:NR Patients:NR Callard2012[11] UK Carequality

Hospital #hospitals:2trusts #providers:NR #patients:NR Post-only 15StepsChallengetoolkit Implementation:Slidesets,briefingnotes,actionplan template Provider:NR Patient:Wardnotice Otherintervention:NR Uptake:2000electronicversionshavebeendownloaded fromMaytoDecember2012 Utility:NR Providers:NR Procedures:NR Patients:NR Kemertzis2018[70] Australia Clinicaldecisionmaking

Hospital #hospitals:1 #providers:59 #patients:11 Pre-post FertilityPreservationToolkit Implementation:Instructionbooklet Provider:Referralforms,informationandconsentforms, researchinformationandconsentforms Patient:Informationsheets,bookletforpatientsandfamilies, leafletsonfertilitypreservationoptionsandbrochureon treatmentsandfertility(maleandfemaleversion) Otherintervention:Educationsessiontointroducetoolkit

Uptake:59of104(56.7%)agreedtoparticipateinthe evaluation Feasibility:Cliniciansweresatisfiedin7/11(64%)discussions, extremelysatisfiedorsatisfiedin11/11(100%).Reasonsfor dissatisfactionweremissingdocumentswithinthetoolkit, organizationofthedocumentswithinthetoolkit,and perceptionthattherewastoomuchwritteninformation whichcouldoverwhelmfamiliesandclinicians.Theclinician perceivedpatientorfamilytohaveextremelyorreasonably wellunderstoodthefertilitypreservationdiscussionin10/11 (91%)casesandwereperceivedtobesatisfiedinallcases. Providers:Therewasanoverallimprovementinparticipant confidencelevelsinprovidingup-to-datefertilitypreservation information(p=0.005). Procedures:Therewasanoverallimprovementinthe provisionofverbal(p=0.003)andwritten(p=0.02) informationposttoolkituse. Patients:NR Pierce2016[81] US Criticalcare

Hospital #hospitals:6 #providers:NR #patients:NR CT CriticalCareProtocolToolkit(CCPT) Implementation:Writtenmaterials(listanddescriptionof stepsinvolvedinprocess),summaryofkeypoints Provider:None Patient:None Otherintervention:NR

Uptake:Thetoolkitgroupfollowedall9stepsinthetoolkit andused14.3ideasfromthetoolkitonaverage.Thenon- toolkitgroupmissedorweaklyaddressed5.3stepson average. Feasibility:Thetoolkitgroupexperienced13barrierson averagecomparedto5.7inthenon-toolkitgroup(p=0.512). Ofthebarriersencountered,thetoolkitcouldhavehelped withbarriers62%ofthetimeinthetoolkitgroupcompared to77%inthenon-toolkitgroup. Providers:NR Procedures:Thecontrolgroupmissedorweaklyaddressed onaverage3.3of9keysteps.TheCCPTreduced implementationtimefrom56.4daysperstepinthenon- toolkitgroupto46daysintheCCPTgroup(p=0.327)NR Patients:NR Han2013[31] US,UK,Australia,NewZealand,Primarycare #hospitals:NRMDPC(MacArthurFoundationDepressionandPrimaryCare) DepressionToolkitUptake:Sinceitslaunch,morethan20,000usersregisteredas memberstoreadanddownloaddepressioncareresources

(11)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient Canada,Ireland Depressioncare#providers:666 #patients:NR Post-only

Implementation:Slidepresentations,trainingmanuals, literaturereferences,costcalculator Provider:Patienthealthquestionnaire(PHQ-9),guidesto diagnosticandtreatmentapproaches,specialtycarereferral form Patient:Educationalmaterials Otherintervention:NR

Utility:Respondentsgenerallyratedresourcesasusefulbut rangedfrom68%(PHQ-9)to10%(costcalculator).Eight-four percentratedinformationonscreeninganddiagnosisas goodorexcellent(70%fortreatment,66%forpatient educationmaterials,and67%forcaremanagementmaterial). Sixty-threepercentwereveryconfidentinthecontent;more thanhalfindicatedthatexpertanswerstoclinicalquestions ondepression,expertdiscussionsresearchfindings,and contentoncommonmentalillnesseswouldimprovethetool Providers:60%ofrespondentsansweredtheyhadsomewhat changedtheirpracticeafterviewingthewebsiteresources Procedures:NR Patients:NR Gray2017[65] Canada Diabetescare

#hospitals:NR #providers:462(baseline),132(follow-up) #patients:NR Pre-post

BuildingCompetencyinDiabetesEducation:PhysicalActivity andExercise Implementation:seeFowles,2014 Provider:seeFowles,2014 Patient:seeFowles,2014 Otherintervention:Trainingworkshopsofferedinwhicheach componentofthemanualwaspresented,andparticipants weregiventimetopracticeinstructionsinmotivational interviewingandhowtoperformandinstructpatientsin resistanceexercises.

Uptake:NR Feasibility:Participantsfoundtheresourceshelpful(M5.1,SD 1.49).Frequencyanalysesofthe7specificresourcesprovided inthetoolkitrevealedthatbrochuresconcerningresistance trainingwerereferredtoonaregularbasisduringsessions withpatients(58%).Lessthanhalfreportedreferringregularly totheremaining6resources(informationalbrochures, counselingworksheets,decisiontree,at-a-glancesummary sheets,datacollectionsheets,resistancetrainingbrochures). Alargemajorityreportedthattheymostoftenusedthe resourceswhenworkingwithpatientswhowereinactivebut werereadyfororpreparingforphysicalactivity(80%). Reportedchallengesincluded:time(37%),patients’resistance (36%),patients’physicallimitations(27%),patients’personal barriers(23%),participants’lackofexpertise(16%),lackof physical-activityresources(12%),andother(24%).Analysisof the4-pointimpactrankingscalerevealedthattheimpactof thechallengeswasmoderate(M2.4,SD0.82). Providers:Immediatelyfollowingtheworkshop,particiapnts intendedtoimplementtheresourcesintheirsessionswith clients(M[median]4.5/5,SD0.64);93%reportedthatthey wouldverylikelyordefinitelyusetheresources.Participants intendedtoincreasetheprioritygiventophysicalactivity, employthetoolsprovidedinthetoolkit,incorporate resistancetrainingintopractice,andengageinnovelwaysto promotephysicalactivity.Themajorityreportedanincrease inconfidenceacrossphysicalactivitycounseling.Themost frequentlyreportedareaswereconfidenceinproviding informationandadviceregardingthebenefitsofactivity (86%);providingadviceandinstructionregardingresistance training(86%);andprovidingaphysicalactivityprogramthat accommodatedpatients’individualneedsorlimitations (78%).Sixty-fourpercentreportedanincreaseinconfidence intheirtypicalpatients’abilitiestoperformphysical-activity behaviorsappropriatefortheirfitnesslevels.Themajority

(12)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient indicatedthattheirconfidenceinmakingexercisereferrals hadnotchangedsincetheworkshoptraining. Procedures:Priortohavingattendedtheworkshop,58% includedphysical-activitycontentinmorethanhalfoftheir sessions,while29%addressedphysicalactivityin<25%of patientsessions.Inthesessionsinwhichphysicalactivitywas discussed,73%spent<25%ofthesessiononphysical activitycontent,and30%discussedphysicalactivityfor< 10%ofeachsession.Atthe8–12-monthfollowup,66% includedphysical-activitycontentinmorethanhalfoftheir sessions,whileonly18%addressedphysicalactivityin<25% ofsessions.However,inthesessionsinwhichphysical activitywasdiscussed,78%spent<25%ofthesessionon thistopic.8themesemerged:morefrequentlyand confidentlydiscussingphysicalactivityinsessions(27%); increasingfocusonresistancetraining(26%);providing patientswithphysical-activityproceduresandwritten information(14%);feelingbetterequippedtoassesscurrent physical-activitylevels(7%);assistingpatientsinworking aroundbarrierstobeinginvolvedinphysicalactivity(5%); recommendingspecificctivities(4%);encouragingother healthprofessionalstointegratephysicalactivityintopractice (4%)andother(12%). Patients:NR Fowles2014[44] Canada Diabetesmanagement

Communityhealthcenter #hospitals:7 #providers:NR #patients:198 CT

DiabetesBuildingCompetencyinDiabetesEducation: PhysicalActivityandExercise Implementation:Resourcemanual,literaturereview Provider:Guidelinesforriskstratification,assessmentsof readinessforexercise,referralprocessguide,clinicaldecision tree Patient:Sampleexerciseprograms,goalsettingworksheets, decisionalbalancesheets,informationalbrochures,online material,resistanceexercisevideo Otherintervention:3-htrainingand3-hregionalworkshop fordiabeteseducators

Uptake:NR Utility:NR Providers:Increasedconfidenceinabilitytoprovidephysical activityandexercisecounselingininterventiongroup (p<.001);greaterknowledgeaboutphysicalactivity(p<.03) butperceivedphysicalactivitycounselingtobemoredifficult afterreceivingthetrai Procedures:NR Patients:Nosignificantdifferenceinpatientreportedphysical activityandexerciselevels,efficacyperceptions,ormean glycatedhemoglobin;nodifferencesinrelativeuseof medicationorchangesinmedicationsordietbetween toolkitandstandardcaregroups Alidina2015[55] US Electivedelivery

Hospital #hospitals:1 #providers:NR #patients:1065 Pre-post EliminationofNon-medicallyIndicated(Elective)Deliveries Before39WeeksGestationalAge Implementation:Writtenmaterials,decisionsupportflow chart,schedulingflowchart,sampleschedulingform Provider:Patienteducationtalkingpoints,patienteducation poster Patient:Patienteducationbrochure,flyer Otherintervention:NR Uptake:Usedtoolkittoimplementnewschedulingprocesses. Feasibility:NR Providers:NR Procedures:Inthestudygrouptherewere4patientsversus 42patientsinthecontrolgroup(p<0.0001)delivered between37/0and38/6weekswithoutanindication. Patients:Therewere0transferstotheNICUcomparedto5 preintervention(p<.022)fornon-medicallyindicated deliveriesbetween37/0and38/6weeks. Chesis2015[58]HospitalandstaffobstetricianofficesEliminationofNon-medicallyIndicated(Elective)DeliveriesUptake:NR

(13)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient US Electivedelivery#hospitals:1 #providers:NR #patients:NR Pre-post Before39WeeksGestationalAge Implementation:seeAlidinaetal.,2015 Provider:seeAlidinaetal.,2015 Patient:seeAlidinaetal.,2015,flyer Otherintervention:Additionalmeasuringtooldevelopedby theAdvocateSystemObstetricSafetyCommitteetohelp driveappropriatepatientschedulinganddatacollection Feasibility:NR Providers:NR Procedures:Ratefornon-medicallyindicatedelectivedelivery was25.0%pre-implementationin2011;post-implementation bytheendof2011itwas11%.Inthe18monthspriorto publication,itwas0.0%. Patients:NR Clancy2012[13] US Emergencypreparedness

Hospital #hospitals:116 #providers:NR #patients:NR Post-only

NYStateDepartmentofHealthPediatricandObstetric EmergencyPreparednessToolkit Implementation:Trainingmaterialstratifiedbyhospitaltype, linkstoonlinecourses,educationalmaterialandclinical guidelines,glossary Provider:Safetychecklists,triagealgorithm,assessmenttools, dosageguidelines Patient:Factsheetforparents/caregivers Otherintervention:NR

Uptake:91%wereawareofthetoolkit,86%hadreviewedit Utility:Reasonsfornotappointingpediatricphysician/nurse coordinators:implementationnotstarted,noperson available,cost Providers:NR Procedures:1yearaftertoolkitdistribution,60%offacilities hadappointedapediatricphysiciancoordinator,49%a pediatricnursecoordinator.Toolkitreviewwasnotassociated withthepresenceofanemergencymanagementplan Patients:NR Wyte-Lake2016[89] US Emergeycypreparedness

Home-basedprimarycare #hospitals:NR #providers:77 #patients:NR Post-only

Home-BasedPrimaryCare/HomeHealthAgencyDisaster PreparednessToolkit Implementation:Writtenmaterials,sourcedocuments Provider:Checklists,suggestions,examplesoftools Patient:NR Otherintervention:IntroducedoveranationalcallforVHA HBPCprogramdirectors,postedtoVApulse,written invitationtoparticipateinweb-basedtoolkitevaluation, reminderemailsandletters

Uptake:NR Feasibility:Ofthoserespondentswhofoundthetoolkitvery helpful(forclarityofdesign,comprehensivenessof information,andoverallimpressionofthetoolkit) approximately60%hadbeenpartoftheHBPCprogramfor 5yearsorless.Thepercentageofrespondentswhoreported thetoolkittobehelpfuldecreasedaslengthoftimeinthe HBPCprogramincreased(22–25%for6–10yearsand15–18% for≥11years).Theseresultsindicatethathelpfulnessofthe toolkitwasassociatedwithfewernumberofyearswiththe HBPCprogram(p<0.05).LengthoftimeintheHBPC programmanagerrolewasnotfoundtobeassociatedwith perceivedhelpfulnessofthetoolkit.Ona4-pointLikertscale, respondentswereaskediftheyagreedordisagreedthatthe topicscoveredinthetoolkitwererelevanttotheir preparednessprotocol.Ofthosewhoimplementedtheir disasterpreparednessprotocolmorefrequently(3–5times/ yearor1–2times/year),two-thirds(66–67%)stronglyagreed thatthetopicscoveredinthetoolkitwererelevant. Conversely,ofthosewhoimplementedtheirprotocolvery infrequentlyornever,only23%stronglyagreedthatthe topicscoveredinthetoolkitwererelevanttotheirwork(p< 0.05).Whenasked,Howoftendoyouseeyourselfusingthis toolkit?,8%indicatedthattheywillneverusethetoolkit.The restindicatedthattheywouldusethetoolkitmoderatelyor extensively(datanotshown).HBPCprogramrepresentatives wereaskedtodescribethetypesofsupporttheywould needtoimplementthetoolkit.Theysuggestedspeakingwith otherswhohaveimplementedthetoolkit,sharingitwith leadershipandhospital-widecommittees,collaboratingwith

(14)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient locallawenforcementandreceivingonlinetraining, especiallydiscipline-specifictraining.Theyalsowould appreciateremindersaboutthetoolkit. Providers:NR Procedures:NR Patients:NR Henry2012[14] Mongolia Emergency/surgerycapacity

Hospital #hospitals:338(hospitalsin12provinces) #providers:NR #patients:NR Pre-post

IntegratedManagementofEmergencyandEssentialSurgical Care(IMEESC)toolkit Implementation:WHOrecommendationsforminimum standardstoimprovequalityandsafety,equipmentlistsand needsassessment,manual,trainer'sguide,detailedtraining material,modeltrainingworkshops,researchtool Provider:Bestpracticeprotocols,careguides,diagnostictools, safetychecklists,linktovideosonsurgicalprocedures Patient:NR Otherintervention:5-daytrainingprogramandsitevisits givenbyparticipantsinWHOTrainingofTrainersworkshop Uptake:67%ofprovincesand53%ofhospitalsimplemented theprogram Utility:NR Providers:NR Procedures:Increaseinnumberofsurgicalprocedures performed,decreaseinnumberofsurgicalprocedures referredtootherfacilities Patients:NR Cox2017[60] UK End-of-lifecare

Carehomes #hospitals:6 #providers:78(pre),103(post) #patients:NR Pre-post EndofLifeCareToolkit(partofCareHomesandhOspitals InnovatingCollaborativelytoincreaseEndoflifecareoptions [CHOICE]Project) Implementation:Writtenmaterials Provider:Writtenmaterials Patient:NR Otherintervention:Threetrainingsessions(1heach)ineach carehome

Uptake:NR Feasibility:NR Providers:Aftertheintervention,therewasatrendforstaffto reportfeelingmoresupportedintermsofemotionaland clinicalsupportinthecarehomeandfeelingabletosource externalsupportGP/districtnurse;Q11hospice/palliativecare nurse),evenoutofhours.Theresultssuggestedconfidence inabilitytodiscussdeathanddyingwithresidentswaslower post-intervention,althoughthischangeinconfidencedidnot reachstatisticalsignificance(p≥.05).Meanscoressuggest interventionsdidnotaffectstaffconfidenceintermsof discussingdeathanddyingwithrelatives,identifyingendof life,orthecreationofEoLCplans.Staffconfidencein managingeachofthe24endoflifesymptomsincluding pain,anxiety,nauseaandvomiting,andmouthcare increasedpost-intervention,however,thistrenddidnotreach statisticalsignificance. Procedures:NR Patients:Acomparisonofa5-monthperiodbeforeandafter theinterventionindicateda59%reductioninthenumberof residentswhodiedinthelocalNHShospitalincomparison toa21%reductionfromthecomparisoncarehomes. Carroll2012[16] US Fallprevention

Hospital #hospitals:8 #providers:NR #patients:364 RCT FallTIPS(TailoringInterventionsforPatientSafety) Implementation:NR Provider:Fallriskassessmentscale,individualizedbedposter, planofcare Patient:Peronalizedpatient/familyeducationhandout Otherintervention:NR Uptake:NR Utility:NR Providers:NR Procedures:Patientsontheinterventionunitsweremore likelytohavefallriskdocumented(89vs64%;p<.0001); thereweresignificantlymorecomprehensiveplansofcare forthepatientsontheinterventionsdocumentedbutno differencewerefoundregardingdocumentationof

(15)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient completedinterventions Patients:NR Coe2017[59] US Fallprevention

Clinicalandcommunity-basedorganizations #hospitals:23clinicaland27community-based organizations #providers:NR #patients:20317 Post-only

StoppingElderlyAccidents,Deaths,andInjuries(STEADI) Implementation:Storiesaboutfallspreventionprogramsand successes Provider:Tests,factsheets,casestudies,additionalresources Patient:Educationalmaterialsandbrochures Otherintervention:Statewidefallspreventionlearning collaborative(includesfulldaystatewidelearningsessions twiceperyear,expert-ledfallspreventionwebinarsare offeredseveraltimesperyear,ongoingtrainingsoneach interventioninmultiplemodalities),referralofqualified patientstocommunity-basedinterventions(TaiChi,Matterof Balance[MoB],andAssistedHomeSafetyAssessmentand Modification[AHSA]).

Uptake:Overaperiodof21monthsofimplementation clinicalsitesassessedpatientsusingtheSTEADIprotocoland referred4726individualstoPWTFcommunitysitesforfalls preventioninterventions.Ofthose,44%enrolledinthePWTF- sponsoredcommunityinterventionsandofthoseenrolled 45%completedtheinterventions.Organizationsalsorecruit individualsdirectly.Therewere2256“walk-ins”and989 “completers.”Overall,>4359individualsenrolled,and~1945 completedinterventions. Feasibility:StoppingElderlyAccidents,DeathsandInjuries implementationwaschallengingfortheprimarycaresitesas fallsriskassessmentwasanewareaandrequiressystems change.PWTFsitesexperiencedchallengessuchassecuring supportfromseniorleadershipandclinicalstaff;thelackof reimbursementforspecificclinicalcomponents;nodata fieldsinEMRtocaptureorassessfallsassessments;andlack ofworkflowsandprocessesforimplementingSTEADI.Clinical andcommunitystafffacedchallengesinreferringand enrollingindividualsintocommunityinterventionsdueto reluctanceduetothetimecommitment,lackof understandingofrisk,andunfamiliaritywithprogramsor organizationsrunningtheprograms.Partnershipstested multiplestrategiestoovercometheseissues. Providers:NR Procedures:Duringa9-monthperiod48%ofpatientswere screenedforfallsriskand30%ofthosewhoscreened positivereceivedanevaluationoftheirgait,strengthand balance.Ofthosewhoscreenedpositive,37%receivedaplan ofcareandamultifactorialclinicalriskassessment.Ofthe patientsscreened,6%receivedreferralstoacommunityfalls preventionintervention.Ofthosereferred,44%enrolledin thecommunityinterventions. Patients:NR Dykes2009[61];Zuyev2011 [109] US Fallprevention

Hospital #hospitals:4 #providers:NR #patients:685 Pre-post FallTIPS(TailoringInterventionsforPatientSafety)Toolkit Implementation:seeCarrolletal.(2012) Provider:seeCarrolletal.(2012) Patient:seeCarrolletal.(2012) Otherintervention:NR

Uptake:Adherencewithtoolkitadoptionmeasuresranges from72%(bedposter)to97%(fallriskassessment completed),varyingbysiteandcomponent Utility:Feedbackfromendusersispositive Providers:NR Procedures:Meannumberoffallriskassessmentscompleted perdayincreasedfrom1.7to2.0onemonthafter implementation(p<.003) Patients:Themeanfallratedecreasedfrom3.28to2.80falls per1000patient-dayspostinterventionandthemeanfall withinjuryratedecreasedfrom1.00to0.54per1000patient- days.

(16)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient Fisher2013[10] UK Fallprevention

Hospice #hospitals:1 #providers:NR #patients:NR Pre-post Fallspreventionandmanagementtoolkit Implementation:Policydocument(frameworkformulti- factorialassessmentanddefinitionoffall) Provider:Assessmentandcareplantool,incidentreportform Patient:NR Otherintervention:NR

Uptake:Therearehospicesthatcontinuetousethetoolkit, somehavemadesmallorsubstantialadaptationstomeet localrequirements Utility:NR Providers:NR Procedures:NR Patients:Theauthor'shospicereducedfallsto4.3per occupiedbedperyearin2011–2012from5.1in2007 Stalhandske2008[52] US Fallprevention

Hospital #hospitals:65outof70initialvolunteers #providers:42 #patients:NR Pre-post NationalFallsToolkit Implementation:CD,informationalbrochures,posters,flyers, samplebuttonsforidentifyingadvocatesorresources,online resources Provider:MorseFallScalepocketcard,videoonperforminga balanceassessments Patient:Videosonhipprotectorsforpatients/caregivers Otherintervention:Fallsdatamonitoring

Uptake:NR Utility:NR Providers:NR Procedures:Changedorimplementedafallriskassessment (14%),changedthesystemoftrackingfallsdata(12%),honed inonanareaofvulnerability(10%),changedorimplemented afallspolicy(7%),changedorimplementedafallsteam(7%), increaseduseofdocumentationorfallsprevention interventions(5%),becamemoreproactiveinfallsprevention (5%),otherwayssuchasimplementingspecificinterventions (21%) Patients:Overthecourseofthe2years,therewasa reductioninmajorinjuries(e.g.,64%inbehavioralhealth setting);fallrateremainedrelativelystable Ryan2013[30] Canada Geriatriccare

Primarycare #hospitals:181outof220invited #providers:NR #patients:NR Post-only

Geriatrics,InterprofessionalPractice,andInterorganizational Collaboration(GiiC)Toolkit Implementation:Topicoverviews,FAQs,materialsforbroader reading,interprofessionalandinterorganizational collaborationelements(eg,teamassessment,recognizing statesofteamdevelopment,teamproblemsolving, understandingorganizationaloutcomeexpectations,v Provider:Pocketguides,algorithmsandclinicaltools,self- directedlearningmaterialsdownloadabletopersonal computersorflashdrives Patient:Bilingualpatienthandouts Otherintervention:16-h“train-the-facilitator”workshops, small-groupdiscussionstosupportimplementation,2–4 monthscoachingfromconsultants,6monthsrefresherday andpracticereview,annualmeeting Uptake:79%ofidentifiedfamilyhealthteamsand85%of communityhealthcentersparticipatedintheinitiative;the toolkithasbeendownloaded41,556times;48%of participantsreportedmoderateoralotofchangeintheir team'scareoffrailseniors Utility:NR Providers:Participantratingsofknowledgegainand confidenceingeriatriccompetencieswas3.46(1notatall,5 agreatdeal).Participantratingsofperceivedlearningand facilitatorconfidencewere3.67,and4.15ininterprofessional and4.02 Procedures:NR Patients:NR Dore2013[34] US Healthliteracy

Rheumatologypractice #hospitals:NR #providers:18 #patients:NR Pre-post

HealthLiteracyUniversalPrecautionsToolkitfor Rheumatology(HLUPTK-R) Implementation:Materialonformingteams,raising awareness,organizationalassessment,Plan-Do-Study-Act worksheets,educationalmaterial,videos Provider:Quickstartguideforproviders,medicationdosing form,checklists,tipsforaddressinglanguagedifferencesand patienteducation Patient:Linkstoonlineresourcesforpatients Otherintervention:20-minintroductorypresentation Uptake:72%participantsstatedHLUPTK-RQickStart techniqueswereincorporatedintotheirpractice Utility:54%ofusersthoughtthatincorporatinghealthliteracy techniquesaddedtimetothepatient'svisit,butallthought thetimewasworthwhile;allbelievedthetechniqueswere helpfulintheirpractice Providers:77%ofusersagreedthattheirknowledgeofhealth literacywasimprovedandthatincorporatingthequickstart techniqueshadapositiveimpactonpatientcare Procedures:Alltoolkitusersusedtheencouragingquestions

(17)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient technique,62%usedtheteach-backmethod,23%usedthe medicationreviewtechnique Patients:NR Mabachi2016[75] US Healthliteracy

Primarycare #hospitals:12practices #providers:3individuals/site #patients:NR Post-only

HealthLiteracyUniversalPrecautions(HLUP)Toolkit Implementation:NR Provider:NR Patient:NR Otherintervention:Practicesidentifiedandrankedthetop fourof11prioritytoolstheywantedtoimplement;practices wereassignedtoimplementtwotools.Technicalassistance providersfromtheresearchteamsconductedchec-incallsat 2,4,8,and16weeks.

Uptake:PracticesusedtheToolkit“flexibly.”Theydidnot alwaysimplementallportionsoftheirassignedtools. Practicesfoundthatsometoolswerebestimplementedin tandemandrecognizedtheefficienciesinimplementing theminthismanner. Feasibility:Specificimplementationbarrierswerenotedby theparticipatingpractices,including(1)competing demands/staffcapacity,(2)bureaucraticchallenges,(3) technologicalchallenges,(4)limitedqualityimprovement experience,and(5)limitedsupportfromleadership.Linking healthliteracyimplementationactivitiestootherpractice- wideQIinitiatives(e.g.,patient-centeredmedicalhome accreditation)raisedstaffawarenessandincreased engagement. Providers:Whilepracticesworkedindependentlyontoolkit implementation,theybenefitedfromhavingexternalsupport andaccountability.Three-fourthofthepracticesreported thattheyplantocontinuetousetheToolkitasaresourceto guidetheirhealthliteracy-relatedQIwork.8/12practices indicatedtheywouldcontinuewiththeir2assignedtools andtheimprovementimplementedduringthestudyperiod. Procedures:NR Patients:NR Koelling2006[28] US Heartfailure

Hospital #hospitals:14 #providers:NR #patients:1806 CT GuidelinesAppliedinPractice-HeartFailure(GAP-HF)Tool Kit Implementation:Qualityperformancecharts Provider:Heartfailurestandardadmissionorders,heartfailure specificclinicalpathway,heartfailurepatientdischarge contract Patient:Self-managementdiary Otherintervention:6monthlylearningsessions Uptake:NR Utility:NR Providers:NR Procedures:NR Patients:Baseline-adjusted30-dayreadmissionratewas statisticallyreduced(p=.003)butnot30-daymortality (p=.101)comparinginterventionandcontrolgroup Perumalswami2016[80] US HepatitisCcare

Primarycare #hospitals:NR #providers:NR #patients:NR Post-only

HepCure(HepatitisCeducationandpatientengagement) Implementation:NR Provider:Openaccesstoolkit(adashboard)thatenhances providers’abilitytodeliverguideline-basedHCVcare;anda tele-educationplatformformedicalproviders Patient:Linkedpatientappthatprovideseducation, medicationreminders,andaplatformfortrackingadherence andsymptoms Otherintervention:None Uptake:Weeklytele-educationsessionshavebeenconducted 57timessinceFebruary2015,with322uniqueattendeesand anaverageof22(plusorminus9)attendeesaweek.Five hundredforty-sixdownloadsofpatientappfromNovember 2014toMay2016. Feasibility:NR Providers:NR Procedures:NR Patients:NR Adams2014[45] US Hospitalreadmission

Hospital #hospitals:1 #providers:NR ProjectRe-EngineeredDischarge(ProjectRED)Toolkit Implementation:Detailedimplementationguide,workbooks Provider:AfterHospitalCarePlanforms,post-discharge Uptake:NR Utility:NR Providers:NR

(18)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient #patients:336 Pre-postfollow-upphonecallscript Patient:Booklet,checklistforappointments Otherintervention:Patientdischargequestionnaire,contact sheets(inEnglishandSpanish)

Procedures:94–95%ofpatientsreportedhavingreceived writteninformationregardingmedications,theircondition, andwhentoseekmedicalattention Patients:Re-admissionswerereducedby32%;99–100%of patientsreportedknowingtheirmedicationregimentand whentocallthedoctororseekemergencycarepost- intervention Mitchell20015[78] US Hospitalreadmission

Hospital #hospitals:10 #providers:NR #patients:NR Pre-post ProjectRe-EngineeredDischarge(ProjectRED)Toolkit Implementation:Detailedimplementationguide,workbooks Provider:AfterHospitalCarePlanforms,post-discharge follow-upphonecallscript Patient:Booklet,checklistforappointments Otherintervention:8-hourtraining,monthlytelephone-based technicalassistancecallsfor1year

Uptake:7/10successfullyimplementedtheREDprogramas planned.EighthospitalschosetoinitiateREDimplementation in1or2unitsorwardsand/orforpatientswithaparticular diagnosis(i.e.,CHFpatientsonly). Feasibility:The7implementinghospitalshadthefollowing commonfeatures:highlyvisiblecommitmentfromsenior leadership,empoweredinterprofessionalimplementation team,establishedmethodsforsharingresultsandassessing accountability,buy-infromstaffandstakeholders,andflexible in-houseITsupport.Nineoftheparticipatinghospitals implementedasite-specificadaptationoftheREDprotocol duringthestudyperiod.All10hospitalsrevisedthe2-day postdischargetelephonecallscriptincludedintheRED Toolkit. Providers:Membersoftheimplementationteamsbelieved thattheREDprocessesenhancedpatientcare,providedtools tohelppatientsbettermanagetheirmedicalconditions,and hadanimportantimpactonjobsatisfaction,staffmorale,and engagement. Procedures:Fourhospitalsdidnothirepersonneltoperform REDresponsibilitiesandinsteadusedunitnursesfor dischargeeducation;twoteamsusednonclinicalpersonnel orthird-partyvendorstoconductthe2-daypost-discharge phonecall. Patients:All7hospitalsreportedmodestreductionsin30-day readmissionsforatleastoneofthe3diagnosticareas targetedbyCMSforpaymentpenalties(congestiveheart failure[CHF],acutemyocardialinfarction[AMI],and pneumonia[PNA]).Allbut2hospitalsreporteda0.5%or greaterreductionin30-dayall-causereadmissionsafterthe implementation.5hospitalsachievedagreaternetdecrease inreadmissionratesthanthenationalaverageforCHF patients,4surpassedthenationalaveragedecreaseforAMI readmissions,and5exceededthenationalaveragedecrease forPNAreadmissions. Enfield2014[40] US Hospital-acquiredinfections

Hospital #hospitals:1 #providers:NR #patients:NR Pre-post CenterforDiseaseControlandPrevention'sCarbapenem- resistantEnterobacteriaceae(CRE)Toolkit Implementation:Researchoverview,materialtoraise awareness,implementationstrategies(e.g.,regionalapproach toCRE),organizationalriskassessmenttool,references Uptake:Allcomponentsimplemented,butsomewere alreadyinplacebeforetheintervention Utility:NR Providers:NR Procedures:Compliancewithhandhygieneincreasedfrom

(19)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient Provider:Laboratorystandardschart,carealgorithm Patient:NR Otherintervention:Limitedaccesstoroomsandcommon areas,terminallycleanroompolicyenhanced,monitoringof environmentalcleaningandfeedback

71%beforeto86%after(p<0.001);surfacetestingrevealed 39%oftesteditemsexceedingthresholdlevelsbeforeand 12%after(with6%and14%insubsequentperiods,showing asustainedeffect) Patients:Before,incidencerateofCREwas7.77cases/1000 patient-days,after1.22cases/1000patient-days(p=.001);rate ofextensivelydrug-resistantAcinetobacterbaumannii(XDR- BA)was6.79cases/1000patient-daysbefore,afternocases wereidentified Randle2006[26] UK Hospital-acquiredinfections

Hospital #hospitals:6 #providers:127 #patients:43 Pre-post Clean-Your-HandsCampaignToolkit Implementation:Educationalmaterial,postersforstaffand patients,marketingmaterials(aprons,badges,etc.),evaluation tool Provider:Alcoholhandrubs Patient:Leaflet,posters Otherintervention:Campaignprojectmanagertrainedpilot teamsinevaluationtooluse

Uptake:NR Utility:Respondentsindicatedtheframeworkfor implementationwashelpfulandthemajorsuccesselement werethealcoholhandrubs Providers:70%ofnursesand60%ofdoctorsagreedthatthe presenceofwipesencouragedthemtocleantheirhands, 76–84%indicatedpostersmadestaffthinkabouttheirhand hygiene,74%indicatedtheycleanedtheirhandsmore frequently Procedures:Compliancewithhandcleaningincreasedfrom 32%(beforetoolkit)to41%at3monthsand63%at6 months Patients:16/43patientshadaskedstafftocleantheirhands, 35/43foundpostersetc.useful Septimus2016[84];Huang 2013[91] US Hospital-acquiredinfections

Hospital #hospitals:136ICUsin95hospitals #providers:NR #patients:305583admissions(pre-period)/102220 (post) Pre-post UniversalICUDecolonizationToolkit:AnEnhancedProtocol Implementation:Protocoloverview,scientificrationale,flow chart,readinessassessmentandFAQ,trainingand educationalmaterials,chlorohexanebathingskillsassessment Provider:Nursingprotocol,safetyinformation Patient:NR Otherintervention:Fivecoachingcalls

Uptake:NR Feasibility:Challengesidentifiedincludedconcernsabout mupirocinresistanceandquestionsaboutpeerreviewofthe originaltrialresults,Mostfacilitieswereabletoeasily implementdailyCHGbathing,asthispractice,fitwithin normalnursingworkflowanddidnotrequireaphysician order. Providers:NR Procedures:NR Patients:TherawCLABSIrate(CLABSIeventsdividedby numberofcentralline–day)droppedfrom1.1/1000to0.87/ 1000centralline–dayspostintervention.Therewere672 CLABSIsper587891centralline–daysinthe24-month preinterventionperiod,and181CLABSIsper208175central line–daysinthe8-monthpost-interventionperiod.After implementation,therateofCLABSIdecreasedby23.5%(CI 9.8–35.1%;p=.001). Kuhlman2014[47] US Infantsafesleep

Pediatricandobstetricclinics #hospitals:2 #providers:NR #patients:309 Post-only SafeSleepToolkit Implementation:NR Provider:Briefhealthcareproviderscript Patient:Parentalchecklist,nationallyavailableresources(links tovideos,posters,brochures,doorhangers) Otherintervention:NR Uptake:NR Utility:NR Providers:NR Procedures:Providersengagedindiscussionregardingsafe sleepwithmostparentswhoreportedintentions/behaviorin oppositiontotherecommendationsforsafesleep Patients:NR

(20)

Toolkitnameandcomponents OtherinterventionevaluatedinthestudyResults Uptake Feasibility Provider Procedures Patient Haley2015[68] US Kidneydiseasecare

Primarycareandnephrologypractices #hospitals:9primarycareand5nephrology practices #providers:25pre,24postinterviews #patients:292chartaudits Pre-post

AdvancedChronicKidneyDisease(CKD)PatientManagement Toolkit Implementation:Guidetotoolselection,patientidentification tools,patientmanagementtools,physicianeducation materials.slidepresentation,clinicalpracticeguidelines, awarenessletter, Provider:Identificationandactionplancard,identification andactionplanposter,glomerularfiltrationrate(GFR) calculator,CKDchartflags/stickers,referringphysicianfaxback form,CKDpost-consultletters,advancedCKDmanagement flowsheetandalgorithm Patient:CKDpatientdiary,CKDpatienteducationresources, venipunctureremindercard,vascularaccesspassport Otherintervention:Educationsessionsateachsite,made modificationstosometoolsandoneadditionaltoolcreated (CKDScreeningProtocol/WhentoRefer)

Uptake:Sitechampionsandphysicianleaderswerecontacted 3yearsaftercompletionofthestudyandaskedwhetherthe improvementsinawarenessofCKD,communication,the referralprocess,andcomanagementofpatientswithCKD hadbeensustainedandwhethertoolswerestillinuse.Of thosewhohadremainedactiveintheirrespectivepractices overthattimeframe,5respondedwith14of15answers beingaffirmative. Feasibility:Increasedcommunicationbetweenpracticeswas associatedwithenhancedsatisfactionscores.OnaLikert scale,satisfactionwithcomanagementreportedby nephrologistsimprovedfrom2.6to4.3.Corresponding satisfactionlevelsofPCPswere4.3and4.7.Ofthe16 respondentswithpreimplementationlevelslessthan “satisfied,”15notedimprovement,with6improvingfrom “somewhatunsatisfied”to“satisfied”or“verysatisfied.” Providers:Preimplementation,fewpracticesreported familiaritywithCKDclinicalpracticeguidelines,andCKD screeningwaslimitedmostlytodiabeticpatients. Postimplementation,allpracticesreportedincreased awarenessofriskfactorsforkidneydisease.Pre- implementation,fewusedspecifictriggersfornephrology referrals,althoughseveralcitedcreatininelevel.Then,2.0mg/ dLorwhendialysisquestionsarose.Postimplementation transcriptsrevealedincreasedconsistencyofreferraltiming, withpracticesprovidingmorevigilantmonitoringofhigh-risk patients:managingCKDuptostage3andallreporting referralbystage4.Severalnephrologistsandtheirsite championsnotedtheneedfortimelynephrology appointments.Postinterventionnephrologyinterviews revealedheightenedattentiontocommunicationand comanagement.Practicesreportedtheprojectalteredthe contentofnephrologypostconsultletters,advanced comanagementgoals,andimprovedteamworkamongoffice staff. Procedures:Attheoutset,careprocessesandmechanisms variedamongpractices.Postimplementationimprovement wasobservedforCKDidentification,referral,and communicationandexecutionofcomanagementplans.166/ 171tasksand124/144subtaskswereinplace postimplementation,comparedwith78and51pre- implementation.Nephrologypracticeslikewiseimproved postimplementation,particularlyintheirreferralprocesses andcommunication.Postinterventionquestionnaires confirmedthatpatientswithCKDwerebeingreferred earlier—nonelaterthanstage4.Analysisofauditsrevealed improvementinGFRdocumentation(p=.01);mostperformed wellwithrespecttoorderingcreatininelevelswithin1year