A quantitative approach for integrating multiple lines of evidence for the evaluation of environmental health risks

Submitted23 July 2014 Accepted 29 December 2014 Published15 January 2015 Corresponding author Robert K.D. Peterson, bpeterson@montana.edu

Academic editor Thomas Hartung

Additional Information and Declarations can be found on page 9

DOI10.7717/peerj.730 Copyright 2015 Schleier III et al.

Distributed under

Creative Commons CC-BY 4.0 OPEN ACCESS

A quantitative approach for integrating

multiple lines of evidence for the

evaluation of environmental health risks

Robert K.D. Peterson

Department of Land Resources and Environmental Sciences, Montana State University, Bozeman, MT, USA

ABSTRACT

Decision analysis often considers multiple lines of evidence during the decision mak-ing process. Researchers and government agencies have advocated for quantitative weight-of-evidence approaches in which multiple lines of evidence can be considered when estimating risk. Therefore, we utilized Bayesian Markov Chain Monte Carlo to integrate several human-health risk assessment, biomonitoring, and epidemiology studies that have been conducted for two common insecticides (malathion and permethrin) used for adult mosquito management to generate an overall estimate of risk quotient (RQ). The utility of the Bayesian inference for risk management is that the estimated risk represents a probability distribution from which the probability of exceeding a threshold can be estimated. The mean RQs after all studies were incorpo-rated were 0.4386, with a variance of 0.0163 for malathion and 0.3281 with a variance of 0.0083 for permethrin. After taking into account all of the evidence available on the risks of ULV insecticides, the probability that malathion or permethrin would exceed a level of concern was less than 0.0001. Bayesian estimates can substantially improve decisions by allowing decision makers to estimate the probability that a risk will exceed a level of concern by considering seemingly disparate lines of evidence.

Subjects Agricultural Science, Entomology, Environmental Sciences, Toxicology Keywords Decision analysis, Uncertainty analysis, Mosquito management, Pesticide, Bayesian Markov Chain Monte Carlo, Risk assessment

INTRODUCTION

the risk-management options, a quantitative treatment of uncertainty and variability is needed to discriminate between management options to make informed decisions (National Research Council, 1994;National Research Council, 1996;National Research Council, 2009).

When making decisions regarding risk, there are often multiple lines of evidence that need to be considered. Information often is generated and gathered from different sources, so risk analysts and managers are confronted with the issue of combining data from these sources to improve the decision-making process. However, the ability of people to make precise and significant statements about risks diminishes with increasing amounts of information and complexity (Zadeh, 1965). The incorporation of multiple lines of evidence into a weight-of-evidence framework allows risk assessors and managers to generate a single estimate of the risk (Dale et al., 2008). Currently, the most common way to incorporate dissimilar lines of evidence is by determining the weight-of-evidence estimate through qualitative risk assessments or through listing evidence (Chapman, McDonald & Lawrence, 2002;Hull & Swanson, 2006;Linkov et al., 2009;Menzie et al., 1996;Sanchez-Bayo, Baskaran & Kennedy, 2002;Suter II & Cormier, 2011;United States Environmental Protection Agency, 2005a;Weed, 2005), which can have fundamental mathematical limitations compared to quantitative estimates (Cox Jr, Babayev & Huber, 2005). These methods are important contributions to the decision making process, but they do not provide a comprehensive and structured approach for integrating multiple lines of evidence from different study types (Linkov et al., 2009).

Rather than testing for a specific relationship (e.g., the probability of obtaining values as extreme or more extreme than the values observed in the study), decision makers may ultimately be interested in making inferential conclusions about environmental health risks (Assmuth & Hilden, 2008;Ellison, 1996;Hill, 1996). Bayesian inference can address inferential conclusions by providing a framework, based on probability calculus, by quantifying the uncertainty in parameter estimates and determining the probability that an explicit endpoint is exceeded given a set of data (Ellison, 1996;Hill, 1996). Bayesian inference is a way of updating prior knowledge given new information becoming available to generate a posterior estimate of the parameters of interest (i.e., risk) (Ellison, 1996).

Currently there are few quantitative frameworks that integrate data into a framework that can be utilized by risk managers (Assmuth & Hilden, 2008). A quantitative framework for integrating and interpreting multiple lines of seemingly disparate evidence into an overall risk estimate is critically needed for complex risk assessments (Dale et al., 2008).

multiple lines of evidence into an overall estimate. Similar approaches have been used for different applications in risk assessment, toxicology, and environmental modeling, but they have not been utilized to update the risk estimates for anthropogenic chemical stressors as new information becomes available (Bernillon & Bois, 2000;Brand & Small, 1995;Devine & Qualters, 2008;Schenker et al., 2009;Taylor, Evans & McKone, 1993).

There are many advantages of using Bayesian techniques for weighing evidence, including full allowance for all parameter uncertainty in the model, the ability to include other pertinent information that would otherwise be excluded, and the ability to extend the models to accommodate more complex models (Hill, 1996;Sutton & Abrams, 2001). Studies utilizing Bayesian approaches have considered separate studies with the same study type to estimate an overall value for the parameter of interest (Smith, Lipkovich & Ye, 2002;Wheeler & Bailer, 2009). Therefore, to address the need for a quantitative approach for environmental health, we utilized Bayesian Markov Chain Monte Carlo (MCMC) to provide a logical and consistent method for estimating the risk of chemicals when multiple studies are available. To demonstrate how Bayesian statistics can be used for decisions regarding environmental and public health risks, we chose insecticides used for adult mosquito management as our case study.

CASE STUDY

To effectively manage infection rates, morbidity, and mortality due to mosquito-borne pathogens, there must be a reduction in contact between infected mosquitoes and humans and animals (Marfin & Gubler, 2001). One of the more effective ways of managing high densities of adult mosquitoes that vector human and animal pathogens is ultra-low-volume (ULV) aerosol applications of insecticides. Since West Nile virus (WNV) was introduced into the U.S., more areas of the country have been experiencing large-scale insecticide applications. Consequently, there has been greater public attention on human-health and environmental risks associated with ULV insecticide applications (Peterson, Macedo & Davis, 2006;Reisen & Brault, 2007;Roche, 2002;Thier, 2001).

A decade after the initial response to WNV, several quantitative human-health and ecological risk assessments have been conducted to estimate the magnitude of risks associated with the insecticides (Davis, 2007;Davis, Peterson & Macedo, 2007;Gosselin et al., 2008;Macedo, Peterson & Davis, 2007;New York City Department of Health, 2005;

Table 1 Risk quotient estimates for each study.

Malathion Permethrin

Karpati et al. (2004)c NAa 0.99b

United States Environmental Protection Agency (2005c)dand United States Environmental Protection Agency (2005d)d

0.018 0.025

Currier et al. (2005)e NAa 0.99b

O’Sullivan et al. (2005)c 0.99b NAa

Peterson, Macedo & Davis (2006)d 0.0076 0.0021

Suffolk County (2006)d 0.015 0.013

Macedo, Peterson & Davis (2007)d NAa 0.023

Valcke, Gosselin & Belleville (2008)d 0.64 NAa

Schleier III (2008)d NAa 0.00025

Schleier III et al. (2009a)d 0.02 NAa

Schleier III et al. (2009b)d 0.0017 0.000068

Notes.

a_{Not applicable because the chemical was not assessed.}

b_{A risk quotient of 0.99 was used because it provides a conservative estimate of the risk for biomonitoring and}

epidemiology studies and due to a lack of knowledge about the true value, which must be below 1 if no effect is seen.

c_{Epidemiological study.} d_{Risk assessment.} e_{Biomonitoring study.}

combined the results from risk assessment, epidemiology, and biomonitoring studies, and their seemingly disparate data metrics, to obtain an overall estimate of the risk.

DATA AND METHODS

In environmental and human health risk assessments of pesticides, risk quotients (RQ) are often used to quantitatively express risk (Peterson, 2006). Risk quotients are calculated by dividing the potential exposure (PE) by its respective toxic endpoint value. Estimated RQs are compared to a RQ level of concern (LOC) or other threshold which is set by the USEPA or another regulatory agency to determine if regulatory action is needed. The RQ LOC used in our assessment was 1.0. An RQ>1.0 means that the estimated exposure is greater than the relevant toxicological endpoint. If an RQ breaches a regulatory LOC (RQ≥1) at a lower tier, then risk managers decide to restrict the product use, progress to higher tier risk assessments, or use field-verified models (United States Environmental Protection Agency, 2006d).

found that contained mosquito ULV risk assessments, biomonitoring, or epidemiological measurements for permethrin or malathion were included in this assessment.

The estimated RQs for each study are summarized inTable 1for each chemical. The same toxicological endpoints were used for all of the risk assessments, which are based on the U.S. EPA’s ingestion reference dose for mammals (Macedo, Peterson & Davis, 2007;

Peterson, Macedo & Davis, 2006;Schleier III, 2008;Schleier III et al., 2009a;Schleier III et al., 2009b;Valcke, Gosselin & Belleville, 2008), and in the case of probabilistic risk assessments we used the 95th percentile RQ for conservatism.

The literature search found two epidemiological studies and one biomonitoring study for permethrin and malathion.Karpati et al. (2004)analyzed hospital admissions for asthma in New York, NY three days before and after ground based ULV applications of permethrin (n₌510 before spraying and 501 after spraying) and found no increase in admissions for asthma.Currier et al. (2005)analyzed urine samples for metabolites of permethrin in 125 persons in the treated area and 67 persons from two control areas after ground-based ULV applications in Mississippi and found no increase in urinary metabolites. The persons selected in the study were geographically random and were verified by mapping the GPS location of the ground-based applications.O’Sullivan et al. (2005)analyzed hospital admissions for asthma in New York, NY after ground-based ULV applications of malathion in September of 1999, and compared those to September 1997 and 1998 when no malathion treatments occurred (n=1,318 patients presented with a diagnosis of asthma exacerbation). They found no statistical difference between the 1999 asthma admissions and the asthma admission in 1997 and 1998. To incorporate the epidemiology and biomonitoring studies, we assumed that if the researchers did not observe an effect or increase in urinary metabolites of the pesticide, the RQ was assumed to be 0.99 (Table 1). We assumed a RQ of 0.99 to be conservative because of a lack of knowledge on the value, which must be below 1.0 if no effect is observed.

Bayesian inference treats statistical parameters as random variables, and uses a likelihood function to express the plausibility of obtaining different values of the parameter when the data have been observed (Ellison, 1996). To define a RQ for adult males we used Bayes’ theorem:

p(θ_|y)₌p(y_|θ )p(θ ) (1)

wherepis the probability mass,θ is the value of a random variable selected from the prior distribution,yis the evidence being considered,p(θ )is the prior probability,p(y_|θ )is the likelihood function for the evidence (Congdon, 2006;Gelman et al., 2004). We assumed a normal distribution for the likelihood function and used log-transformed risk quotients fromTable 1. The central limit theorem of classical statistics and the Bayesian analog justify the normal density as an approximation for the posterior distribution of many summary statistics, even when they are derived from non-normal data (Congdon, 2006). To estimate the posterior density,

p

y_|θ_=√1

2π σexp− 1

2σ2(y−θ )

Figure 1 Posterior probability distributions for malathion with all available studies and all studies excluding epidemiological and biomonitoring.

whereyis a single scalar observation from the RQ’s inTable 1from a normal distribution parameterized by a mean ofθand a variance ofσ2(Gelman et al., 2004).

We have no knowledge of the prior distribution, so we assumed an uninformative or diffuse prior which we defined as a normal distribution with aµ₀of 1 and aτ₀2of 1. We chose an uninformative prior because the effect of the prior and data on the updated beliefs depends on the precision of the density ofp(θ )(Congdon, 2008). We used MCMC simulation utilizing the Metropolis-Hasting algorithm to obtain the posterior distribution forEq. (2)using Matlab®R2010b (MathWorks, Natick, MA, USA). We sampled the purposed posterior distributions usingEq. (2)by iterating 100,000 purposed values for the posterior distribution and discarded the first 1,000 samples for burn in.

RESULTS AND DISCUSSION

The mean posterior RQs after all studies were incorporated were 0.4386 with a variance of 0.0163 for malathion and 0.3281 with a variance of 0.0083 for permethrin (Figs. 1and2). The mean posterior RQs for all studies excluding the epidemiological and biomonitoring studies slightly decreased the mean to 0.4119 with a variance of 0.0158 for malathion and a mean of 0.302 with a variance of 0.0081 for permethrin (Figs. 1and2). Using the posterior mean and variance from the normal distribution, the probability that exposure to malathion or permethrin after ULV applications would exceed a level of concern was less than 0.0001, regardless of whether all of the studies were incorporated or the epidemiological and biomonitoring studies were held out (Figs. 1and2).

Figure 2 Posterior probability distributions for permethrin with all available studies and all studies excluding epidemiological and biomonitoring.

(Figs. 1and2). The probability of exceeding a threshold is most likely more intuitive for risk managers and the public to understand than an estimate of the 95th percentile of exposure or risk, which is typically reported in probabilistic risk assessments (Hill, 1996). In fact, risk can be defined as the probability and severity of adverse effects (Aven & Renn, 2009), which Bayesian statistics directly addresses. The majority of weight-of-evidence studies do not quantify both a risk estimate and variability or uncertainty around that estimate, but Bayesian MCMC methods quantify both (Linkov et al., 2009).

The USEPA provides guidance on how to perform risk assessments that address variability and uncertainty (National Research Council, 2009;United States Environmental Protection Agency, 1989;United States Environmental Protection Agency, 2004), but they do not provide a simple method for integrating multiple lines of evidence. Our case study directly addresses the need for a standard approach by which multiple lines of evidence can be interpreted in a framework that ecologists, risk assessors and managers, and NRC have highlighted (Dale et al., 2008;Linkov et al., 2009;National Research Council, 1994;National Research Council, 1996;National Research Council, 2009). Our method also could be utilized by the Network Reference Laboratories for Monitoring of Emerging Environmental Pollutants in the European Union for integrating risk assessments and biomonitoring to prioritize pollutants (Tilghman et al., 2009).

assessments could provide a refined RQ estimate if the amount of chemical the person is exposed to is calculated. Bayesian inference can also incorporate expert knowledge of a system which can be used as prior information that is updated by data (Gargoum, 2001;

Morris, 1977).

In ecotoxicology and other disciplines, there are multiple estimates of values like the lethal concentration that kills 50% of a population (LC50) (Wheeler & Bailer, 2009). This technique could be used to estimate an overall LC50 for use in risk assessments or setting total maximum daily load limits.Stauffer (2008)showed that in natural resource management there are often multiple estimations for a population of interest. Therefore, Bayesian MCMC methods can be used to estimate the probability of the population being above or below a given threshold.

Bayesian analysis provides a systematic approach for guiding the decision-making process by incorporating new knowledge in the estimate of risk, which directly addresses NRC recommendations (National Research Council, 1994;National Research Council, 2009). However, Bayesian inference does not address the uncertainties inherent in each risk assessment. For example, there is large uncertainty surrounding the estimate of insecticide air concentrations and deposition on surfaces after ULV applications for adult mosquito management (Schleier III et al., 2009a;Schleier III et al., 2009b). Models used by the USEPA and other researchers to estimate concentrations are either over- or under-estimating depending on the model (Schleier III & Peterson, 2010;Schleier III et al., 2008b). In addition, probabilistic risk assessments demonstrated that the estimated air concentration and deposition of insecticides are contributing the largest amount of variance to the potential exposure (Schleier III et al., 2009a;Schleier III et al., 2009b). However, the estimate presented here most likely is robust against these uncertainties because the studies used a variety of models, exposure pathways, and monitoring techniques which were not dependent on a standardized assessment protocol.

We recognize that the assumptions about RQ distributions may affect the final results; however, we attempted to reduce the potential biases by making conservative assumptions erring on the side of safety, which is common practice in risk assessment. In addition, probability distributions other than normal can be utilized if enough is known about the underlying distribution of the population, like those used for toxicological studies. Bayesian MCMC also can be utilized with the current data and the incorporation of expert judgments to aid in the determination of risk estimates (Grist et al., 2005).

Bayesian analysis techniques have been underutilized with respect to environmental and public health, risk assessment, ecology, and environmental sciences (Clark, 2005). Our method is a quantitative approach to statistically derive risk estimates from multiple lines of evidence, which is a relatively simple way of integrating multiple lines of evidence into a framework that can be used by assessors and managers (Assmuth & Hilden, 2008;

derive a distribution for the toxicological endpoints like LC50 or no-effect concentration when multiple values are available for the same species. Future refinements to our Bayesian model would be the development of a method to convert epidemiological study results into a RQ to reduce the uncertainty and conservatism. In addition, biomonitoring studies can quantify the exposure (if exposures are above background levels) and convert those estimates to RQ.

ACKNOWLEDGEMENT

We thank K Irvine (U.S. Geological Survey) for reviewing an earlier version of the manuscript.

ADDITIONAL INFORMATION AND DECLARATIONS

Funding

This research was supported by grants and fellowships from the USDA Western Regional IPM grant program (2009-34103-20034), Montana State University Institute on Ecosystems National Science Foundation Final Year Ph.D. Fellowship, the U.S. Armed Forces Pest Management Board’s Deployed War Fighter Protection Research Program (W911QY-11-1-0005), and by the Montana Agricultural Experiment Station, Montana State University, Bozeman, Montana, USA. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Grant Disclosures

The following grant information was disclosed by the authors: USDA Western Regional IPM grant program: 2009-34103-20034.

Montana State University Institute on Ecosystems National Science Foundation.

U.S. Armed Forces Pest Management Board’s Deployed War Fighter Protection Research Program: W911QY-11-1-0005.

Montana Agricultural Experiment Station, Montana State University. Competing Interests

Ryan S. Davis is an employee of Electrical Consultants, Inc.

Author Contributions

• Jerome J. Schleier III conceived and designed the experiments, performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, wrote the paper, prepared figures and/or tables, reviewed drafts of the paper.

• Lucy A. Marshall analyzed the data, contributed reagents/materials/analysis tools, wrote the paper, reviewed drafts of the paper.

• Ryan S. Davis conceived and designed the experiments, contributed reagents/materials/analysis tools, reviewed drafts of the paper.

REFERENCES

Ascough JC, Maier HR, Ravalico JK, Strudley MW. 2008.Future research challenges for incorporation of uncertainty in environmental and ecological decision-making.Ecological Modelling219:383–399DOI 10.1016/j.ecolmodel.2008.07.015.

Assmuth T, Hilden M. 2008. The significance of information frameworks in integrated risk assessment and management. Environmental Science and Policy 11:71–86 DOI 10.1016/j.envsci.2007.07.006.

Aven T, Renn O. 2009.On risk defined as an event where the outcome is uncertain.Journal of Risk Research12:1–11DOI 10.1080/13669870802488883.

Bernillon P, Bois FY. 2000.Statistical issues in toxicokinetic modeling: a Bayesian perspective.

Environmental Health Perspectives108:883–893DOI 10.1289/ehp.00108s5883.

Brand KP, Small MJ. 1995.Updating uncertainty in an integrated risk assessment: conceptual framework and methods.Risk Analysis15:719–731DOI 10.1111/j.1539-6924.1995.tb01344.x. Chapman PM, McDonald BG, Lawrence GS. 2002.Weight-of-evidence issues and frameworks for

sediment quality (and other) assessments.Human and Ecological Risk Assessment8:1489–1515 DOI 10.1080/20028091057457.

Clark JS. 2005.Why environmental scientists are becoming Bayesians.Ecology Letters8:2–14 DOI 10.1111/j.1461-0248.2004.00702.x.

Congdon P. 2006.Bayesian statistical modelling. West Sussex: John Wiley and Sons, Ltd.

Congdon P. 2008.Bayesian statistics in quantitative risk assessment. In: Everitt B, Melnick E, eds.

Encyclopedia of quantitative risk analysis and assessment. Hoboken, NJ: John Wiley and Sons Inc., 119–136.

Cox Jr LA, Babayev D, Huber W. 2005.Some limitations of qualitative risk rating systems.Risk Analysis25:651–662DOI 10.1111/j.1539-6924.2005.00615.x.

Currier M, McNeill M, Campbell D, Newton N, Marr JS, Perry E, Berg SW, Barr DB, Luber GE, Kieszak SM, Rogers HS, Backer SC, Belson MG, Rubin C, Azziz-Baumgartner E, Duprey ZH. 2005.Human exposure to mosquito-control pesticides—Mississippi, North Carolina, and Virginia, 2002 and 2003.MMWR Morbidity and Mortality Weekly Report54:529–532. Dale VH, Biddinger GR, Newman MC, Oris JT, Suter GW, Thompson T, Armitage TM,

Meyer JL, Allen-King RM, Burton GA, Chapman PM, Conquest LL, Fernandez IJ,

Landis WG, Master LL, Mitsch WJ, Mueller TC, Rabeni CF, Rodewald AD, Sanders JG, Van Heerden IL. 2008.Enhancing the ecological risk assessment process.Integrated Environmental Assessment and Management4:306–313DOI 10.1897/IEAM 2007-066.1.

Davis RS. 2007.An ecological risk assessment for mosquito insecticides. Master’s Thesis, Montana State University.

Davis RS, Peterson RKD, Macedo PA. 2007.An ecological risk assessment for insecticides used in adult mosquito management.Integrated Environmental Assessment and Management3:373–382 DOI 10.1002/ieam.5630030308.

Devine OJ, Qualters JR. 2008. Bayesian updating of model-based risk estimates using imperfect public health surveillance data.Human and Ecological Risk Assessment14:696–713 DOI 10.1080/10807030802235094.

Duprey Z, Rivers S, Luber G, Becker A, Blackmore C, Barr D, Weerasekera G, Kieszak S,

Flanders WD, Rubin C. 2008.Community aerial mosquito control and naled exposure.Journal of the American Mosquito Control Association24:42–46DOI 10.2987/5559.1.

Gargoum AS. 2001.Use of Bayesian dynamic models for updating estimates of contaminated material.Environmetrics12:775–783DOI 10.1002/env.499.

Gelman A, Carlin JB, Stern HS, Rubin DB. 2004.Bayesian data analysis. Boca Raton: CRC Press. Gosselin N, Valcke M, Belleville D, Samuel O. 2008.Human exposure to malathion during

a possible vector-control intervention against West Nile Virus. I: methodological framework for exposure assessment.Human and Ecological Risk Assessment14:1118–1137 DOI 10.1080/10807030802493834.

Grist EPM, O’Hagan A, Crane M, Sorokin N, Sims I, Whitehouse P. 2005.Bayesian and time-independent species sensitivity distributions for risk assessment of chemicals.

Environmental Science and Technology40:395–401DOI 10.1021/es050871e.

Hill RA. 1996.From science to decision-making: the applicability of Bayesian methods to risk as-sessment.Human and Ecological Risk Assessment2:636–642DOI 10.1080/10807039609383641. Hull RN, Swanson S. 2006. Sequential analysis of lines of evidence—an advanced

weight-of-evidence approach for ecological risk assessment.Integrated Environmental Assessment and Management2:302–311.

Karpati AM, Perrin MC, Matte T, Leighton J, Schwartz J, Barr RG. 2004.Pesticide spraying for West Nile virus control and emergency department asthma visits in New York City, 2000.

Environmental Health Perspectives112:1183–1187DOI 10.1289/ehp.6946.

Kutz FW, Strassman SC. 1977.Human urinary metabolites of organophosphate insecticides following mosquito adulticiding.Mosquito News37:211–218.

Linkov I, Loney D, Cormier S, Satterstrom FK, Bridges T. 2009.Weight-of-evidence evaluation in environmental assessment: review of qualitative and quantitative approaches.Science of the Total Environment407:5199–5205DOI 10.1016/j.scitotenv.2009.05.004.

Macedo PA, Peterson RKD, Davis RS. 2007.Risk assessments for exposure of deployed military personnel to insecticides and personal protective measures used for disease-vector management. Journal of Toxicology and Environmental Health, Part A 70:1758–1771 DOI 10.1080/15287390701459049.

Marfin AA, Gubler DJ. 2001.West Nile encephalitis: an emerging disease in the United States.

Clinical Infectious Diseases33:1713–1719DOI 10.1086/322700.

McKone TE, Ryan PB, Ozkaynak H. 2009.Exposure information in environmental health research: current opportunities and future directions for particulate matter, ozone, and toxic air pollutants.Journal of Exposure Science and Environmental Epidemiology19:30–44 DOI 10.1038/jes.2008.3.

Menzie C, Henning MH, Cura J, Finkelstein K, Gentile J, Maughan J, Mitchell D, Petron S, Potocki B, Svirsky S, Tyler P. 1996. Report of the Massachusetts weight-of-evidence workgroup: a weight-of-evidence approach for evaluating ecological risks.Human and Ecological Risk Assessment2:277–304DOI 10.1080/10807039609383609.

Morris PA. 1977.Combining expert judgments: a Bayesian approach.Management Science 23:679–693DOI 10.1287/mnsc.23.7.679.

National Research Council. 1994.Science and judgment in risk assessment. Washington D.C.: National Academy Press.

National Research Council. 1996.Understanding risk: informing decisions in a domcratic society. Washington, D.C.: National Academy Press.

New York City Department of Health. 2005.Adult mosquito control programs: environmental impact statement (EIS). New York: New York City Department of Health.

O’Sullivan BCY, Lafleur J, Fridal K, Hormozdi S, Schwartz S, Belt M, Finkel M. 2005.The effect of pesticide spraying on the rate and severity of ED asthma.American Journal of Emergency Medicine23:463–467DOI 10.1016/j.ajem.2004.12.017.

Peterson RKD. 2006.Comparing ecological risks of pesticides: the utility of a risk quotient ranking approach across refinements of exposure. Pest Management Science62:46–56 DOI 10.1002/ps.1126.

Peterson RKD, Macedo PA, Davis RS. 2006.A human-health risk assessment for West Nile virus and insecticides used in mosquito management.Environmental Health Perspectives114:366–372 DOI 10.1289/ehp.8667.

Reisen W, Brault AC. 2007.West Nile virus in North America: perspectives on epidemiology and intervention.Pest Management Science63:641–646DOI 10.1002/ps.1325.

Roche JP. 2002.Print media coverage of risk–risk tradeoffs associated with West Nile encephalitis and pesticide spraying.Journal of Urban Health79:482–490DOI 10.1093/jurban/79.4.482. Samet JM, Schnatter R, Gibb H. 1998.Invited commentary: epidemiology and risk assessment.

American Journal of Epidemiology148:929–936DOI 10.1093/oxfordjournals.aje.a009569. Sanchez-Bayo F, Baskaran S, Kennedy IR. 2002.Ecological relative risk (EcoRR): another

approach for risk assessment of pesticides in agriculture.Agriculture Ecosystems & Environment 91:37–57DOI 10.1016/S0167-8809(01)00258-4.

Schenker U, Scheringer M, Sohn MD, Maddalena RL, McKone TE, Hungerbuhler K. 2009.Using information on uncertainty to improve environmental fate modeling: a case study on DDT.

Environmental Science and Technology43:128–134DOI 10.1021/es801161x.

Schleier III JJ. 2008.Environmental concentrations, fate, and risk assessment of insecticides used for adult mosquito management. Master’s Thesis, Montana State University.

Schleier III JJ, Davis RS, Barber LM, Macedo PA, Peterson RKD. 2009a.A probabilistic risk assessment for deployed military personnel after the implementation of the “Leishmaniasis Control Program” at Tallil Air Base, Iraq.Journal of Medical Entomology46:693–702 DOI 10.1603/033.046.0337.

Schleier III JJ, Davis RS, Shama LM, Macedo PA, Peterson RKD. 2008a.Equine risk assessment for insecticides used in adult mosquito management.Human and Ecological Risk Assessment 14:392–407DOI 10.1080/10807030801934812.

Schleier III JJ, Macedo PA, Davis RS, Shama LM, Peterson RKD. 2009b.A two-dimensional probabilistic acute human-health risk assessment of insecticide exposure after adult mosquito management.Stochastic Environmental Research and Risk Assessment23:555–563 DOI 10.1007/s00477-008-0227-5.

Schleier III JJ, Peterson RKD. 2010.Deposition and air concentrations of permethrin and naled used for adult mosquito management.Archives of Environmental Contamination and Toxicology 58:105–111DOI 10.1007/s00244-009-9353-4.

Schleier III JJ, Peterson RKD, Macedo PA, Brown DA. 2008b.Environmental concentrations, fate, and risk assessment of pyrethrins and piperonyl butoxide after aerial ultralow-volume applications for adult mosquito management.Environmental Toxicology and Chemistry 27:1063–1068DOI 10.1897/07-532.1.

Stauffer HB. 2008.Application of Bayesian statistical inference and decision theory to a fundamental problem in natural resource science; the adaptive management of an endangered species.Natural Resource Modeling21:264–284DOI 10.1111/j.1939-7445.2008.00007.x. Suffolk County. 2006.Draft generic environmental impact statement. Southampton: Suffolk

County Department of Public Works and Department of Health Services.

Suter II GW, Cormier SM. 2011.Why and how to combine evidence in environmental assessments: weighing evidence and building cases.Science of the Total Environment 409:1406–1417DOI 10.1016/j.scitotenv.2010.12.029.

Sutton AJ, Abrams KR. 2001.Bayesian methods in meta-analysis and evidence synthesis.Statistical Methods in Medical Research10:277–303DOI 10.1191/096228001678227794.

Taylor AC, Evans JS, McKone TE. 1993.The value of animal test information in environmental control decisions.Risk Analysis13:403–412DOI 10.1111/j.1539-6924.1993.tb00740.x.

Thier A. 2001.Balancing the risks: vector control and pesticide use in response to emerging illness.

Journal of Urban Health78:372–381DOI 10.1093/jurban/78.2.372.

Tilghman A, Coquery M, Dulio V, Garric J. 2009.Integrated chemical and biomonitoring strategies for risk assessment of emerging substances: report on the 4th thematic workshop of the EU Project NORMAN, Lyon, France, 17–18 March 2008.TrAC Trends in Analytical Chemistry28:1–9DOI 10.1016/j.trac.2008.11.002.

United States Environmental Protection Agency. 1989.Risk assessment guidance for superfund. Volume I. Human health evaluation manual (Part A). Washington D.C.: Environmental Protection Agency.

United States Environmental Protection Agency. 2004.Risk assessment principles and practices: staffpaper. Washington D.C.: Environmental Protection Agency.

United States Environmental Protection Agency. 2005a.Guidelines for carcinogen risk assessment. Washington D.C.: Environmental Protection Agency.

United States Environmental Protection Agency. 2005b.Memorandum from B. Davis, Health Effects Division, to C. Rodia, Special Review and Registration. Re: occupational and residential exposure assessment and recommendations for the reregistration elegibility decision (RED) for piperonyl butoxide. Washington D.C.: Environmental Protection Agency.

United States Environmental Protection Agency. 2005c.Memorandum from S.L. Kinard, Health Effects Division, to T. Moriarty, Special Review and Reregistration Division. Malathion: updated revised human health risk assessment for the reregistration eligibility decision document (RED). Washington D.C.: Environmental Protection Agency.

United States Environmental Protection Agency. 2005d.Memorandum from S.L. Kinard, Y. Yang and S. Ary, Health Effects Division, to J. Guerry, Special Review and Reregistration Division. Re: Permethrin. HED Chapter of the Reregistration Eligibility Decision Document (RED). Washington D.C.: Environmental Protection Agency.

United States Environmental Protection Agency. 2006a.Interim reregistration eligibility decision for naled, Case No. 0092. Washington D.C.: Environmental Protection Agency, 1–130.

United States Environmental Protection Agency. 2006b.Reregistration eligibility decision (RED) for permethrin. Washington D.C.: Environmental Protection Agency, 1–95.

United States Environmental Protection Agency. 2006c.Reregistration eligibility decision for malathion. Washington D.C.: Environmental Protection Agency, 1–101.

Valcke M, Gosselin N, Belleville D. 2008.Human exposure to malathion during a possible vector-control intervention against West Nile Virus. II: evaluation of the toxicological risks using a probabilistic approach.Human and Ecological Risk Assessment14:1138–1158 DOI 10.1080/10807030802493891.

Weed DL. 2005.Weight of evidence: a review of concept and methods.Risk Analysis25:1545–1557 DOI 10.1111/j.1539-6924.2005.00699.x.

Wheeler MW, Bailer AJ. 2009.Benchmark dose estimation incorporating multiple data sources.

Risk Analysis29:249–256DOI 10.1111/j.1539-6924.2008.01144.x. Zadeh L. 1965.Fuzzy sets.Information and Control8:338–353