Pages within the "support" section:

15. Addressing uncertainty in risk-benefit assessment

Risk-benefit analysis is affected by many potential sources of uncertainty which may all contribute to uncertainty in the estimated net health impact. This uncertainty may have important implications for decision-makers. For example, the median estimate for net health impact may be positive but if the probability interval is wide there may be a large chance that the actual impact is negative.

The importance of considering uncertainty in risk assessment is recognised in the Codex Working Principles for Risk Analysis, which state: “Constraints, uncertainties and assumptions having an impact on the risk assessment should be explicitly considered at each step in the risk assessment and documented in a transparent manner. Expression of uncertainty or variability in risk estimates may be qualitative or quantitative, but should be quantified to the extent that is scientifically achievable.” (Codex 2007). Logically, if these principles apply to risk assessment they should also be applicable to risk-benefit analysis.

The Codex principle quoted above implicitly recognises that is not feasible to quantify all sources of variability and uncertainty. It implies that all those that are quantifiable should be quantified, but in fact this is not necessary if a qualitative consideration of uncertainty is sufficient for decision-makers to reach a decision with adequate confidence (e.g. if it is clear that the assessment is conservative). Therefore what is needed is a flexible or tiered approach which:

·          Documents all identifiable sources of uncertainty

·          Evaluates all of them at least qualitatively

·          Quantifies them to the extent that is necessary for decision-making.    

An approach of this sort has been described in guidance published by EFSA (2006b) for dealing with uncertainty in exposure assessment, and is outlined in the following sections below. Although aimed at exposure assessment, the approach is sufficiently general that it can be applied equally to the assessment of adverse and beneficial effects and net health impact.

Qualitative evaluation of uncertainties may be sufficient for decision-making in some cases, e.g. if it provides sufficient confidence that the net health impact of a proposed intervention is positive. In cases where more confidence is required, one option is to obtain additional data to reduce uncertainty. The alternative is to refine the characterisation of the uncertainties, by quantifying one or more of the most important uncertainties.

EFSA (2006b) suggests an iterative process which progressively quantifies more of the uncertainties, either deterministically or probabilistically, until there is sufficient confidence for decision-making. Ideally, priority would be given to quantifying the uncertainties thought to have most influence on the outcome, as indicated by the qualitative evaluation; probabilistic analysis may be targeted on those uncertainties identified as most influential by sensitivity analysis.

Generally only a small proportion of all uncertainties will be quantified, so it is essential that this is always accompanied by qualitative evaluation of the unquantified uncertainties, so as to provide a comprehensive characterisation of the overall uncertainty affecting the assessment.

The overall magnitude of uncertainty associated with a risk-benefit assessment may often be large. This should not be regarded as implying a failure of the assessment process; on the contrary, it provides essential information for decision-making (Codex 2007, Madelin 2004).

Common sources of uncertainty affecting risk-benefit analysis are summarised in Table 1. This list may be helpful as an aide memoire when identifying the uncertainties relevant in individual assessments, but is not comprehensive.

Table 1. Common sources of uncertainty affecting risk-benefit analysis.

Uncertainties affecting problem formulation

·          Specification of reference and alternative intake scenarios

·          Specification of relevant population

·          Identification of sensitive or otherwise important sub-populations

Uncertainties affecting hazard and benefit identification

·          Identification of relevant nutrients and contaminants

·          Identification of relevant health endpoints for each nutrient and contaminant

Uncertainties affecting intake assessment

·          Measurement uncertainty in concentration data

·          Measurements below the limit of detection, quantification or reporting

·          Sampling uncertainty due to limited number of concentration measurements

·          Bias due to intentional targeting of monitoring for contaminants

·          Uncertainty about correlations between concentrations of different contaminants/nutrients

·          Extrapolation of concentrations from measured to unmeasured foods

·          Future changes in levels of chemical use or contamination

·          Uncertainty in recording of foods and food weights in dietary surveys

·          Sampling uncertainty due to limited numbers of persons and days in dietary surveys

·          Measurement uncertainty and bias in body weight data (usually minor)

·          Uncertainty about degree of uptake of dietary recommendations

·          Uncertainty about level of background exposure (other foods, or other routes of exposure)

·          Uncertainty about compensatory changes in existing diet when taking up dietary recommendations

·          Assumptions about how the diets of individuals change over long time periods

Uncertainties affecting dose/response relationships estimated from animal data

·          Within- and between-laboratory variation

·          Sampling uncertainty due to limited number of subjects

·          Experimental errors e.g. in administration of treatments (generally minor)

·          Choice of dose-response model (goodness of fit)

·          Extrapolation to low doses (when required)

·          Extrapolation from animals to humans

·          Provision for within-species variation

Uncertainties affecting dose/response relationships estimated from epidemiological studies

·          Sampling uncertainty due to limited number of subjects

·          Estimation of intakes

·          Choice of dose-response model (goodness of fit)

·          Extrapolation to low doses (when required)

·          Combination of multiple studies (meta-analysis)

·          Relevance of study population to target population

Uncertainties affecting conversion to a common health currency (e.g. DALY, QALY)

·          Estimation of age of onset for health effects (especially those modelled from animal data)

·          Characterisation of disease severity (e.g. DALY or QALY weights)

·          Difficulty integrating effects if more than one stressor affects the same health endpoint

·          Variation of disease severity and duration, including any relation with dose

·          Estimation of recovery rate and time to recovery

·          Estimation of mortality rate and time to death

·          Recovery, death, severity, duration, age of onset may depend on cause of effect, and therefore differ from national statistics

·          Interactions between different diseases or health endpoints (including background diseases not directly affected by the dietary change or intervention), including comorbidity and substitution

Uncertainties in the probabilistic treatment of uncertainties

·          Choice of distributions to represent uncertainties

·          Dependencies between parameters

·          Uncertainty introduced by computational methods, e.g. number of simulations/iterations.

Uncertainties due to factors not considered in the assessment