Pages within the "support" section:

15.3 Probabilistic treatment of uncertainty in Qalibra

The Qalibra software is designed to provide maximum flexibility and control to users in how uncertainty is quantified. This is achieved by allowing the user to enter a sample of values representing uncertainty for any or all of the input parameters. A separate input file is required for each input parameter: for parameters for which uncertainty is not quantified, this file contains only one column of estimates for that parameter; whereas for parameters for which uncertainty is quantified, the input file contains multiple columns containing the sample of multiple estimates for the parameter, representing its uncertainty (the rows in each file relate to different combinations of covariates, e.g. age, intake, gender, etc.). This sample is then used to represent the uncertainty for that parameter in the Qalibra calculations.

The flexibility provided by this approach allows the analyst to select any suitable, or convenient, method to quantify uncertainty. Samples for uncertain parameters may be generated by frequentist or Bayesian statistical methods, as appropriate, or by sampling from distributions specified by expert judgement (preferably using formal methods of expert elicitation). All methods used should be fully justified and documented, and any unquantified uncertainties associated with them (e.g. relating to underlying data and how they are modelled) should be taken into account as part of the qualitative evaluation of uncertainties (see section 15.1).

The Qalibra software executes multiple iterations of the risk-benefit calculation (currently 10,000), each taking different combinations of uncertain values for those inputs where the user has provided samples to represent uncertainty. If the user provides input samples smaller than 10,000 per parameter then Qalibra resamples the input values to obtain the larger sample. If the user provides input samples greater than 10,000 then Qalibra uses this larger value for the number of iterations.  

Intakes for different effects (for both reference and alternative scenarios) are resampled together, i.e. in any given iteration, the same column of the uncertain estimates in the input matrix is taken for the reference and alternative intakes for all of the effects in the assessment. This is done because intakes of different contaminants and nutrients in the diet will generally be inter-dependent: e.g. different substances in the same food will be positively correlated, substances in different foods may be negatively correlated (e.g. if a person eats more fish they may tend to eat less meat), and if an individual’s intake is higher than expected in the reference scenario it may also be higher in the alternative scenario. If the user represents these dependencies in the input matrices, then they will be maintained by the Qalibra resampling process. Note that this makes it important that the user provide sufficiently large samples for the intake parameters to adequately explore the different combinations of the uncertain values, as the linked resampling of these parameters in Qalibra will not explore additional combinations.

Other inputs to the calculation (i.e. parameters other than dietary intake) are resampled independently by the Qalibra software, to explore different combinations of uncertain values amongst these inputs and between them and the intake parameters. If it is thought these other inputs may be significantly interdependent, then this should be taken into account when considering the unquantified uncertainties affecting the assessment (see section 15.1).

Special care is required in the treatment of uncertainty for any parameters which are expressed as a function of individual attributes (age, gender, etc.). For such inputs, values relating to subpopulations with different combinations of attributes (e.g. juvenile females, adult males) are provided by the user as separate blocks of rows in the input matrix. If the user quantifies uncertainty, this is represented as a series of columns for each block of rows. In each iteration of the overall population calculation, Qalibra uses the same column for all rows. This makes the user responsible for ensuring that any dependency of the parameter between subpopulations is represented appropriate in the input matrix. For example, in the example presented later in this document (section 18), there are two effects for which p effect is a function of fish intake. For both effects, the dose-response differs between age-gender classes, and uncertainty in the dose-response is quantified in the model inputs. It seems probable that the dose-responses for different age-gender classes are interdependent – e.g. if the true slope of the dose response for juvenile females is actually towards the upper end of its confidence interval, then the same is likely to apply to the slope for other age-gender classes, e.g. adult females, adult males, etc. To represent this dependency, the user needs to ensure that the uncertainty realisations for each subpopulation are suitably correlated. This can be done rigorously by explicitly modelling the dependency when generating the input values, or more approximately (as in our example) by generating the input values independently and then sorting the columns for each subpopulation in the same order (e.g. so that the right-most column contains the most positive realisation of the dose-response for every age-gender group). If this is not done, then the uncertainty of the population health impact will be underestimated because, in many iterations of the calculation, upper-percentile values for one subpopulation will be offset by lower-percentile values for other subpopulations. In the example presented in section 18, the 95% uncertainty interval for the net health impact of increasing fish intake was -1.4 to -7.5 DALYs per year per 999 people when dependency of dose-responses for different age-gender groups was included, and -3.3 to -5.4 when they were sampled independently [5] . These issues can have a large impact on the apparent precision of results, and must be considered for any parameter which varies between subpopulations and for which uncertainty is quantified.

It is important to ensure that the samples of values provided by the user to represent uncertainty are large enough for the calculations to produce stable outputs. This can be checked by generating a series of different samples, or smaller and larger samples: if the sample statistics (mean and percentiles) differ materially, then the sample sizes should be increased. Similarly, the user may check whether the 10,000 iterations executed by the Qalibra software is sufficient to explore the different combinations of the uncertain inputs in the same way, by repeating the run several times and comparing the results. If the results differ materially, then it would be advisable to increase the number of iterations (by providing a larger number of samples for at least one input, see above) or download and combine the output from multiple runs.

[5] The median estimates with and without dependency were almost identical: -4.32 and -4.29.