10 Sensitivity Analysis
Sensitivity analysis is the process by which the robustness of a cost-utility analysis (CUA) is assessed by examining the changes in the results of the analysis when key variables are varied.
In general, uncertainty can be characterised as either parameter-related or modelling-related.
10.1 Parameter Uncertainty
Key Recommendations: Sensitivity analysis should include univariate (simple) analysis and multivariate analysis. When undertaking detailed analysis, probabilistic sensitivity analysis may be necessary. Any uncertainty in the analysis should be fully tested and described in the report.
The following steps should be undertaken to test the level of uncertainty of a parameter (9, 63):
10.1.1 Identify the Parameters
Parameters to consider include those with the greatest level of uncertainty (eg those derived from opinion), and those with the greatest influence on model outcomes (eg key clinical variables and costs).
10.1.2 Specify the Plausible Range over which the parameters may vary
The range over which parameters should be varied in the sensitivity analyses should be based on the available scientific literature, expert opinions, or a scale that is regarded as plausible.
10.1.3 Calculate Results
The level of sensitivity analysis undertaken should be determined by:
- the impact the results of the analysis could have on the funding decision – if a pharmaceutical is considered to be relatively cost-effective compared with other funding options, but is sensitive to several parameters, more extensive sensitivity analysis should be undertaken than for a pharmaceutical considered likely not to be relatively cost-effective
- certainty in inputs – if there is significant uncertainty in inputs, for example if surrogate endpoints are used or long-term extrapolation of data is required, more extensive testing needs to be undertaken
- quality of clinical trials – if the clinical inputs in the analysis were based on trials with a low grade of evidence (eg open-label, high risk of bias, allowed crossover of treatments), more extensive testing should be undertaken
- risk – further testing is required for high-expenditure pharmaceuticals due to the higher opportunity cost of funding
- results of sensitivity analysis – if the initial results of a sensitivity analysis indicate some uncertainty in inputs, further testing should be undertaken
- level of analysis – rapid CUAs are often based on a number of assumptions that require extensive testing.
PHARMAC recommends considering the approaches described in the table below when undertaking sensitivity analysis (64-66).
Table 13: Sensitivity Analysis Methods
| Method | Description | Advantages | Disadvantages |
|---|---|---|---|
| Univariate (simple) | Assesses the impact on the results of changing one variable. | Quick, simple, and easy to communicate results. Is sufficient if each of the uncertain variables is independent of the others. | There is a risk of ignoring interactions between parameters, hence underestimating overall uncertainty. This method also does not allow for the calculation of confidence intervals. |
| Multivariate | Evaluates the uncertainty related to multiple parameters by varying more than one parameter at once. | Generates more pragmatic results than univariate sensitivity analysis. | If there is a large number of uncertain variables, it may be difficult to present and interpret the results, particularly if parameters are correlated. |
| Threshold | Calculates the value a variable would need to reach in order to change the outcome of the analysis. | Useful when a parameter is indeterminate, such as the price of the pharmaceutical. | May require a ‘cost-effectiveness threshold’, which PHARMAC does not have. |
| Probabilistic | Based on Monte Carlo simulations. Examines the impact on the results of the analysis when variables are varied simultaneously according to predefined distributions. | Permits varying all parameters in the model simultaneously and enables calculation of the expected value and variance of decision variables. Uses information from clinical trials on distributions of effect size. | Can only handle uncertainty in data inputs. It has also been criticised on the basis that it introduces further assumptions into the model – in particular, the choice of distribution to represent uncertainty. |
At a minimum, the analysis should include univariate and multivariate sensitivity analyses. When undertaking detailed analyses, probabilistic sensitivity analysis should be considered. However, probabilistic sensitivity analysis should only be reported in addition to, rather than instead of, univariate and multivariate sensitivity analysis.
10.1.4 Interpret Results
PHARMAC recommends that sensitivity analysis be presented and interpreted using table format, graphical depiction, and/or elasticities.
Graphical presentations of CUA results are useful in gaining a visual interpretation of the sensitivity of parameters in the model. PHARMAC recommends tornado graphs for presenting the results of the sensitivity analysis. A tornado graph clearly presents and compares the variability of each parameter.
Elasticities provide information on the degree to which the results of the CUA change when inputs are varied (ie by changing a parameter by x%, the results of the analysis change by y%). The use of elasticity allows for a more objective judgement to be made aboutthe sensitivity of variables in the model.
Regardless of the method used to present the results, the report should fully describe any uncertainty in the analysis, with a focus on the key parameters that influence the results of the analysis.
10.2 Model Structure Uncertainty
Modelling-related uncertainty is uncertainty that depends on the chosen structure of the model, or is related to the overall process for modelling. Model uncertainty can be tested by running different analyses using alternative model structures, and reviewing the appropriateness of the results (9, 67).
Modelling-related uncertainty includes (9, 67):
- choice of functional forms for extrapolating outcomes (eg constant benefits, linear extrapolation, etc)
- choice of health states.
It is recommended that structural uncertainty be formally examined in sensitivity analysis. When testing the model, PHARMAC recommends using extreme sensitivity analysis to verify that the model generates logical results.