Best Practice Regulation Handbook
Appendix F. Risk analysis
Government regulation rarely deals with certainties. Regulation is often designed to reduce the likelihood of harmful or hazardous events occurring. Around half of all new regulations requiring RISs are risk-related. Examples include regulation to:
- reduce the incidence of workplace accidents
- reduce public health hazards (e.g. food standards)
- reduce risks from faulty consumer products (e.g. product safety standards)
- reduce the risk of financial institution failure, and
- reduce the risk of terrorist attacks.
Given the importance of risk-related regulation, the purpose of this guidance is to provide you with advice about how you can approach the evaluation of regulation aimed at managing risks in RISs. An effective approach to the management of risks requires that agencies develop a thorough understanding of the nature of the risks they are seeking to manage. This can be achieved by soundly applying risk analysis and economic evaluation principles.
F1. Defining risk and uncertainty
When talking about risk, a distinction is often made between the terms ‘risk’ and ‘uncertainty’. Risk usually refers to situations where the probability of a hazardous event occurring is reasonably well known and can be reasonably estimated. Uncertainty refers to a situation in which the probability of a hazardous event occurring cannot be reasonably (or reliably) estimated. In practice, this distinction is often difficult to draw because, while probabilities can generally be assigned to most events, there is seldom complete certainty about the size of all risks.
Risk analysis is an important part of the government’s best practice regulation requirements and, where relevant, must be incorporated into RISs. The purpose of risk analysis in a RIS is to shed light on sources of uncertainty about the possible impacts of proposed regulation on economic outcomes. Risk analysis should not be seen as a distinct step in the RIS process but should be considered throughout each step of the RIS process with particular emphasis on the problem and impact analysis sections.
F2. Problem definition
The problem section in the RIS describes the main issues that government action is designed to address. In the case of regulation aiming to manage risks or reduce harms, the problem section should identify what the risk is, and what will be the likely risk into the future in the absence of government action. This means that the problem section should contain relevant information on the size of the actual risk, its likelihood and nature (see discussion below). The problem section should also clearly identify who bears the risk and highlight how the risks are currently being addressed.
Actual versus subjective risks
The problem section of a RIS should focus on objective risks rather than ‘perceived’ risks. Perceptions about risk can be founded on bias and misinformation about the true magnitude and severity of risks (Viscusi et. al. 1997). Individuals can often perceive a risk (or harm) to be much greater than it actually is – especially when there is a lack of information about the risk or strong perceptions about the size of the risk. To this end, in this section of the RIS you should focus on evidence about actual risks and seek to quantify the actual risk.
Perceptions about risk, however, should not be entirely overlooked. If there is inadequate information about a risk, or public misperception about a risk, this may call for government action to reduce the degree of misperception about the risk. For example, there is often considerable public misperception about many health risks (e.g. risks of sun exposure, risks associated with binge drinking). A possible response would be to inform the public through appropriate campaigns aimed at reducing the extent of misinformation.
Size of the risks
The size of a risk is generally characterised by the likelihood of an event occurring (i.e. the probability of the adverse event or harm occurring), and the size of the impact should the event occur.
Measuring risk can be a difficult task but can be achieved using reliable sources of information. Quantifying the magnitude of the risk is an important first step because it will inform the impact analysis (and cost-benefit analysis) at a later stage of the RIS process. Sources of information to identify the likelihood and severity of risks include:
- accident and incident data (e.g. fatality rates, incidence statistics)
- coroner’s reports
- actuarial estimates
- medical research data (e.g. medical studies, epidemiological studies)
- expert panel surveys, and
- targeted surveys.
You can use information from these sources to inform stakeholders and decision- makers about all the relevant facts so that they can make an informed decision. When using data from secondary sources as evidence, it is important to ensure that the estimates used appropriately characterise the risk associated with the problem being considered.
Where perceived risks are deemed important, the problem section should also inform stakeholders about general risk perceptions.
Some risks, however, will be very difficult to quantify; in these cases, sound qualitative assessments can be used to supplement quantitative analysis. The analyst should discuss the size of the risk by reference to the likelihood of the event occurring and the severity of its impact. This assessment may be supported by a discussion of the factors that contribute to the likelihood of the risk and by reference to the impacts of similar events occurring in Australia or overseas.
The nature of the risk
The problem section should describe the nature of the risk and the adverse outcomes that could eventuate in the absence of government action. Is it a risk that is straightforward to estimate (actuarial-type risks) or is the risk characterised by random events (e.g. flood, earthquake, terrorist attack)?
The evaluation and analysis should consider if there is a long delay between an adverse event and the consequences (latent risks). The adverse consequences associated with climate change, for example, will gradually occur over time. This means that the impacts of the associated consequences of climate change will most likely happen slowly. Some risks, for example the risk of terrorism, tornados, floods, and some disease outbreaks, will be characterised by random events that occur instantaneously and may require rapid response to deal with the impacts. Other risks, however, can be easily characterised because their occurrence is more easily observable (e.g. risk of damage to property, risks due to fires, risk of death due to car accidents).
Who bears the risk?
The problem sections should discuss which groups will bear the consequences if an adverse event occurs. The distribution of the risk may have important consequences for efficiency (some parties may be able to bear the risk at lower cost than other parties) and equity outcomes (it may be more socially acceptable for some parties to bear risk than others). The problem section should outline whether the distribution of risk is an important consideration.
F3. Impact analysis section
The impact analysis should be evidenced-based and comprehensive. This means you must identify all groups affected by the problem and its proposed solution, including those directly affected by the options and those indirectly affected. You should also assess the effects on the community as a whole in addition to this.
Assess the impact of the proposal on risk
You will need to appropriately examine the impact of each option identified to manage the risk. Your impact analysis should clearly spell out how each option will impact on the size and distribution of the risk. For each option being considered, this involves assessing the following questions:
- Will the proposal reduce the size of the risk (e.g. standards, information provision)?
- Will the proposal remove the risk altogether (e.g. banning certain activities/ products)?
- Will the proposal transfer risk from one party to another (e.g. mandatory insurance)?
A proposal may aim to reduce risks by imposing specific standards to improve product quality outcomes (e.g. food standards and product quality standards). In some instances the proposal might aim to mitigate risk by directly banning activities (e.g. banning cigarette smoking in vehicles or certain public spaces). Actions can also be proposed to transfer or redistribute risks (e.g. bank deposit guarantee schemes). You should analyse each option clearly in the light of its impact on risk reduction, transfer or elimination. The options should be examined in terms of their impacts on all those affected (e.g. consumers, producers, governments and regulators).
Quantify the risk
Quantify the risks as far as possible. This means that the impact analysis section should contain your detailed assessment of the size of the actual risk. The analysis should be informed based on existing sources of relevant evidence about risks or based on specific studies to uncover the size and magnitude of the risk. The analysis in the impact section will flow directly from the analysis given in the problem section of the RIS. To apply cost-benefit analysis (CBA), you will need to be informed about:
- the likelihood of the risk (an estimate of the actual risk), and
- the impact of the proposal on probable future risk outcomes.
The latter will require that you make forecasts about how the proposed regulation or option will affect the future size and magnitude of the risk.
How are risks and hazards measured in practice?
Risks and hazards are often measured by the ‘rate’ or ‘average number of occurrences’ for an event of interest per 100,000 persons per period of time. The risk of death from drowning in Australia, for example, was estimated at 1.2 persons per 100,000 during the period 2004-05 (Table F.1).
Analysts should take care to ensure that all estimates used provide a reasonable assessment of the probability of the actual hazards being evaluated. Where available, consult data from different sources to compare and contrast estimates.
Decision makers can use actual evidence on risks and hazards to assist them to prioritise their response. Understanding the level of risk is important for the impact assessment. To undertake a meaningful impact analysis it is important to understand and form expectations about how each option will reduce risks into the future.
What if risks cannot be quantified?
When sound quantitative evidence is not available, discussion based on sound qualitative evidence can also be used to inform the RIS. However, you should attempt to quantify all risks as far as possible. When risks cannot be quantified, various approaches can be used to analyse the potential size and impact of risks. Qualitative tools for risk analysis include but are not limited to the following:
Table F.1: Estimates of common risks *
Risk |
Estimate |
|---|---|
Drowning (all persons) |
1.2 |
Drowning (persons in age group 0-4 years) |
1.8 |
Poisoning (drugs) |
3.7 |
Poisoning (other substances) |
1.5 |
Smoke, fire and flames, heat and hot substances |
0.8 |
*All rates per 100,000 persons.
Source: Henley and Harrison (2009).
- using judgements based on expert panels
- risk categorisation matrix (prioritising or categorising to manage according to informed judgements about likelihood and severity), and
- comparative analysis (e.g. comparing the effectiveness of risk management approaches across different jurisdictions).
Assess the impact of each option by applying CBA
Decisions about risk management strategies should be informed by consideration of the costs and benefits of regulation. Risk management strategies will impose costs and benefits on members of society and hence will support different efficiency and distributional outcomes. In practical CBA your aim is to quantify who will be impacted by the regulation and by how much. The key objective is to assess alternative risk management options in terms of their relative efficiency outcomes. This means quantifying all costs and benefits associated with a risk management strategy. You should also consider distributional impacts.
Consider potential unintended consequences of each option
Risk management policies can sometimes elicit adverse changes in the behaviour of economic agents (consumers, suppliers and other actors in the economy). In particular, agents may undertake riskier behaviour if they face a reduced likelihood of adverse consequences of that behaviour or if the consequences are borne by another party. This is known as the moral hazard problem.
Proposals aimed at guaranteeing bank deposits, for example, may elicit excessive risk-taking behaviour by financial institutions, which could result in large financial losses being borne by the broader community. When looking at changing work-related safety regulation, for example, it is important to consider the behavioural consequences of reallocating risks between employers and workers. Regulations that impose too stringent obligations on employers can potentially reduce the care taken by workers, hence resulting in increases in workplace accidents.
When evaluating the impact of government intervention it is important to consider the potential undesirable consequences on the behaviour of all stakeholders affected. This can be taken into account by applying sensitivity analysis within the CBA framework.
Dealing with uncertainty in risk analysis
Many risks are uncertain, this means that their size can be difficult to quantify in practice. On these occasions, you can take uncertainty into account by applying sensitivity analysis to the main assumptions. Various quantitative techniques could be useful where the probability distribution of risk is known. Where it is unknown, sensitivity analysis based on ‘best case’ and ‘worst case’ scenarios can be used. When the analysis of the risks and/or uncertainty is not possible, the RIS should include a qualitative discussion around sources of uncertainty of the risk, and how this might impact on the likely outcomes of the policy.
Tools for dealing with uncertainty
Uncertainty in cost-benefit analysis can be taken into account by using various quantitative tools and techniques. Boardman et. al. (2006) provide a comprehensive account of different techniques that can be applied to deal with uncertainty in CBA. These include:
- Sensitivity analysis: can be used when there is considerable uncertainty about the values of key variables used in cost-benefit analysis. When used in combination with probabilistic modelling, sensitivity analysis can provide a powerful tool to gain useful insights into the basis of ‘worst’ and ‘best’ case outcomes. Sensitivity analysis is often applied to assess the impact of changes in a key variable (single parameter) on the net benefit estimate.
Sensitivity analysis can also be used to check the overall robustness of the CBA results for changes in globally sensitive assumptions (e.g. cost and benefit estimates, and risk estimates).
- Probabilistic modelling: can be applied to gain useful insights into the statistical distribution of uncertain variables used in CBA. Monte Carlo analysis can be used to evaluate the effects of changing various assumptions in the CBA and assessing the impact on net benefit estimates. This method allows analysts to estimate the degree of ‘variation’ associated with key CBA analysis parameters simultaneously and assess the impact on key CBA results.
F4. Implementation and review section
Given that many risks are uncertain (that is, the likelihood of them occurring is relatively unknown), and there is the potential for unintended consequences, it is important for the RIS to contain a detailed discussion of the policy implementation and review process.
How will the preferred option be implemented?
In this section you would provide an analysis of the key strategy to deliver the preferred option and the timings for when the preferred option is expected to be delivered. Your discussion should identify key risks and barriers that could prevent the effective delivery and implementation of the proposed strategy. These factors may include resource requirements, information gaps, administrative and compliance issues, and enforcement requirements. Strategies to overcome delivery barriers should also be identified.
How will the preferred option be reviewed?
The RIS should highlight the key details about how the expected option will be reviewed and assessed for effectiveness. This is particularly important where there is uncertainty about the risks involved because the optimal strategy may change as new information is gathered.
You should identify the most important information gaps and highlight how these gaps will be overcome; what systems or projects will be set up to allow the collection of data and the tracking of performance of the policy over time? Setting up (and describing) these processes early increases the chance of good quality information being collected once the policy is implemented and will help make the subsequent review more meaningful.
| Previous Appendix E. Cost-benefit analysis |
Next Appendix G. Business Cost Calculator |
Contact for information on this page: OBPR contacts page