• Home
• Regulation
• Regulatory Policy
• Policy Guidance Material
• Policy Options Guidelines
• Steps to Take

• Regulatory Policy

• Policy Guidance Material

• Requirements Guidelines
• Reg. Compliance Guidelines
• Policy Options Guidelines
• BCCSs Guidelines

1. Establish the Counterfactual (Status Quo)

24. Before you identify options, you need to establish the status quo (no change from current policy, including future policies that have already been agreed to), which is the counterfactual against which all impacts are assessed. It is important to assess all impacts against the status quo, rather than, for example, comparing costs against a more prescriptive regulatory option.

25. Ensure you have a thorough understanding of the status quo. Often this is difficult. When thinking about what the status quo is, you should think not only about what the current situation is, but also more broadly about what will happen in the future if the Government takes no further action, taking into account any relevant predicted trends. The nature of society, income distribution, cultures, goods, consumers, technology and other factors may change over time. For example, if there is a housing shortage, even if Government takes no new action there may be incentives for commercial companies to build houses, which may partially mitigate the shortage.

26. You should also note that there are two main approaches to comparing the options with the counterfactual.

• The first is to analyse the options in terms of how they deviate from the status quo. This means describing them as how they are different from the status quo.
• The second is to analyse the other options as complete packages. This means describing each option as a whole.

27. Whichever option is chosen, when you present the impacts of the options you should look only at the costs and benefits that are additional to those that would have been incurred/received if no further action were taken (i.e. the marginal costs and benefits). Current costs and benefits are part of the background.

2. Identify the Range of Feasible Options

28. After you have understood the counterfactual (status quo), you need to identify the full range of feasible options. Feasible options are possible options that will solve the problem and meet the public policy objectives. Identifying a number of options is important as it is the only way of knowing whether what you are proposing is the best of the feasible options - if you focus on only one option, it is possible, and in a lot of cases, very likely, that the option you recommend is not the best option (you may have missed a better option). Options need not be alternatives - often a combination of options will be most effective (e.g. regulation accompanied by an information campaign). Options should be developed in consultation with affected parties, experts, literature, seeing what is done overseas etc. It is important that you consider sub-options. Often tweaking an option in one or more ways will improve the net benefit that can be obtained from the option. More guidance on identifying options is available in Identifying Policy Options (Ministry of Economic Development - available on the Policy Development Toolkit [link to Public Service Intranet - limited to government agencies]).

3. Identify and Understand the Impacts of Each Option

29. Arising from each option is a collection of impacts, which are made up of immediate impacts, indirect impacts, and dynamic impacts (which arise over time as a result of the environment changing within which people operate, especially as a result of incentives). The more impacts you identify, the more comprehensive and accurate your analysis will be. The key principle is that the level of analysis should be commensurate with the significance of the proposal.

• You need to understand what each impact is and what its characteristics are (timing, magnitude, and distribution). You need to be thorough in order to avoid recommending an option that has unintended consequences. For example, if one of the impacts of an option is that businesses will be required to do certain things, you should note:
  • which businesses will be required to meet the requirements;
  • what impacted businesses will need to do to ensure they meet the requirements;
  • the impact meeting the requirements will have on their business.

This may require understanding of the obligations that the affected businesses already face.

• The impacts of an option may be complex, and you will need to understand this complexity:
  • Impacts may vary across affected parties (for example, by size of business or business sector);
  • There may be unintended consequences;
  • The causal chain may involve several steps (some of which may be difficult to understand);
  • The causal chain may be influenced by external factors or feedback channels where later impacts influence former impacts and people's behaviour, which may have a complicated interaction with the impacts of the option;
• You will also need to be clear about whether each impact is likely to be ongoing (and how long it will last for).

30. Identify all affected parties:

• Ensure you identify all population groups and sub-groups likely to be affected by the option. A common breakdown is Government, industry, consumers and environment, though it may be more useful to identify the specific groups by affected party. You may need to further subdivide groups of affected parties - for example by size, income, geographical location (regional or rural), age, family unit, cultural background, or the level of information parties have.
• For each affected party you need to identify the full set of impacts on that party: how the option, if adopted, would affect the actions, obligations, circumstances, costs, and benefits to all affected parties.
• You should also note distributional impacts. Distributional impacts occur whenever there are winners and losers from an option (for example, where the costs are borne by one group in society and the benefits accrue to another group). In most cases the impacts will fall differently across people with different income levels.

31. Identify all types of impacts, including:

• health/well-being;⁴
• environmental;
• cultural;
• New Zealand's compliance with international obligations;⁵
• social;
• economic;
• legal and legislative implications;
• impacts on future generations.

32. Move from immediate effects to short term indirect effects and dynamic effects. A good place to start is brainstorming the impacts of the option. Normally you should think of the immediate change as a starting point and from this map out the indirect, dynamic, and feedback effects. For example, if the proposal is to reduce a tax, it may be useful to first note that those who pay the tax will pay less and then note that this will lead to certain impacts on their welfare and behaviour, which may have further indirect effects, dynamic effects and feedback effects.

33. Stem diagrams can be a useful tool to help you think about the causal chain of effects.

34. The arrows indicate causal links (for example, impact A causes E, which causes H and I). Note that while impact B causes G, G also has a feedback effect on B.

35. Identify impacts on those within and outside New Zealand. You need to identify impacts both on those within and outside New Zealand. Even if impacts on overseas parties are not considered important, they should at least be noted. While many proposals will affect New Zealand parties and other parties similarly, there may be occasions where it is appropriate to identify impacts specific to New Zealand parties or non-New Zealand parties. This can become complex when, for example, impacts on businesses are being considered where those businesses are owned by people who reside overseas but the businesses are based in New Zealand.

36. Identify changes to risk levels. You should identify the impacts each option would have on the likelihood, frequency, or consequences of risks resulting in an adverse event. There is a difference between risk and uncertainty - risk is where the probability of an adverse event occurring is less than certain; uncertainty is where the probability is unknown. Identifying changes to risk levels should be one of your central concerns when you are assessing options that are designed to change the risk associated with an event/situation and/or changing who bears the risk.

37. Consider behavioural responses and acknowledge your assumptions about behaviour. In virtually all situations, impacts will depend on people's behaviour. In order to be confident you have correctly identified the impacts of an option, you need to ensure you have considered the likely behavioural responses. If you have made assumptions about behavioural responses, which could include assumptions about how affected parties understand the world, you need to acknowledge these assumptions. For more guidance on this see Behavioural Analysis for Policy.

38. Consider the effects of controls on the impacts. The fact that the world is constantly changing means the impacts of your options may change over time. You need to be aware of how external factors are likely to influence the impacts. For example, if an industry is growing, the effect of requirements in terms of compliance costs may increase over time.

39. Monitoring and enforcement. You should consider how compliance with options and the success of the policy will be monitored and enforced. This is important because an option may be inappropriate or even counterproductive if it is difficult to enforce.

40. Do not assume 100% compliance. There is unlikely to be 100% compliance with existing or proposed changes to the law. Consider the impacts of lower levels of compliance.

41. Consider the risk of Government failure. It is important to consider the possibility of Government interventions not working - for example, where a licensing requirement is imposed and the requirements are so onerous that no-one applies for the licence. Government failure is where Government intervention does not solve the problem or makes the problem worse or worsens the overall situation (even if the problem is solved). Government failure can bring the law into disrepute.

4. Assess the Magnitude of Each Impact of Each Option

42. After you have identified each option's collection of impacts, you should assess the magnitude of each impact. This makes comparison between each of the costs and benefits (and hence comparison of the options) easier. To do this you are likely to need to take similar steps to those required to identify and understand the impacts (see above).

43. In order to be able to conduct a quantitative cost benefit analysis, you need to measure the costs and benefits in dollar terms. Even if you are undertaking a mainly textual cost benefit analysis, you should quantify at least some impacts: the level of analysis needs to be commensurate with the significance of the proposal. As the impacts become more significant (and the relationships between the impacts more complex), more advanced/accurate methods of quantification should be used (see Methods of Quantifying Impacts below).

44. Assessing an impact's magnitude means estimating how large or small the impact is. This may include assessing:

• the number of people affected;
• the monetary cost;
• how long something will take (e.g. an activity that will be required if the options is adopted, such as paperwork);
• how much benefit will be obtained;
• how much value will be lost or gained;
• the number of lives saved.

45. To quantify impacts, there are two basic steps:

• collecting data;
• making calculations to draw inferences from the data.

46. Data collection will vary in difficulty, cost, and feasibility, as will the calculations required (see Methods of Quantifying Impacts below). You should ensure that the data is unbiased and representative of the population upon which the impacts are likely to fall.

47. If you are having trouble quantifying impacts, note:

• It is better to provide estimates than no quantification. Estimates may include the relative size of impacts.
• Base your calculations on "expected" impacts, rather than "best case" or "worst case" potential impacts. The expected impact is the impact that will occur on average and is calculated by multiplying the magnitude of the impact if it did occur by the probability of the impact occurring.
• Some impacts are simply difficult to quantify, such as comfort, environmental, and cultural impacts. The quantification techniques below may be applicable, but if you are unable to quantify certain impacts, you should ensure that you have a good idea of the significance and relative importance of these impacts and be able to convey these qualitatively (i.e. in words). Note that the fact you cannot quantify some impacts does not mean that you cannot conduct a quantitative CBA (see paragraph 16).

48. Understand the complexities of each impact's magnitude. As with understanding the qualitative characteristics of the impact, you should understand the complexities of the magnitude, including variations in the size of impacts across different parties.

49. Avoid double counting. Double counting is a mistake that can be made. For example, if costs to businesses will be passed on to consumers, the cost should only be counted once. In this example, it may be difficult to determine how much of the cost will be passed on to consumers; the effect will depend on the elasticities of demand and supply in the relevant market.

Methods of Quantifying Impacts

50. There are a number of approaches to quantifying impacts. Which approaches you use depends on what you are quantifying and the resources available to you. The methods below can be relevant to both a textual or quantitative CBA.

51. Normally a basic level of quantification is not difficult. You need to turn your mind to what you can calculate, especially simple methods of quantification:

• Time - You could estimate wages multiplied by time taken.
• Health improvements - You could estimate the cost of treating any illnesses that will be averted.
• Compliance costs to businesses - You could ask businesses how long it would take them to meet requirements and multiply this by an estimated hourly rate.

52. Talk to affected parties. Affected parties are likely to have a detailed knowledge of how options will impact upon them, including information that will help you quantify impacts (especially costs). For example, businesses are likely to be in the best position to know how much it will cost them to meet new requirements and how changes to the law will influence their activities, including the way they do business and their profits.

53. Estimating by analogy. One way of quantifying impacts is to look at the impacts of similar policies and make adjustments to these to arrive at your best estimate of the impacts of the option you are considering. For example, when estimating the registration costs to psychotherapists of new registration requirements under the Health Practitioners Competence Assurance Act, the Ministry of Health established that the determinants of the cost are the size and complexity of the profession. After comparing psychotherapy with other similarly sized and similarly regulated health professions it came to a figure of approximately $500 per registered psychotherapist.

54. Quantifying social impacts. Some examples of methods of quantifying social impacts are:

• estimating gains in years of education that may result from a policy;
• the value of reassurance;
• estimating quality adjusted life years.

You may need to apply other methods in to calculate these. Information on these approaches can be found in a number of sources.

55. Market prices. Where market prices are relevant, quantification is relatively simple - the value lost or gained can be measured by the expected change in the price of the good or service, which indicates how much people are willing to pay for the good or service or how much they would be willing to accept to give up the good or service.

56. Econometric analysis. Econometric analysis involves collecting data on certain variables and testing the causal relationships between these variables (e.g. changes over time to income levels, education levels, profits made in certain industries etc). If similar policies have been implemented in the past, it may be possible to use econometric analysis to directly estimate the impact of an option on a problem. Otherwise you can estimate the effect on a problem by assuming/estimating how the option affects other explanatory variables, and then analyse the effect of changes to those variables on the problem.

57. Data from previous studies, including previous policy development work, can be used to quantify impacts. There are several extensive "benefits transfer" databases of previously quantified impacts resulting from policies worldwide. Note that benefits transfer calculations should only be done by experienced economists, given the uncertainty of transferring values.

58. Revealed preference techniques that use current behaviour to estimate future behaviour:

• "Hedonic pricing" is a technique used to estimate impacts by considering their influence on market prices. If influences can be understood, the insights can be transferred. Example 1: The value of education improvements can be estimated by examining the increase in incomes (of those who have the education) that results from these improvements. Example 2: The benefits to residents from restricting industrial development to non-residential areas could be estimated by comparing the value of houses adjacent to a factory with those in the same suburb that are not. Example 3: The value of a park could be estimated by comparing the value of houses adjacent to similar parks with houses not adjacent to parks.
• "Travel costs" estimate impacts by considering the direct cost a party encounters to achieve or avoid something. Example 1: The value of a labelling requirement to consumers could be estimated by finding out how much effort a typical consumer expends to find out the information themselves. Example 2: The value of a park could be estimated by calculating how much trouble people would go to in order to visit the park.

59. Stated preference techniques that ask parties the value to them of impacts. This can include asking a party how much they would be willing to pay to accrue benefits, or would need to receive in compensation to incur costs. For example, a company operating a factory might be asked how much it would be willing to pay to continue polluting, while individuals might be asked how much compensation they would require to tolerate the pollution. Such surveys need to be carefully designed to minimise the potential for respondents to misrepresent their true valuations. In the pollution example, the company, if it believes it will not actually be required to pay to pollute, may overstate the amount it is willing to pay to continue polluting, or the individual may overstate the amount of compensation required for them to be willing to tolerate the pollution. Further, the amount individuals are willing to pay to prevent a polluting factory from locating nearby is likely to be less than they would be willing to accept as compensation for the nearby pollution.

60. Seeking expert advice. In some cases it may be useful to engage experts to help you assess the magnitude of each option's impacts. However, as with all analysis, you should ensure value judgements of experts do not influence the analysis.

5. Review the Options

61. Take an iterative approach. Inquiry into the impacts of options will tend to lead to more refined options as you realise how options can be altered to reduce negative impacts and increase positive impacts. When you change an option, you need to ensure your list of impacts remains thorough and accurate.

6. Compare the Options and Identify the Best Option

62. Once you have finalised all of the options and have a thorough and accurate list of each option's impacts, including a proper understanding of the qualitative characteristics and magnitude of each impact, you need to make a decision as to which option is best. In order to do this, you need to:

• determine what your decision criteria are (benefit cost ratio and/or net benefit) and apply the decision criteria to each of the options;
• make sure you have acknowledged your criteria and assumptions;
• test the sensitivity of your results to changes in your assumptions.

Decision Criteria

63. For a mainly textual/quantitative cost benefit analysis, the decision criterion is likely to be: Which option is the best when the benefits are compared with the costs? This will necessarily be a judgement call and will depend largely on estimates of the relative value of impacts. The more quantification you have done, the easier it will be to make a judgement. Where there is virtually no quantification, in many cases the decision as to which option is best will be very much a "best guess".

64. As set out above (see paragraph 7), in a quantitative CBA a net benefit and/or benefit-cost ratio calculation are required in order to identify which option is best.

65. Sometimes the option with the highest net benefit will not be the option with the greatest benefit-cost ratio. For example:

66. In this example, the net benefit criterion favours option one, while the benefit-cost ratio favours option two. If the options involve Government spending, generally you should choose the option with the highest benefit-cost ratio because this results in the highest net benefit over all of the Government's spending projects. The exception to this is where you are faced with two options that have higher benefit-cost ratios than proposals in other areas (for example, if transport proposals have higher benefit-cost ratios than proposals in other areas). In these situations it may be better to go with the option that has the largest scale to ensure that Government spending generates the highest net benefit overall.

67. If the proposal is regulatory, it is less clear which criterion you should prefer, but in general you should prefer the benefit-cost ratio because, like a Government budget constraint, society's resources (for example, the amount of time people can spend on complying with requirements) is finite so getting the best return for society's resources from each policy development process is likely to generate the highest net benefit overall.

68. More guidance on formal quantitative cost benefit analysis is available in Treasury's Cost Benefit Analysis Primer [link to Treasury website].

Acknowledge Your Criteria, Values, and Assumptions

69. You need to acknowledge the criteria, values, and assumptions that have determined your conclusion as to which option is best. If, for example, the sole criterion is the net benefit calculation, you should state this. However, if you have applied weightings to different costs and benefits, have used additional criteria (such as the policy objectives), or have made any assumptions (such as the discount rate - see Appendix 3), you should be explicit about this. Where there is uncertainty, you should acknowledge the assumptions you have made about what the likely scenarios are. Be careful not to set up your assumptions in such as way that they lead to a pre-destined outcome.

Testing the Sensitivity of Your Results to Changes in Your Assumptions

70. Sensitivity analysis involves varying the main assumption(s) (e.g. the discount rate) of the analysis to determine the extent to which the results are dependent on those assumptions. It also provides valuable information on reliability. It is particularly useful where there is considerable risk or uncertainty surrounding the estimates of the main cost(s) and or benefit(s). Where this is the case, the net benefit and benefit-cost ratio calculations should be repeated using other reasonable assumptions on the value of the major impacts. An option should be able to demonstrate a positive outcome under most of the relevant scenarios tested if confidence is to be had in its appropriateness.

7. Present the Results

71. In presenting the results of your assessment of the options (and showing how the preferred option would change things - i.e. by comparing the status quo with the preferred option), it is important to document the methods used to calculate the costs and benefits, including:

• the criteria you have used to come to your conclusion;
• all major assumptions;
• any weightings that have been applied to certain costs and benefits;
• deficiencies in the information used;
• the scope of the analysis;
• possible estimation biases;
• any major intangible costs and benefits; and
• any other information considered necessary for a third party to be able to assess the analysis objectively.