4. Review of Regulatory Impact Statements
We assessed 12 RISs in terms of their effectiveness in communicating necessary information to decision-makers and other interested parties. We assessed each RIS according to the framework in Table 1.
To give a sense of the spread of quality we gave each RIS a mark out of 10, with 5 being poor, 7 satisfactory, and 8 good. This mark is an experience-based judgement on how well the RIS overall does its job as a stand-alone piece. These judgements are made without taking into account any pressures and constraints faced at the time. This is because we do not have information about pressures and constraints, and also because to a customer (the reader) such explanations are generally not valid justifications; management is there to resolve such constraints.
Strictly, the review of RISs is not intended to make a judgement on whether the RISs are adequate as set out in the RIA Guidelines. But, looking at the correlation between the quality of the RIS and the RIAs we looked at in-depth, those RISs with a mark of 7 or higher are likely to correspond with adequacy.
4.1 Overall findings
Overall, the results are disappointing. We rated well over half of the sample as failing in their task; 42% we rated as satisfactory or better. A similar proportion was rated as poor. Just one paper we rated as good. See Figure 1.1
Figure 1: Summary of assessment results
Percent of Regulatory Impact Statements
4.2 Areas done well
The majority of the RISs we assessed displayed the following strengths:
- Short. RISs tended to span about 4 pages which appeared to be about the appropriate length for the issues under consideration.
- Plain language. There was a welcome absence of unnecessary jargon and bureaucratese. But in a number of cases the problem could be explained better.
- Consultation. Most of the RISs showed evidence of a reasonable amount of consultation with stakeholders as part of the process of developing the advice. We also saw some pithy and carefully worded summaries of issues identified in the consultation and how they were dealt with.
Most RISs used the template provided and used the appropriate headings. Where they did not, it tended to be because the RIA/RIS process had begun before the 2007 changes to the approach. The RISs we judged to be satisfactory tended to be useful as stand-alone documents.
4.3 Areas for improvement
The main weaknesses in the sample are as follows:
- What is broken? A few papers struggled to communicate what the problem really was. Often this seemed to be related to a lack of a credible framework. This then also undermined the rationale for change and option selection, particularly when combined with an absence of cost-benefit analysis.
- How big and who is affected? Too many papers struggled to give a sense of the scale of the issue, and most did not consider how different groups would be affected. This weakness can often be resolved with only a limited amount of effort, citing for example the number of adverse events, or the number of entities or people affected.
- Costs and benefits? The biggest concern is the number of proposals in this sample that omitted a credible assessment of relevant costs and benefits, whether they be financial or in terms of other outcomes (health, freedom, etc). For example, how effective in qualitative or quantitative terms would a proposal be at reducing adverse events. Often the omission betrayed a lack of a framework that would have brought out the trade-offs, for example, between accuracy and cost or the cost and benefits of dealing with a regulation that captures some people or activities it shouldn't, or does not catch all those it should.
- Risks? Just a few statements explicitly considered key policy risks and how these would be managed. None really outline the risk in the analysis (strength of evidence, etc.).
- Stand-alone? The poorer quality RISs did not convey the necessary information to be of much use as a stand-alone document. (In some of the RIA-related interviews we conducted it transpired that there a lack of awareness that the RIS should be a stand-alone document).
Back to Top