3. Findings from 8 In-Depth Evaluations
3.1 Overall level of compliance
Our assessment of the 8 in-depth assessments we undertook is that:
- 3 were adequate
- 5 were not adequate.
We detected two distinct groups; RIAs (and the associated RISs) were either well below what could be expected or of reasonable quality. Even so, the latter did not rate very highly. It was clear, though, that with some additional effort those RIAs and their RISs could be very good.
There is a close correlation between our assessment of the adequacy of the regulatory impact statements and that of the regulatory impact analysis. This is a helpful finding for future monitoring: if the RIS is not of high quality, chances are that the quality of the RIA was similarly disappointing. But in a few cases it was clear that there was much more substance than was apparent from the RIS.
Seven of the eight Cabinet papers we looked at had a RIS attached. All seven certified that the RIA met the adequacy requirements. Possible reasons for this are:
- capability: in a number of cases the main analyst was inexperienced at preparing regulatory impact analyses. For example, in the case of one agency the lead group would prepare a Cabinet paper only on rare occasions. (Some also mentioned that this was the first time they undertook the RIA under the new requirements)
- clarity: we detected uncertainty about the requirements; some interviewees thought the requirements ambiguous, questioned the distinction between preparing a Cabinet paper and undertaking a RIA, or were not sure whether the RIS had to be able to stand-alone or would always be read with a Cabinet paper
- feasibility: a couple of people we interviewed mentioned that it may not be feasible to sign-out a paper that states the RIA is inadequate. One agency faced the problem when it felt that due to timing constraints it had not been able to undertake adequate consultation
- tougher standards: perhaps we expect more than decision-makers do. But that is not what we have found in previous evaluations of policy advice undertaken for various government agencies when we checked with the intended audiences. In subsequent audits it may be useful to ask Ministers and their advisors directly what their impressions are.
3.2 Strengths and weaknesses of the RIAs
The overwhelming area of weakness was the analysis of scale and scope of the issue and the closely related issue of costs and benefits of change.
In only a few cases was there a serious attempt at assessing the relative sizes of benefits and costs in a qualitative sense; quantification was even rarer. A fully quantified cost benefit analysis may not always be appropriate or possible. The RIA requirements are only that the analysis must be of an appropriate level given the magnitude of the proposal. But in most cases even a basic qualitative cost benefit assessment appeared to have been put largely in the too-hard basket; basic indicators of current volumes or numbers likely to be affected by the proposal were missing. For example, it can be sufficient to state that a proposal affects 3 small firms on the one hand, or 10,000 employees, or five transactions a year. Some of that information was readily available (even mentioned in discussion documents); it had just not been exploited as part of the RIA nor documented in the RIS.
Generally speaking, the risk assessment tends to be weak also. There was just one reasonably good example of risks having been considered explicitly. This was reflected in a brief table in the RIS summarising risks and mitigation.
In at least three out of the eight cases the problem definition was weak, leading to a questioning of the need for change or the rationale for the preferred option. In three out of eight cases the key features of preferred and/or alternative options were not made clear.
Generally speaking, our impression was that consultation tended to be adequate, even though it was not always reflected in the RISs, or even Cabinet papers. In one case the compressed timeframes did not allow consultation, and in another case it appeared that one important group had not been consulted (yet).
3.3 Process followed
The interviews revealed a range of different approaches and philosophies to RIA.
Half the agencies surveyed have explicitly built RIA responsibilities into their processes. One agency set up a special RIA committee with documented processes; another has RIA as a performance expectation; another has "named" RIA experts for people to draw on for advice. RIA requirements are considered early on in the process and explicitly built in the work programme/project plan. The strong commitment to the RIA process stemmed from a belief that a good RIA would give a strong foundation to the policy advice (and thus the Cabinet paper and RIS).
The other half of departments had no special process. The people interviewed at these agencies believed that the thought-process promoted by the RIA "reflected good practice" and was "helpful" and one that "good policy analysts" would follow "intuitively". But certainly in two cases it was clear that the RIA/RIS was an afterthought.
In most agencies, the author would determine the adequacy of the RIA/RIS and then rely on the usual internal peer review and sign-out processes and consultation with other agencies. Sometimes the RIAU is consulted where the requirements were felt to be ambiguous.
In one agency the special RIA committee provides a recommendation about adequacy to the responsible policy manager, who would then decide how to proceed.
There was mixed awareness of the Code of Good Regulatory Practice; some of the interviewees indicated they had "heard of it". But three out of the eight agencies indicated that the Code was explicitly built in their process for determining adequacy.
Most interviewees were aware of the Test for Significant Impact on Economic Growth. Some interviewees mentioned that the Test was ambiguous. However, in just a few cases there was doubt about the "answer" to the test, and in those cases the RIAU was consulted.
Bar one, none of the agencies had a comment on the exemptions. The exception was a case where there was some uncertainty about the regulatory vehicle that was going to be used to progress policy options, and ultimately a call was made that a declaration about the regulatory impact assessment or the provision of a RIS was not required at this stage. The decision was made following discussion with the RIAU.
Almost all RISs were published on a website; some were hard to find or had disappeared following website revamps. Just in one case was release not approved by a Minister – it was not clear from the information gathered during the evaluation process why it was not.
3.4 Barriers, facilitators and lessons
All RIAs we rated as adequate originated from departments with explicit responsibilities or structures for RIA processes. The sample is too small to confidently state that there is a relationship. It certainly would be a big leap to assume a causal link. There may not even be any direct link.
Agencies offered a number of different reasons for shortcomings:
- time constraints: in a few cases there was only little time for the analysis or the consultation process. Sometimes the timing was imposed by Ministers or Cabinet committee process; in a couple of cases the responsibility for carrying through elements of the policy work was transferred to the authoring agency late in the process; they can then only work with the materials they have inherited
- inexperience: there were instances where the main author was new to the job or was part of a group that only write any Cabinet papers on rare occasions
- answer-driven: some interviewees believed they were working to implement an option that was already "decided" or was a "no-brainer". The implication seemed to be that more in-depth analysis of costs and benefits was therefore not necessary. In our view this does not follow
- ambiguity of requirements: There were mixed views on the helpfulness of the guidance provided. Some found the guidelines very useful, some about right, some far too high level, and yet others who felt the material too detailed with too many questions and references to be useful. A couple mentioned that direct contact with the RIAU on specifics was helpful; a couple of others felt frustrated.
Each reason may explain why a RIA may not be adequate. It does not explain why all RIAs were certified as adequate. On the basis of this sample, self-certification is not yet working as expected. It seems to us that there are few incentives on analysts or responsible managers to self-declare that their analysis is inadequate. To overcome the incentive and feasibility issue (see 3.1), one of the agencies has tried to devise sample text to allow a declaration of inadequacy that is neutral. This could be something the RIAU may consider promulgating to other agencies. An alternative would be to allow for more categories than the current fail or pass in the self-assessment.
Timing constraints and inexperienced staff (and a degree of confusion around the time the "new" requirements were introduced) are features of the more general issue of resource constraints. No agency mentioned resource constraints explicitly as an issue. Solutions here lie in the approach to internal quality assurance, as well as management. There are implications in the generic areas of staff training, looking at the robustness of the QA process, better use of expert policy advice support, as well as the process of work acceptance, prioritisation, and allocation.
Our own observation is that there are too many cross-referenced statements of requirements and "help" documents. It was also clear from the responses that a number of requirements (and in particular the Code of Good Regulatory Practice) were not well-known at all. The information in each of the help documents is reasonably helpful, but analysts would be helped if the paper trail were rationalised. That is, it should be made much shorter, with all requirements and guidance put in a single resource – for example it could be turned into a single top page with the key requirements and principles, with a small number of supporting pages providing guidance. Less is more, in this case.
Back to Top