Appendix Two: Principles for Evaluation of Interventions
1. All evaluation should apply good practice evaluation principles. The principles agreed to by Cabinet for Industry and Regional Development interventions are:
- using a range of methodologies to assess impacts of a programme or a suite of programmes where possible. This recognises the complexities in which a programme or suite of programmes operates, and how a single approach may only reveal part of the effects of a programme activity. This includes, more broadly, research to improve our understanding of policy foundations;
- design - evaluations should be designed in tandem to policy development;
- independence of approach - for objectivity; and
- timeliness -to ensure that that value of new knowledge is maximised by being synchronised with decision making.
2. Evaluation should also meet generic standards of:
- utility - that evaluation serves the information needs of users;
- feasibility - evaluation should be realistic and prudent; and
- accuracy - evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the programme(s) being evaluated;
Issues to Consider When Developing an Evaluation of Government's Framework for Sector Engagement
3. There are different types of evaluations focused on different aspects of programme/policy performance: implementation evaluation (has the programme/policy been implemented as specified); effectiveness evaluation (is the programme/policy meeting its objectives); and impact evaluation on broader government outcomes that may sit above this.
4. When considering effectiveness and broader outcome evaluation, stability and maturity of the government's policy framework, along with recognition of time lags, are critical for meaningful design of outcome evaluation for sectors. Evaluation milestones must recognise the longer-term view. In the short term, evaluation activity is likely to focus on monitoring of achievement of sector engagement activities. In the longer term, evaluation, in conjunction with relevant research, is more likely to assess whether progress is being made towards policy objectives and broader outcomes.
5. Accuracy of attribution of the programme of government's activities on sector performance (however defined) will be difficult given that multiple factors affect sector performance, as well as the potential multiplicity of activities which will support this policy initiative. The approach used will need to balance any expectation of quantifying the value added to sector performance, versus the benefit, ability and cost of doing so.
6. Evaluation activity does not need to be seen as an arbitrary quantification of "outcomes". Other evaluation questions include "why" something is working in contrast to "whether" something is working. An investigation of why something is working may be appropriate depending on whether we want to improve the instruments used to address the policy problem/opportunity.
7. The evaluation must take a wider-context and clear strategic direction. This means, among other things, interfacing and positioning any proposed evaluation with the current evaluations for sector engagement and support activities undertaken by NZTE (planned for 2006), and evaluation activities related to the implementation and outcomes of the GIF. This also means anticipating how future policy development work on the GIF will effect what is proposed as part of sector engagement evaluation.
8. Related to the previous point, the evaluation should use appropriate existing sources of information valid for the evaluation's defined purpose before commissioning new pieces of research and evaluation activities. There is an increasing expectation from Ministers and Treasury that selected evaluation of government expenditure is undertaken. In this context, we must ensure that synergies related to information currently being collected are used as appropriate. We wish to flag at the outset, however, that there are existing limitations to available data sets we use currently to monitor sector performance.
Approach to Be Taken
9. It is inappropriate to specify an evaluation methodology within this position paper because the methodology must drop out of the policy development phase for sector engagement. A specific evaluation and research plan will be initiated through the policy phase, and will consider, among other issues:
- the evaluation purpose, questions to be answered, and how the evaluation results will be used;
- statement of outcomes to be assessed for the policy's objectives, and related effectiveness measures;
- statement of any aspects of delivery to be assessed, and related measures;
- an indicative evaluation methodology as to how added value from the policy will be evaluated;
- an indicative data collection and analysis strategy;
- what reports are to be produced, who are responsible for these, and when will they be produced; and
- the evaluation and research plan will reflect good practice principles for evaluation planning and design.
Back to Top