Annex E: Design Principles and Evaluation
Designing effective interventions is not easy. There have been many examples of programmes that have failed to achieve their objectives. Applying a set of design principles can increase the chances of interventions that work. Such principles, which are outlined in Annex E, will be applied to the design of the operational detail of all new programmes
- programme objectives are clearly defined and able to be evaluated
- programmes are accurately targeted
- compliance costs are as low as possible (programmes are easily accessible, eligibility requirements are clear)
- programmes create "additionality". This means that they encourage activity that would not otherwise have taken place
- programmes are flexible to meet differing and evolving needs (one size does not fit all). In particular, it is important that all programmes are accessible and relevant for the needs of Maori, Pacific Peoples and women
- evaluation is built into programme design and implementation and programmes are effectively evaluated on a regular basis
- using public-private sector partnerships wherever possible (contestable delivery, cost-sharing, sponsorship etc)
- programmes include consideration of capability development for programme deliverers
- effective coordination between programmes
- consistency with New Zealand's international trade policy obligations
- consistency with the Government's goals for sustainable development.
Monitoring and Evaluation
Programme evaluations conducted both on individual programmes, and groups of programmes, are a crucial device for bringing greater coherence to industry development policy.
A programme evaluation is fundamentally about measuring the impact of a programme on its participants. A best practice approach, which will be applied to the development of new industry development initiatives, contains all the following components:
- Implementation Evaluation - is the delivery and administration system adequate; do providers have the capability to deliver; are they meeting their targets?
- Impact Evaluation - does the programme achieve its objectives?
- Efficiency Evaluation - does the programme, or elements of the programme, achieve the objectives at a lower cost than other feasible alternatives, or at least a low cost for the return provided?
- Policy Evaluation - assesses the validity of the rationale behind the programme; are the market failures/problems the programme is designed to meet being addressed and do the failures still exist?
In broad terms, there are five factors central to a well-designed evaluation:
- Independent approach - a review of programme effectiveness is best undertaken by groups independent of the administration and delivery of the programme.
- Adequate resourcing - ensuring adequate funding and resources for evaluation at the outset is an important issue.
- Range of methodologies - Industry development initiatives can have a variety of effects, including social, economic and environmental. Generally, assessing the benefits of an industry development programme is difficult because:
- it is very difficult to establish causal relationships between government support and final economic outcomes, and to distinguish actions taken solely on the basis of programme recommendations from those that would have taken place anyway or undertaken for other reasons
- the benefits of any business assistance, such as upskilling, will often flow to the client in the long-term (which may be a number of years). The costs of the programme, however, are often relatively short-term
- many of the firms that are successful in obtaining assistance from government assistance programmes are more outward looking in the search for assistance and more willing to take on assistance (self-selection). This contrasts with firms for which programme assistance may be an important factor for improving performance but which are slower or unable to obtain assistance. The important requirement for any programme is to ensure that these firms are able to fully participate
- an important distinction needs to be drawn between net and gross effects. Assessment of a programme's gross effects includes basic measures such as counting the number of clients participating. To arrive at the net effects (the additionality), the dead-weight factor (activities that would have occurred anyway in the absence of the programme) and the displacement effect (where participating firms improve their performance (say employment, sales) by reducing those of competing but non-participating firms should ideally be estimated. Assistance programmes may redistribute benefits rather than create benefits.
In practice, no one evaluation methodology is ideal. It is thus important to get information from a range of sources rather than relying on one method. Evaluations should involve empirical investigations, such as surveys of participants and analysis of data relating to the programme's targets, and qualitative research using interviews and case studies.
- Design - it is also important to design the evaluation together with the implementation of the programme, as early preparation is needed to secure the collection and provision of the data needed during the evaluation.
- Timeliness - rather than being ad hoc, evaluations need to be carried out regularly, as they have substantial learning effects. In terms of the four components outlined above, implementation, impact and efficiency should be reviewed annually, and more detailed impact and policy analysis should be considered three-yearly.
Back to Top