Executive Summary
Background
1. The Maori Tourism Facilitation Service (MTFS) was established in October 2004 with a budget allocation of $580,000 (inclusive of GST) from Vote Tourism for the period April 2005 to June 2006. Decisions about the future of the programme beyond this period will be based on the effectiveness of the programme and the availability of resources.
2. In February 2005 Te Puni Kokiri (TPK, the Ministry of Maori Affairs) was appointed as the delivery agent of the MTFS and the Ministry of Tourism as the purchasing agent. A programme pilot commenced in April 2005 in Auckland and the Bay of Plenty with 20 businesses selected to participate. In August 2005, the wider rollout of the programme commenced with an additional 60 businesses targeted nationwide.
3. The aim of the programme is to provide assessment and mentoring to Maori tourism businesses in early development (i.e. post start-up) and established phases to improve individual performance and to ensure delivery of a quality tourism experience.
Method
4. This evaluation was based on multiple sources of information and perspectives, including interviews with 21 of the participating businesses, interviews with the programme delivery staff and contractors (two account managers, seven assessors, and two mentors), review of clients' Business Development Plans, review of clients' business snapshot results, TPK's MTFS client database, monthly reports from MTFS Account Managers, and interviews with key stakeholder agencies.
Key Findings
Outputs and participants
5. From commencement of the programme pilot in April 2005 to June 30, 2006 a total of 84 businesses have been enrolled in this programme. This exceeds the target of 80 enrolments in the programme by June 30, 2006. However, 14 of the clients have withdrawn from the programme, leaving a total of 70 clients currently in the programme.
6. Most of the businesses participating in the MTFS are small and relatively young businesses – i.e. less than five years old, have fewer than five employees, and annual turnover of less than $100,000.
7. Most (i.e. 79%) of the MTFS participants have had a Business Development Plan prepared and signed-off. However, only 28 (i.e. 40%) have only been assigned Mentors as of June 30, 2006. It is estimated that at least 50% of clients will still be in the mentoring phase in June with most completing July or August.
8. The delay in the mentoring phase resulted mainly from programme implementation and start-up taking longer than anticipated (e.g. in terms of recruiting and training Assessors and Mentors) and the limited availability of clients to engage with the programme during the peak tourist season from November 2005 through March 2006.
Outcomes
9. The majority of MTFS participants appeared to be satisfied with the programme as a whole (80%) and with the extent to which they perceive their Business Development Plans address the needs of their businesses (76%). These satisfaction rates are a good result for the programme's start-up year. Given that many of the teething problems associated with start-up have been ironed out, the programme management expect this satisfaction rate to improve in the programme's second full year.
10. Over 60% of MTFS participants interviewed for this evaluation indicated they had already experienced improvements in the intended intermediate outcomes of the programme (i.e. market knowledge and development, strategic planning, and product development) between the time they started the programme (June to October 2005) and when they were interviewed in April 2006. The participants expect further improvements in these outcomes by the time they complete implementing the recommendations in their Business Development Plans.
11. Over a quarter of MTFS participants indicated they have already experienced improvements in the various intended ultimate outcomes of the programme, i.e. revenue, product reputation and quality, and business sustainability, between the time they started the programme (June to October 2005) and when they were interviewed in April 2006. Over 40% expected further improvements in these outcomes by the time they have completed implementing the recommendations in their Business Development Plans.
Implementation and delivery
Programme cost
12. The 2005/06 budget for the MTFS programme is $650,000 consisting of $390,000 funded by Vote Tourism for assessment and mentoring services, and $260,000 in TPK resources for salaries and overheads for two FTE Account Managers. This works out to an average budget of $8,125 per client (based on 80 clients enrolled and assessed). It is recommended that TPK continue to monitor the cost of this programme and to look for opportunities for efficiency improvements.
Eligibility criteria
13. Some MTFS participants did not appear to meet one of the necessary criteria for programme eligibility i.e. willingness to participate fully in the programme. Between a quarter and a third of the MTFS participants did not meet some of the discretionary eligibility criteria of having turnover in the previous year of greater than $30,000, the business being the business operator's primary source of income, and the business having growth potential.
14. Although Account Managers are provided with some discretion to grant exemptions from the discretionary criteria, the number of MTFS participants who were granted these exemptions appears too high. These criteria are important for ensuring that the MTFS programme complements rather than duplicates other programme and services, by serving its target audience of developing and established Maori tourism businesses focused more on product development issues, rather than start-up or part-time businesses that tend to be focused more on basic business capability issues.
Coordination and complementarity with other programmes and services
15. The MTFS appears to be playing an important role in helping Maori tourism businesses access the services they need to improve business capability. The programme appears to be well coordinated with and complement the role of key stakeholders and related service providers in three key aspects:
- Participant selection- businesses participating in the MTFS were selected by TPK based on recommendations from the New Zealand Maori Tourism Council, Poutama and Maori Experience NZ.
- Assessment and mentoring– most of the assessors and mentors engaged to work with MTFS clients have experience with other business assistance providers such as Poutama, Maori Business Facilitation Service, Qualmark, and New Zealand Trade & Enterprise. They are also actively involved with Maori Regional Tourism Organisations and/or are known within the tourism industry as tourism consultants.
- Referrals – In addition to providing direct assistance to Maori tourism businesses, one of the key aims of the MTFS is to increase the ability of these businesses to access mainstream business assistance programmes as needed. Among the 21 clients in the interview sample, client feedback and reviews of their BDP indicated that the over 60% received referrals for assistance with more generic business issues from other government and/or private programmes such as the Enterprise Training Programme by New Zealand Trade and Enterprise (NZTE), TPK's generic Maori Business Facilitation Service, and Poutama Maori Business Trust. The MTFS is intended to serve as a stepping stone which helps prepare businesses to access more advanced programmes when needed, such as NZTE's business assistance grants.
Overall conclusion
16. The MTFS programme appears to be well designed and the client engagement and assessment components seem to be delivered well with the majority of clients reporting satisfaction and improved capability from the programme. This present evaluation, however, was not able to fully assess the effectiveness of the programme as most of the participants have only started the mentoring phase of the programme in May or June 2006. This early evaluation therefore recommends that this programme be continued, with some minor policy and operational improvements, until a full impact evaluation is conducted in 2008.
Recommendations
Evaluation
- If the MTFS programme is continued, it is recommended that a full impact evaluation of this programme be conducted by July 2008 to examine the following questions:
- To what extent is the MTFS achieving the intended policy outcomes?
- Would the outcomes have been achieved without the programme? How much value is the MTFS adding?
- Is the MTFS well coordinated with related government and private business assistance services?
- Is the MTFS filling an important need?
- Is the delivery of the MTFS efficient compared over time and to similar programmes?
- What are the opportunities for improvement?
- Can the MTFS model or lessons from it be applied to other parts of the tourism sector or other business assistance programmes?
Policy improvements
Target group
- To continue to avoid duplication with other existing programmes and services, there needs to be increased clarity in communicating the policy intention that the MTFS programme is not designed for start-up businesses but only for Maori tourism businesses that are developing or established and focused on business growth, rather than more basic business capability issues.
Intended programme outcomes
- The specific outputs and outcomes intended from the MTFS were identified from within relevant policy documents, and where required, were clarified through consultation with the Ministry of Tourism and TPK. These are set out in Section 1.5 - Programme Logic Model. Based on the lessons learned from this evaluation, some refinements are proposed to this logic model to increase clarity in terms of intended key outputs, immediate, intermediate and ultimate outcomes. The revised logic model is presented at the end of this report.
Eligibility criteria
- Tighten the criteria regarding length of time in business from a six month minimum to a minimum requirement of 12 months in business.
- Clarify that to be eligible for the MTFS programme, all participants must meet the necessary criteria of being willing to participate fully in the programme by investing time and/or money into assessment and/or mentoring.
- Clarify that for a business that does not already incorporate Maori aspects (e.g. tikanga, te reo, cultural performances etc.) as part of its products and services, but is considered to have the potential to incorporate Maori aspects, this potential needs to be demonstrated as part of their Business Development Plan.
- Clarify the growth potential criteria with an objective definition would enable better application of this criteria. It is proposed that ‘growth potential' be defined as having a willingness and plan to develop the business to achieve annual growth exceeding a target rate. This target should be defined by TPK with assistance from the Ministry of Tourism and Ministry of Economic Development.
- Consider setting a limit (e.g. no greater than 10%) on the number of businesses granted exemptions from the discretionary eligibility criteria.
Operational improvements
Initial assessment
- Revise the Business Snapshot tool currently used for initial assessment of participants to ensure it identifies all of the critical issues for the business, i.e. the financial status (e.g. profitability, debt levels), ownership, governance, and growth potential of the business.
- Request participants to present the financial statements of their business for the past year or two (for older businesses) to enable accurate assessment of financial status.
Transition between Account Managers, Assessors, and Mentors
- Improve the transition to Mentors from Account Managers and Assessors and minimise the potential duplication between these roles through better briefing among the parties at key transition points.
- Once the mentoring phase has been completed by the current participants, TPK should review the roles of the Account Managers, Assessors, and Mentors to identify if there are opportunities to improve efficiency and effectiveness.
Referrals to other services or training
- To ensure that Account Managers and Assessors are able to make appropriate and timely referrals when necessary, it is important that there is a systematic process to keep them up to date with the list of mentors and programmes or services of other providers available to work with MTFS clients.
Mentoring
- Within the budget constraints of the programme, ensure that the timing, intensity, and nature of mentoring provided to participants is tailored to the client need.
Information sharing with key stakeholders
- In order to maximise the potential value of information collected by the MTFS programme in informing the development of policies and programmes for the Maori tourism sector, it is recommended that TPK explore opportunities to improve information exchange with key stakeholder organisations such as the New Zealand Maori Tourism Council.
Monitoring and evaluation of programme outcomes
- Continue to measure, record and monitor the programme outcomes in terms of business capability and performance of participants at the start of engagement, at regular intervals during engagement, and up to two years after completion of the programme.
Back to Top