4.Findings: Outcomes
This chapter presents findings on the achievement of the intended immediate, intermediate, and ultimate outcomes of the MTFS programme as set out in the Programme Logic Model on Table 2 in Chapter 1 of this report. These findings are based on the interviews with 21 clients (described in Chapter 2).
4.1 Immediate outcomes
4.1.1 Client satisfaction with the MTFS programme
Figure 10
As shown in Figure 10, most (80%) clients are satisfied with the programme. About half (57%) indicated high satisfaction (i.e. with a satisfaction rating of 8 or higher). Some reasons for the satisfaction are illustrated by the following quotes:
"Assessor and mentor went over and above what they were contracted to do. Loved the mentors assigned to me. I'm very happy with the quality even though I hadn't heard of them before. I couldn't have selected better people myself."
"The Assessor took time to talk and the BDP was excellent."
"MTFS doubled my product price from $60-$80, to $120. It helped me enter a different market. They had higher expectations but we were able to meet them. It doubled the volume of my business!"
"It is wonderful to be part of the programme."
The four (20%) dissatisfied clients did not appear to be significantly different from the rest of the interviewed clients in terms of region, type of business, and revenue size. However, while 60% of interviewed clients reported revenue growth in turnover between start of engagement in 2005 and when the interview was conducted in April 2006, none of the four dissatisfied clients reported revenue growth during this period.
The 80% satisfaction rate is a good result for the programme's start-up year, in-spite of some teething problems which caused delays in implementation. According to the Programme Manager, the delivery of the programme has improved with the process now well mapped out, better flow, increased consistency, and better transitions between initial assessment, preparation of Business Development Plans, and mentoring. The Manager expects that the client satisfaction rate will improve further and achieve a target of 95% by the end of year two.
4.1.2 How well BDPs addressed the needs of clients' businesses
Figure 11
Most (76%) clients are satisfied with the extent to which their Business Development Plans (BDP) address the needs of their businesses (see Figure 11). Only 20% appeared dissatisfied. According to one Assessor, some clients find that the process of preparing a BDP validates their own assessments and strategies, while others find it enlightens and helps them identify opportunities or challenges they have not previously seen.
4.1.3 Priority areas for Business Development Plan (BDP)
As described previously, of the 84 businesses enrolled in the MTFS programme, 55 (79%) businesses have had a BDP prepared and signed-off by the Account Manager. Figure 12 below shows the development areas identified as priorities in these 55 BDPs. Analysis by the Account Managers showed that the most frequently identified priority area for business development among the MTFS participants is marketing and market development followed by financial management, HR/business support systems, and product development or improvement. Of the six priority areas identified, the issues more likely to require tourism specific mentoring concern marketing and market development, strategic and business planning, and product development or improvement. For areas such as financial management, HR, business support and IT systems, generic business mentoring is likely to be sufficient.
Figure 12
4.1.4 Completion of actions set out in BDP
As stated earlier in this report, at the time of this evaluation, most clients have not yet entered the phase of receiving mentoring assistance in implementing the recommendations set out in their BDPs. However, according to the Account Managers' report, 13 clients have completed at least one priority area in their BDP. The Account Managers estimated that at least 50% of clients will still be in the mentoring phase in June with most completing July or August.
4.1.5 Other business assistance programmes engaged through MTFS referrals
One of the key objectives of the MTFS programme is that it is aligned with existing regional initiatives or programmes. The September 2005 Implementation Review of the MTFS noted that it is expected that MTFS providers will be aware of and work collaboratively with other existing regional initiatives and programmes. This point has been stressed at assessor training workshops. Regional participant selection has taken account of other existing regional initiatives available to Maori tourism operators.
Among the 21 clients in the interview sample, client feedback and reviews of their BDP indicate that the majority (62%) received referrals to other government and/or private programmes. Almost half (48%) received referrals to Poutama Maori Business Trust.
About one in five (19%) received referrals to New Zealand Trade and Enterprise (NZTE)'s Enterprise Training Programme which provides workshops and follow-up coaching (e.g. Bruce Cheeseman workshops in Rotorua for generic business issues and Catalyst Training tourism courses by Craig Wilson in Taranaki). One Assessor commented that these workshops are a good way to address generic business issues, enabling the MTFS mentoring (which is limited to $3000 worth of mentor time per client) to focus on specific issues that cannot be easily accessed through other services. The NZTE workshops also provide opportunities for clients to learn from and network with their peers, and access follow up coaching. Apart from Enterprise Training courses, most MTFS businesses do not access other NZTE programmes as these businesses are very small and young, and do not fall within the criteria required for assistance from NZTE programmes such as grants for high growth businesses.
The client interviews indicated that some MTFS businesses wanted to attract investors to grow their businesses. Some MTFS participants have or are accessing NZTE's Investment Ready Training (part of the Enterprise Training Programme). NZTE also operates the Escalator programme which provides assessment and advice on investment readiness, workshops on capital raising, and deal brokering assistance to raise capital up to $5m from the venture capital, private investor or corporate investor communities. As a result of the Ministry of Economic Development's recent evaluation of the Escalator programme, linkages will be strengthened between the Investment Ready Training and the Escalator programme. This should increase the ability of eligible MTFS businesses to access the Escalator programme, if needed.
Some businesses which do not meet the eligibility criteria for MTFS are referred, when appropriate, to TPK's other programme, the Maori Business Facilitation Services (MBFS) which provides mentoring on generic business issues mainly for start-up businesses. Businesses which meet the eligibility criteria for MTFS may also be referred to MBFS Mentors (called Accredited Business Mentors) for generic mentoring if the BDP process identifies that these businesses need assistance with more generic business issues e.g. financial management. The September 2005 Implementation Review noted that many participants did not know of the MBFS prior to their contact with the MTFS. Engagement has created awareness of, interest in, and participation in the MBFS programme.
4.2 Intermediate outcomes
4.2.1 Market knowledge and development
4.2.1.1 Capability improvement
The majority of the interviewed clients (18 out of 21) indicated that improving marketing capability was a key priority for their business.
Figure 13
Figure 13 above shows that 61% of the interviewed clients indicated they experienced improvement in marketing capability between the start of engagement in 2005 and April 2006 when they were interviewed. A larger group (78%) indicated they expect further improvements between 2006 and when they complete implementing the recommendations in their BDP in the future.
Figure 14
Figure 14 above shows the ratings of each client on where they thought their capability was at during the start of engagement in 2005, in April 2006 when they were interviewed, and in the future when they complete implementing the recommendations in their BDP. The average ratings across all interviewed clients improved from 5 at the start of MTFS engagement to an average of 6.5 when interviewed (after completed assessment and developed BDP) and expected improvement to 8 in the future when they have implemented the recommendations in the BDP.
4.2.1.2 Have own website
Among the 21 clients who were interviewed, 70% reported they already have a website for their business, and the others have websites under development with expected completion within the next 6 months.
4.2.1.3 Registered on www.newzealand.com
Among the 21 clients who were interviewed, 68% indicated they are already registered on The official site for New Zealand Travel & Business [external website], a website dedicated to providing tourism, travel and other information about New Zealand. One business is expecting to be registered within two months, and the other 26% did not appear to have plans to be registered on this site. The rate of registration on The official site for New Zealand Travel & Business [external website] appears to be much higher among MTFS participants than among Maori tourism business that are not in MTFS. Data from NZMTC indicates that only 19% of the 357 businesses in their database are registered on this website.
4.2.1.4 Media coverage
Among the 21 clients who were interviewed, 57% indicated their business has received positive media coverage and 29% indicated they have not had any media coverage. The others did not respond.
4.2.1.5 Offshore marketing
According to the Account Managers, most of the MTFS businesses had not participated in offshore marketing initiatives.
4.2.1.6 Wholesale tourist operators
Most of the clients who were interviewed did not seem to be aware of the size and potential impact of the wholesale tourist operators as a source of business.
4.2.2 Strategic planning, management and business operations
4.2.2.1 Strategic and business planning
About half of the interviewed clients (10 out of 21) indicated that improving strategic planning capability was a key priority for their business.
Figure 15
Figure 15 above shows that majority (70%) of the interviewed clients indicated they experienced improvement in planning capability between the start of engagement in 2005 and April 2006 when they were interviewed. An even larger group (90%) indicated they expect further improvements between 2006 and when they complete implementing the recommendations in their BDP in the future.
Figure 16
Figure 16 above shows the ratings of each client on where they thought their capability was at during the start of engagement in 2005, in April 2006 when they were interviewed, and in the future when they complete implementing the recommendations in their BDP. The average rating improved from 4.3 at the start of MTFS engagement to an average of 5.9 when interviewed (after completed assessment and developed BDP) and expected improvement to 7.25 in the future when they have implemented the recommendations in the BDP.
4.2.2.2 Business systems/processes
Only two of the interviewed clients indicated that improving business systems (such as IT systems) capability was a key priority for them. Figure 17 below shows that both clients indicated they improved between the start of engagement in 2005 and April 2006 when they were interviewed. Only one of the two clients expects further improvements in the future when they complete implementing the recommendations in their BDP. The average ratings across both interviewed clients improved from 4 at the start of MTFS engagement to an average of 6.5 when interviewed (after completed assessment and developed BDP) and expected improvement to 7.25 in the future.
Figure 17
4.2.2.3 Financial systems
Five of the interviewed clients indicated that improving financial systems capability was a key priority for them. Figure 18 shows that 3 of the 5 clients indicated they improved between the start of engagement in 2005 and April 2006 when they were interviewed and all 3 expect further improvements in the future when they complete implementing the recommendations in their BDP. However two of the clients indicated they neither experienced nor expected any improvement. One of them was a client who indicated no improvement on all dimensions explored in interview and was very dissatisfied with the programme- the client had little education and probably lacked capability required to benefit from the programme- wanted more close and regular mentoring). The average ratings across both interviewed clients improved from 4.3 at the start of MTFS engagement to an average of 6.8 when interviewed (after completed assessment and developed BDP) and expected improvement to 8.1 in the future.
Figure 18
4.2.2.4 Human Resources
Only two of the interviewed clients indicated that improving human resources capability was a key priority for them. Figure 19 below shows that both clients indicated they improved between the start of engagement in 2005 and April 2006 when they were interviewed, and both expect further improvements in the future when they complete implementing the recommendations in their BDP. The average ratings across both interviewed clients improved from 5 at the start of MTFS engagement to an average of 7 when interviewed (after completed assessment and developed BDP) and expected improvement to 8.75 in the future.
Figure 19
4.2.3 Product development
Over a third (38%) of the interviewed clients indicated that product development was a key priority for their business.
Figure 20
Figure 20 above shows that majority (75%) of the interviewed clients indicated they experienced improvement between the start of engagement in 2005 and April 2006 when they were interviewed. Just over 60% indicated they expect further improvements between 2006 and when they complete implementing the recommendations in their BDP in the future.
Figure 21
Figure 21 shows the ratings of each client on where they thought their capability was at during the start of engagement in 2005, in April 2006 when they were interviewed, and in the future when they complete implementing the recommendations in their BDP. The average rating improved from 6.7 at the start of MTFS engagement to an average of 8.1 when interviewed (after completed assessment and developed BDP) and expected improvement to 8.9 in the future when they have implemented the recommendations in the BDP.
4.2.4 Increased integration of business with the industry
Information gathered by the New Zealand Maori Tourism Council (NZMTC) on the 357 Maori tourism businesses in their database indicated that about 40% of these businesses are not actively linked into the industry, i.e. they do not have a website, are not listed on The official site for New Zealand Travel & Business [external website], are not listed in the Rough Guide to Maori New Zealand (a travel guide), have not participated in an international media programme, are not a member of the Tourism Industry Association of NZ, are not a member of the Inbound Tour Operators Council NZ, have not attended TRENZ, and are not Qualmark rated. The NZMTC information also indicated that the MTFS programme is engaged with businesses who are more actively integrated into the industry, which is consistent with the policy intention for this programme.
4.2.4.1 Maori Regional Tourism Organisation (MRTO) or other tourism clusters
At the time of enrolment into the MTFS programme, TPK client data indicates that the majority (81%) of the businesses are active members of a regional tourism group or organisation. Data from the NZMTC indicates that about 66% of the 357 Maori tourism businesses in their database were members of MRTOs.
The evaluation interviews examined the extent to which MTFS participants are playing a leadership role in their local MRTO or tourism cluster. Figure 22 shows that a third of the interviewed clients indicated they experienced improvement in leadership in Maori tourism between the start of engagement in 2005 and April 2006 when they were interviewed. About two-thirds indicated they expect further improvements between 2006 and when they complete implementing the recommendations in their BDP in the future.
Figure 22
The NZMTC has found a wide variation in the ability and willingness of Maori tourism operators to play an active role in contribution to, or leadership within, the Maori Tourism sector. Some operators may play an active role in the sector even though their own tourism business is not performing well financially. For these operators, the MTFS programme would help them improve their business capability and performance and increase their ability to play a leadership role in the sector. However, there are other operators whose businesses are performing successfully but are not interested in leadership in Maori tourism.
4.2.4.2 Tourism Rendezvous New Zealand (TRENZ)
Among the 21 clients who were interviewed, the majority (71%) indicated they have not participated in Tourism Rendezvous New Zealand (TRENZ) which is New Zealand's most significant annual international tourism tradeshow. This event aims to deliver unique business opportunities for an extensive range of New Zealand tourism operators and international tourism buyers from around the globe. The Account Managers reported that some of the businesses had attended TRENZ as collectives or with the MRTO or RTO. Two (i.e. 9.5%) of the interviewed businesses thought that TRENZ was not suitable for their businesses. The rate of participation in TRENZ among MTFS clients appear to be higher than Maori tourism businesses that are not in MTFS. Data from NZMTC indicates that among the 357 Maori tourism businesses in their database, 86% have not attended TRENZ.
4.2.5 Qualmark accreditation
Another indicator of the quality of a tourism business is having the Qualmark® license which is New Zealand tourism's official mark of quality. All accommodation and tourism businesses which carry the Qualmark® have been independently assessed as professional and trustworthy, so customers can book and buy with confidence. Qualmark accreditation focuses on business practices, safety standards, and overall quality of product and service. The September 2005 Implementation Review reported that there were only 16 Qualmark® licensed Maori tourism businesses nationwide at that time. According to the NZMTC, as at June 2006, this has increased to 48 out of the of the 357 Maori tourism businesses in their database. Of the 48 Qualmarked Maori tourism businesses, half are MTFS participants.
Among the 21 MTFS clients interviewed for this evaluation, a third indicated they have Qualmark status. Another third of the interviewed MTFS clients indicated they are working towards achieving this status within the next two years. One Assessor commented that Qualmark certification is a long way off for some clients. A third of interviewed clients indicated they have no plans to get Qualmark status within the next two years. Of these two participants (i.e. 9.5% of 21 interviewed clients) said they were not interested:
"No. Not worthwhile since it doesn't differentiate." [A 5 year old water activities tourism business with over $500,000 in annual turnover, and estimated that they have experienced 12% revenue growth between start of MTFS engagement in 2005 and April 2006, and predict another 10% growth when they have completed implementing their BDP by 2008.]
"No. Don't see point, since they've operated all these years without it, and the problem with labour is the dominant one." [A 29 year old land activities tourism business with less than $50,000 annual turnover which may no longer be viable due to death of business partner. This business estimated that they have experienced 17% revenue decline between start of MTFS engagement in 2005 and April 2006, and predict another 60% decline by 2008.]
4.2.6 Other intermediate outcomes
Interviews with other key stakeholders provided indication that the MTFS programme is also achieving the following other intermediate outcomes identified in the Programme Logic Model in Section 1.5 of this report:
- Add to the credibility/profile of TPK's Maori Business Facilitation Service (MBFS): The September 2005 Implementation Review noted that many participants did not know of the MBFS prior to their contact with the MTFS. Engagement with MTFS has increased awareness of and participation in the MBFS programme.
- Provide improvement ideas for TPK's MBFS: TPK Programme Managers commented that from the experience of operating the MTFS programme, they have identified lessons from the MTFS model, and made improvements to the MBFS programme.
- Provide Tourism New Zealand with increased confidence in marketing wider range of Maori tourism experiences: A Business Development Manager at Tourism New Zealand commented that the participation of a business in the MTFS programme gives the agency greater confidence in marketing that business offshore. The agency has confidence that participation in MTFS will increase the capability of the Maori tourism business for a few reasons - the programme is not a private sector money-making venture, is focused at addressing capability issues at the right level, and the MTFS assessors and mentors are well known and respected within the tourism industry.
4.3 Ultimate outcomes
4.3.1 Business growth in revenue and FTEs
Figure 23
Figure 23 above shows that the majority (over 75%) of the interviewed clients expect that participation in the MTFS will increase the revenue growth of their business especially when they have completed implementing the recommendations from the BDP (by 2008).
The MTFS appeared to be already having an impact on some clients. During the period from start of engagement in 2005 to April 2006:
- 25% indicated their revenues grew by over 20%;
- 60% indicated no growth and 5% (1 client) indicated revenue decline (not due to MTFS, but rather due to death of key business partner, and difficulties in replacing him); and
- Growth estimates varied widely among the 21 interviewed clients from -17% to 220%. The median was 0%.
Over the period 2006 to 2008 when they have completed implementing the recommendations in their BDPs, clients have the following growth expectations:
- Over 75% expect growth;
- 43% expect over 20% growth; and
- Growth expectations from 2006 to 2008 varied widely among clients with a median of 15%.
The revenue growth experienced and forecasted is greatly influenced by macroeconomic factors as pointed out by one client: "revenue has gone down as tourist numbers have declined. No longer flavour of month as Lord of the Rings stimulation has worn off and the New Zealand dollar is high".
Only four of the interviewed clients provided FTE growth estimates. Of these, only one indicated their FTE numbers grew from 2005 to 2006, and 3 indicated they expect growth in FTEs from 2006 to 2008.
4.3.2 Improved quality of product and service received by customers
Figure 24
Figure 25
Figures 24 and 25 above show that majority (71%) of the interviewed clients indicated they did not experience improvement in the reputation of their product among customers or peers (as measured by how they thought their customers or peers rated the quality of their product) between the start of engagement in 2005 and April 2006 when they were interviewed. Less than half indicated they expect improvements between 2006 and when they complete implementing the recommendations in their BDP in the future. This low experience and expectation of improvement could be due in part to client perception that their products already have a good reputation, as shown in the next Figure. This suggests a high degree of self confidence in their products and services, which is an important attribute for business success.
Figure 26
Figure 27
Figures 26 and 27 above show the ratings of each client on how they thought their customers and peers rated the quality of their products at the start of engagement in 2005, in April 2006 when they were interviewed, and in the future when they complete implementing the recommendations in their BDP. The average rating of reputation among customers improved from 7.8 at the start of MTFS engagement to an average of 8.5 when interviewed (after completed assessment and developed BDP) and expected improvement to 8.9 in the future when they have implemented the recommendations in their BDP. Clients' perception of their reputation among their peers was also high and showed a similar trend, but was slightly more modest than their perceptions of their reputation among customers.
4.3.3 Increased business sustainability
Figure 28
Figure 28 above shows that half (55%) of the interviewed clients indicated they experienced improvement in the sustainability of their business between the start of engagement in 2005 and April 2006 when they were interviewed. These clients also expect further improvements between 2006 and when they complete implementing the recommendations in their BDP in the future.
Figure 29
Figure 29 shows the ratings of each client on their business sustainability. The average rating improved from 5.9 at the start of MTFS engagement to an average of 7.1 when interviewed (after completed assessment and developed BDP) and expected improvement to 7.8 in the future when they have implemented the recommendations in the BDP.
4.3.3.1 Comparison between ratings of Assessor and client on improvement
Since assessment of client improvement on the intended programme outcomes were based on subjective client self-ratings, these ratings were checked against the ratings obtained from the assessors. Each assessor was asked to rate the clients in terms of improvement from start of engagement to when the assessor was interviewed for this evaluation (February to March 2006) to expected improvement once BDP recommendations have been implemented. On all outcomes assessed in this evaluation, a consistent trend was found when comparing Assessor with client rating. Assessors' ratings were lower than clients' self-ratings, but the assessors were more likely to indicate that improvements have been experienced or are expected once the BDP recommendations have been implemented.
Back to Top