2.Method
2.1 Overview
This evaluation was based on multiple sources of information and perspectives including interviews with the participating businesses, interviews with the programme delivery agents (account managers, assessors, and mentors), review of clients' Business Development Plans, review of client's business snapshot results, TPK's MTFS client database, monthly reports from MTFS Account Managers, and interviews with key stakeholder agencies.
2.2 Interviews with programme delivery agents
Interviews were conducted with both MTFS Account Managers and the MTFS Programme Manager to get their perspectives on how well the programme is working. Interviews were also conducted with the seven Assessors and two Mentors who have worked with the businesses in our client interview sample (described below) to get the Assessors' and Mentors' views on how well the programme is meeting the needs of these clients.
2.3 Client interviews
Interviews were conducted with 21 businesses participating in the MTFS programme. These clients were not selected randomly, but the selection was based on:
- Length of time in programme – Given that the MTFS programme has been in operation for barely a year, none of the participants have completed the course of the programme. At the time of this evaluation, only 25 clients have had a BDP prepared and only a few have been assigned mentors to assist with the implementation of their BDP recommendations. The evaluation interviews focused on those clients who have been in the programme longer and have had a BDP prepared. The average number of months the 21 interviewed clients had been in the programme was 9 months (compared to the average of 7 months for all participants in the programme); and
- Client availability to be interviewed in person or by phone during the last two weeks of April 2006 - Face-to-face interviews were completed with 9 clients and phone interviews completed with 11 other clients. Compared to total participants in the programme, a larger proportion of the interview group were from Rotorua (i.e. 27% compared to 43%). This is not unexpected as the programme was first introduced in this region before a national rollout. The rest of the interviewed clients were from Taranaki (14%), Whanganui (14%), Auckland (10%), Gisborne (10%), Hamilton (5%) and Whakatane (5%). In terms of age of business, the 21 clients in the interview group had similar age distribution with the total MTFS participants.
2.4 Review of Business Development Plans
The Business Development Plans (BDP) prepared by Assessors for the businesses in the client interview sample were reviewed alongside findings from the client interviews.
2.5 Business Snapshot results
Clients who were interviewed for this evaluation were also invited to complete a follow-up Business Snapshot assessment during the first two weeks of June 2006. The purpose was to compare clients' self-rating of their business during the June follow-up with their self-rating at the start of the programme to see if any improvements have been experienced since participating in the programme.
However, due to the following difficulties, the results from this follow-up snapshot could not be used to assess extent of improvement attributable to the MTFS programme:
- Difficult to interpret changes in self-rating between follow-up and start of programme. For several of the clients, there is a decrease in their self-ratings on some questions in the Business Snapshot. For example, on the question "We use a business vision to guide strategies and decisions", one client rated themselves ‘Mostly' back in May 2005 but ‘Sometimes' in the follow-up assessment in June 2006. It was difficult to interpret if this apparent decline in rating is due to increased awareness and realistic assessment or actual decline in business practices. The original intention was that during the June 2006 follow-up assessment, clients would be asked to indicate for each question, a) where they thought their business was at during the start of the engagement as well as b) where think they are at now. However, given the length of the Snapshot (103 questions in total), this approach was not feasible as it was too demanding on clients' time.
- Response rate was too low. A total of 32 clients who have had a BDP signed-off were invited to complete the follow-up snapshot (this included the 21 clients who were interviewed for this evaluation). However, of the 32 clients, only 7 completed the follow-up snapshot, resulting in a response rate of 22% which is very low and raises a concern that the results may not be biased and not representative.
2.6 TPK client database (SmartFund)
Information from TPK's MTFS client database (called SmartFund) was used to analyse participant numbers and characteristics.
2.7 Account Manager monthly reports
Monthly reports prepared by MTFS Account Managers on the progress of the programme were also reviewed.
2.8 Feedback from other stakeholder agencies
The following key stakeholder agencies were invited to provide feedback on the MTFS programme - NZTE, Poutama Maori Business Trust, New Zealand Maori Tourism Council, and Tourism New Zealand.
2.9 Limitations
The main limitation of this evaluation is that due to the programme's infancy, it is too early to measure many of the intended impacts on practices and performance of participating businesses. None of the participants have completed the course of the programme. At the time of the evaluation interviews in April 2006, only five clients have actually been assigned a mentor. Hence this evaluation was performed when most clients had not received the full programme which will impact client perceptions of the service.
Another key limitation is the lack of a control group (i.e. a comparable group of firms which did not participate in the programme) which is important for an objective assessment of the extent to which the outcomes measured are attributable to the programme.
Back to Top