2. Summary of Findings
2.1 Awareness Survey of SMEs
This component of the evaluation was designed to establish the awareness of the BIZ programme, and its different components, amongst owners and decision-makers of SMEs who make up its target client group. Some of the results were compared to a similar survey undertaken in 1999. A number of responses were disaggregated according to the size and length of operation of the SME and the gender and ethnicity of the owner or decision-maker.
It consisted of a random survey of 800 owners/directors of SMEs (defined for the purposes of this report as enterprises employing up to 20 full time employees). Names were obtained from the Universal Business Directories database and the survey was conducted nation-wide by telephone. Key findings were:
- Overall awareness for both BIZinfo and BIZwas significantly higher amongst SME owners than it was a year ago, although BIZinfo has much better market penetration than BIZ.
- Although a significant proportion of SME business owners were aware of BIZinfo and BIZ training, they did not generally distinguish between the two services in terms of what they offer. A small number of SMEs believed that BIZ offers services that are not actually part of the programme.
- Television, mailouts and print advertising were the most common media by which SME owners became aware of BIZ, although recent start-ups and Maori owners were less likely to mention mailouts as a source of awareness.
- More than a third of those who were aware of BIZ were unable to recall the specific services offered without prompting. Amongst those people who knew of some BIZ services without prompting, mentoring with one-on-one advice was the most commonly mentioned service. The next most commonly mentioned services were information and referrals, assessment of business needs, training seminars and networking. Maori were more likely to mention mentoring, one-on-one advice, information referral and assistance and analysis/assessment of business needs.
- The majority of respondents who were aware of BIZ also knew that the service is free, although there was a diversity of views regarding the impact of "free services" on programme uptake. Those running recently established businesses were strongly and positively influenced by the fact the services are free, as were Pacific Peoples clients.
- The 0800 number and the website were both seen as effective ways to contact BIZ but the local BIZ centres ranked a poor third as a means of contact, and even after prompting most business owners were unaware of these centres. Pacific Peoples clients were less aware of any of these methods of contacting BIZ than other clients.
- Just 47 respondents (17%) out of the 276 aware of the service have contacted BIZ, and 30 (11%) have used its services.
- More than half of those contacting BIZ rated themselves as "pleased" with the experience; however there were more than a third of those who contacted BIZ who were not especially pleased or somewhat disappointed.
- When asked to identify the service used, almost two-thirds of these users identified training workshops/seminars first. This was followed by mentoring, information or referrals, assessment of business needs and networking. Nearly three-quarters of those using BIZ services rated the benefit of using one or more services as "very good" or "fairly good". A quarter rated the benefits as "not good, but not bad" or "not good at all".
- In terms of the likelihood that SME owners would use each of the five types of BIZ service, out of the total (800) respondents;
- the largest proportion (39%) reported that theywere likely to participate in training, seminars or workshops;
- 37% were likely to use the information and referral service;
- 30% were likely to use networking;
- 28% were likely to use an assessment of business needs; and
- 27% were likely to use mentoring and one-on-one support.
- Businesses that had at least one female owner or decision-maker were examined as a separate subgroup. This subset, comprising 64% of the 800 respondents, revealed few differences based on gender.
- Businesses with at least one Maori and/or Pacific Peoples owner or decision-maker were also identified, although the small size of the data set made drawing conclusions tentative. Maori were less likely to mention mailouts as a means by which they heard about the BIZinfo/BIZ programme and were more likely to be aware of mentoring, information referral and assessment of business needs than the rest of the sample.
- Businesses employing up to 5 staff and also businesses employing more than 15 were more likely to be influenced to go to BIZ by the fact that its services are free, while businesses employing 6 to 14 staff were less concerned about whether or not BIZ provides free services.
- Length of time in business also had a relationship to BIZ awareness. Those with recently established businesses had a higher awareness (21%) than those whose businesses are 10 or more years old (11%). The older businesses were also less likely to mention other services offered by BIZ, except for networking.
2.2 Survey of BIZ Training Clients
This component of the evaluation was designed to investigate the experience of SME owners and/or decision makers who had participated in BIZ training, in terms of issues discussed below. It consisted of a random survey of 800 people who had used BIZ services. Names were obtained from provider listings, and the survey was conducted nationwide by telephone. Findings may be broadly summarised as follows:
- Satisfaction with the content and delivery of the courses was high. Since business people have marketplace reality against which to test what they are taught, we accept satisfaction is a relatively good proxy for real value.
- Media advertising (for newer businesses) and direct mail advertising (for established businesses) were the most common ways that firms come to hear about BIZ services. Referral from BIZinfo is relatively low.
- The assessment or diagnosis of individual firm requirements appears to be either not done or not consciously experienced by nearly half of BIZ clients. While this did not seem to lead to clients ending up in the wrong courses or at the wrong level to any extent, assessment or diagnosis should be seen as the first stage of an integrated programme for assisting clients, along with (existing) training and enhanced mentoring.
- Seminars and block courses are the most common form of business education received from BIZ. Mentoring is most utilised by newer firms.
- Marketing and then finance are the two subject areas most frequently used by firms. However, a range of topics including human resources, taxation, information technology, legislation, and others, are used to a substantial level.
- The main impact of the use of BIZ courses on the participants was that it helped them configure a better planned path for going forward. Fairly high beneficial impacts were also reported for understanding customers and how to market to them, raising the business skills of staff, handling the financial needs of the firm, reducing waste of time and materials, transferring good ideas from a business network; and raising the quality of the product or service.
- Mentoring rated highest for the form of assistance considered most likely to yield a good result for the client firm, but seminars were not far behind.
- In the course of preparation of this survey, it became evident that standards of registration and identification of participants are mixed. Auditing, follow-up performance surveys, and eligibility monitoring, are compromised when this occurs. A lift in the precision of record keeping is desirable.
2.3 Maori and Pacific Peoples Focus Groups
As a complement to the client survey, six focus groups were conducted - four of Maori and two of Pacific Peoples who appeared on the lists of BIZ training providers as having received a BIZ service. These focus groups aimed to allow a more free-ranging capture of the experience and interface of the BIZ programme with business people of Maori and Pacific Peoples origin.
These focus groups were intended to provide complementary information to the quantitative survey of users, specifically of a more open-ended and qualitative kind. The information, by its nature, is designed to provide insights and impressions rather than definitive statements of fact.
2.3.1 Maori Focus Groups
Four focus groups, two made up of women and two of men, were conducted among Maori clients of BIZ training providers.
- It was evident from the groups that BIZ providers had operated the eligibility criteria liberally, since the groups included people ranging from those figuring out how to set up, to those setting up, to those firmly in small business in the classically understood sense.
- The most common reaction among Maori to the experience of having done one course with BIZ, was the feeling that they would like to do more and generally needed to do more. With the exception of where a person had been essentially misplaced into a course (i.e. for which they were too advanced or not advanced enough), people felt the courses were effective in their content and delivery.
- Among those at entry level, the presence of a Maori person as trainer was valued, largely in giving confidence and, on occasion, inspiration. The more established the Maori business person, the less significant this consideration became.
- Although it was uncommon for people to have much awareness of having been individually assessed for their needs, this did not reflect itself in any dissatisfaction with the courses people subsequently were placed in. With some exceptions, small business people in these groups felt the courses they attended filled a need for them.
- Where the Maori business person was already relatively established in business, certificates for the courses were of low interest. Where the person was at entry level, the initial interest expressed was higher, but faded upon closer consideration. Certificates were not seen to be the route of the owner-operator, nor were they commercial currency.
- Fees for BIZ courses were resisted largely on the grounds that cash was tight for the entry level person and the value of a course not easily judged in advance of payment. At the same time, it was acknowledged that people like themselves may not value something that was given free and that a charge would ensure more diligent attendance. It might therefore be appropriate to apply small "registration and resources" style fees.
- For those at early or entry level to small business, the seminars and courses were appealing as the form of delivery. The more established the business, the greater the appeal of the mentor approach for the very specific and actionable understanding and advice that could be obtained.
- These Maori small business people raised some ways of thinking which they felt stood in the way of them, and Maori people generally, advancing more rapidly in business. These included being less commercially aggressive in things like market pricing, often having backgrounds in community work where work was for aroha rather than money and where payment by koha blurred the true economic value of the services given. Lower market acceptance through having shorter track records, and the possibility of people being reluctant or over-cautious when doing business with Maori due to media highlighting instances of under-performance, were also raised.
- These Maori business people reflected an expanding awareness of business paradigms and a sense of needing to go further in developing the savvy that was required to make a business succeed. BIZ was seen as a useful source for accomplishing this.
- Finance and the difficulty of obtaining it, was spoken of as the largest obstacle to developing their small business. They expressed some frustration at not being able to inspire lenders, or obtain business credit.
- Although an equal number of women and men were involved as participants in the focus groups, the moderators could not detect systematic differences between the two. This may be a factor of the small scale of the research, or may reflect a genuine similarity of approach to BIZ training between the two genders.
2.3.2 Pacific Peoples Focus Groups
Two focus groups were conducted among Pacific Peoples clients of BIZ training providers.
- Many Pacific People in small business, or on the verge of it, are first generation New Zealanders or, if not, are closely associated with communities and churches based on people who emigrated from the islands.
- It is part of a candid self knowledge among these people that they have not grown into a business mindset, and indeed that aspects of business thinking in a money economy are not easy to fit to Pacific Peoples cultural ways.
- They are keen to acquire business thinking, practices and routines. They also express a need to find ways of having others in their community understand that these are legitimate and necessary ways of operating business, and not a reflection of their abandoning the cultural mores of sharing and giving, nor an expression of selfishness or greed.
- It is evident that Pacific Peoples business people at the stages of business exploration, formulation, and set-up, are appearing in BIZ courses. The intention of the programme designers was that only established business would be eligible. It was felt that many people were on the threshold or either side of it, and that a stricter adherence to the rule would stem the growth of business mindset development among Pacific Peoples business people. Fees would likely limit participation disproportionately.
- The notion of "trusts", charitable and other, arose in ways which suggested that among Pacific Peoples, there was over reliance on this format for business ventures. More sophistication in capital and credit raising was a need in evidence.
- An expectation that they would emerge from business training able to do something, rather than merely "knowing things" emerged as a useful lever in engaging the interest of Pacific Peoples small business people. Certificates held little appeal.
- Both focus groups were confident that the way to disseminate knowledge of BIZ training to Pacific Peoples business people was after Sunday church services, at the church, and with the co-operation of the minister. The feeling was that this would give both good reach, and credibility to the message.
- Mentoring was easily the most favoured approach to business training. Participants envisaged the idea of coaching the business onto its feet. However, this preference was not seen as a substitute for training courses, but as a complement to them.
- The one day and the power-packed courses were seen as unproductive in the sense that the owner and/or staff could not absorb and implement that much change at one time.
2.4 Survey of BIZ Training Providers
This component of the research was designed to obtain information about the experience and opinions of BIZ training providers, covering areas discussed below. It consisted of 20 face to face interviews with BIZ providers, selected on the basis of their contract size and/or specific delivery to target groups. Another 12 were interviewed by telephone. In addition the partners of the National Business Information Service (the BIZinfo provider) were also interviewed.
- The scope of the provider organisations that were interviewed varied widely: some offered social services as well as assistance with economic and business development, while others were more narrowly focused on the provision of specific training courses (e.g. computer training).
- In terms of BIZ services there was no "standard" set of services available to clients, although most could be described as falling into the four categories of i) seminars, ii) courses, iii) mentoring iv) networking - ideally, after pre-assessment of business needs.
- There are a variety of ways in which the providers deliver training by: i) employing specialised trainers (and expecting them to run the entire programme by themselves); ii) doing all the administrative work for the trainers and just expecting them to do the face to face delivery; and iii) taking on a high level of responsibility for the entire delivery.
- Most organisations had similar evaluation and quality control systems that asked attendees to assess the training received.
- There was an almost unanimous perception that BIZ has had an identifiable impact on the individuals who have attended the training, based on the feedback from the evaluation forms and the direct feedback from attendees.
- While almost all appeared familiar with the BIZ Unit eligibility criteria, there were a number of reasons why these were not always applied stringently. In particular, the boundaries of what constitutes a "business" are not always clear and providers tended to interpret the criteria liberally to assist the greatest number of clients.
- In terms of target groups, the providers' perception was that delivery to their women clients had been easiest to achieve. Maori were perceived as being the next easiest to deliver to. Almost all respondents commented on the difficulty of achieving Pacific Peoples targets. There was some feeling that not all the potential target groups had been recognised, particularly business immigrants and business start-ups.
- Questioning providers on services for Maori and Pacific Peoples generated a diverse range of opinions that are not easily categorised by provider type. Some were of the view that Maori clients want to be part of "mainstream" services so that they can network and interact with other business owners, whereas others felt that they want customised services that recognise a Maori cultural context.
- There was a variety of monitoring systems in use for counting numbers of clients from target groups, and it appeared to the researchers that their quality varied. There is a perception that there would be value in the BIZ Unit developing a single system for providers to use.
- In relation to the BIZ Unit, most respondents primarily identified contract management as its primary role. A significant number of respondents also commented on its role in disseminating information, liasing between providers, and facilitating networking. Many respondents identified services they would like the Unit to provide in the future.
- In terms of the performance of the BIZ Unit, an overall summary of the responses indicated that they found its staff "responsive" and "supportive".
- Several respondents commented on the need to improve the contract negotiation process. Specific comments were concerned with the transparency of the negotiation/purchasing process and the ease of preparing for the negotiations.
- In terms of the tendering process, some were happy with the degree of flexibility it offered, while others suggested a more structured framework, with a "menu" of services that the provider can pick and choose from in terms of delivery.
- Overall there was overwhelming support for BIZ to continue. It was felt that more resources should be allocated and that the government needs to signal a long-term commitment to business development.
- There was considerable concern about the relationship between BIZ and BIZinfo. Several respondents suggested that more resources should be directed at BIZ training and less at BIZinfo.
- There was a consistent response that charging (a nominal fee) might help minimise the numbers of people enrolling in courses who subsequently failed to attend.
- All providers were able to identify some service improvements that they would make, and how the BIZ programme as a whole could develop in the future. Most answers focused on improving the services for clients, for example in offering services to start-up businesses, and assisting existing businesses to develop further.
- A number of respondents also talked about the need to broaden the scope of BIZ beyond the provision of training and information, to a more formally integrated package of business assistance, which could include pre-assessments, mentoring and other one-to-one interventions.
- There was an almost unanimous feeling that standardising courses across the country would be valuable, with the caveat that the standardisation should be a framework only (i.e. not a whole course in detail).
- Accrediting courses was not seen as desirable. One respondent commented that imposing accreditation of any sort would unfairly penalise non-accredited providers.
- There was an overwhelming desire from respondents to explore new ways of sharing best practice amongst providers.
2.5 Mystery Shopper Trial of BIZinfo
This component of the research was designed to test the quality of response of the BIZinfo call centres and local sites - in this case, their response to callers who did not identify themselves as part of a survey (i.e. as if they were "real" queries).
Thirty-five "mystery shops" were conducted by real-life small business owners, who contacted BIZinfo call centres via the 0800 number (23 calls) and provincial sites (12 calls), and raised queries about issues they selected as important to their own businesses. The respondents made their own assessments of the response, the calls were monitored by professional survey staff, and tape recordings of most of the calls were made. In addition, five website evaluations were also conducted by real-life small business owners.
- Good ratings were obtained for leaving callers feeling they had been given a "next step" to take, for prompt phone response, and for leaving callers with the impression of receiving a competent service.
- The provincial sites failed to make use of the BIZinfo name, using instead their local organisation's identity. This can be rectified by having a separate phone number/line so that the person responding can differentiate the call.
- The "no fees" availability and basis for subsequent support or training was rarely brought to the caller's attention.
- This evaluation left the impression that the BIZinfo staff (in both call centres and local sites) tended to under-service the opportunities that came their way. There was often no thoughtful elicitation of the character of the caller's need. For instance, they appeared overly ready to divert the caller onto a government department without assisting him or her directly, or referring him or her to a suitable specialist.
- Nor did staff take the opportunity to go further than dealing with the immediate query and to demonstrate how BIZinfo could be an avenue to personal business upskilling, through asking about other business issues or suggesting referrals to develop skills in other areas.
- Because caller profile information is required for BIZinfo statistics, some BIZinfo staff spent much of their contact time gathering this information.
- The sample was structured to enable separate comment for Maori and for women. Differences between these groups and the overall sample were not large. Maori rated the service lower on communicating the availability of local training. Women rated it lower on defining what its role was and what services were available.
- Opportunities for BIZinfo to increase its effectiveness and offer additional information could lie with the use of enhanced screen-based information approaches, and standardisation of operator training to raise overall standards.
- Ratings by small business respondents about BIZinfo were generally positive. However, these appeared generous when set against the evaluator's knowledge of how much more informative and professional such services can be.
2.6 Analysis of BIZ Provider Reports and Analysis of Management and Administration of the Programme
This component of the research was designed to investigate a number of issues about reporting processes, contract management and implementation of the BIZ programme, in terms of the operations of the Ministry and the BIZ Unit. To do this internal BIZ Unit documents were reviewed, statistical data compiled from those documents and the BIZ Unit database, and there were extensive discussions with BIZ Unit and Ministry staff.
- All of the standard information providers are required to supply to the Ministry in their quarterly reports is necessary. While conscious of the costs to providers meeting reporting requirements, the current requirements are not considered excessive.
- The main weakness of the current reporting regime is that it provides information only about the quantitative aspects of service delivery, and does not enable judgements to be made about the quality of services delivered. This is a result of the current contractual provisions which focus largely on the number of courses seminars and workshops delivered, and participating clients and SMEs, but have few if any corresponding provisions relating to the standards of these services.
- Over the period from the commencement of contracts to 31 March 2000, there has been a small shortfall of 4.3% between the number of number of clients on courses, seminars and workshops targeted for and actually provided.
- In terms of involvement Maori and Pacific Peoples and women in the programme, of the total number of participants on BIZ courses, Maori comprised 24%, Pacific Peoples 9%, and women 61%. All of these represent higher rates of programme participation by these groups than would be implied by representation within the population of SME owners and operators.
- Most, but not all, providers are providing reports on time and with the required information. However, reporting requirements were initially specific to each provider and have only partially been standardised since implementation. Some information and reporting requirements are still not mandatory. Moreover, inaccuracies in information provided have been uncovered in the course of compliance reviews undertaken by the Ministry, and some providers require more frequent reviews until the Ministry is confident of the integrity of their information systems. In future, the Ministry could include timely, reliable and complete reporting as a contractual term within required performance measures.
- In general, the systems operated by the BIZ Unit are satisfactory. The main areas where improvements could be made include: additional resourcing for the BIZ Unit; introduction of standardised, professionally developed measures of client satisfaction (operated independently of providers); development of systems to assess quality of services delivered; and removal of the ambiguity in the eligibility criteria for use of BIZ services.
- A critical issue is unit costs of services, i.e. costs per course, per participant or per participant-day, which are a fundamental measure of the cost-effectiveness of individual providers, and of the programme as a whole. It was not possible to make any judgement as to whether unit costs are reasonable or otherwise, as these are not systematically monitored by the Ministry (and derivation of reliable unit cost benchmarks is not a small task). Consideration by the Ministry of requiring providers to derive unit cost measures (or at least attempting to do so) should be a key priority for future development of the programme.
- The programme was detrimentally affected by the truncated timeline available for design, tendering and implementation. There were a number of unsatisfactory aspects of the tender and contract negotiation process, and in programme implementation and management, all of which can be directly attributed to the timeline.
- To rectify these deficiencies, it is necessary to continue the incremental improvements to programme management, introduce more deliberate processes for future tender rounds, and conduct them with more realistic timeframes than the first one.
- Problems which are apparent in the contracts relate largely to the number of non-standard provisions, particularly in relation to performance measurement. Further standardisation is desirable with respect to reporting in a complete, timely and reliable way. Furthermore, contractual provisions are needed to manage non-compliance, including remedies when termination would be a disproportionate response, such as suspension clauses and contingent payments.
- There will be a continuing need for flexibility and variations in contracts according to local demand. The BIZ Unit's current process for dealing with contract variation requests appears to be effective, provided the Unit follows the procedures set out in the Key Procedural Manual.
- In the future it is critical to measure performance in terms of client satisfaction, quality of services delivered and unit costs. These performance measures provide an important part of an agenda for future development of the programme.
- Another part of the agenda for the future is to build on the experience of establishing the programme; using the positive relationship between the Government and providers to enhance the body of expertise and spread best practice amongst providers. Existing activities such as networking and seminars should be extended to assist BIZ providers to exchange information, etc. about their activities, so that standards of best practice are spread more widely amongst them, and the quality of their services can be enhanced.
Back to Top