When the BIZ programme was first initiated, a standard method of reporting did not exist - it was up to providers to choose how they were to present their reports. The majority of providers attached reports to their invoices, which were normally sent to the Ministry on a monthly basis. Some contracts, however, provided for bi-monthly or three monthly invoicing, with corresponding reporting requirements.

In September 1999, the reporting requirements changed in that all providers were required to provide standard information on a quarterly basis. The shift to quarterly reporting was initiated to make the task of providing and managing reports less onerous for the providers and the BIZ Unit respectively.

Accompanying this change was the provision of standard report templates which simplified the reporting process for providers. This also gave rise to greater consistency in the reports, thereby making the Ministry's task of analysing and processing the reports more straightforward.

While providers are not contractually obliged to use the standard template, most choose to do so. The completed template forms can be sent back to the Ministry electronically if desired.

The reporting requirements fall into two distinct categories. The first can be labeled "compliance reporting". Its purpose is to enable the Ministry to determine whether providers are meeting the performance measures contained in their contracts. Performance measures consist of tasks and targets that providers must fulfill, such as running a set number of courses, with a certain number of clients attending each service delivery.

The second reporting category can be labelled "statistical reporting". This component consists of information relating to BIZ programme participants. The purpose of this is to enable the Ministry to obtain an overall profile of numbers and the distribution of programme participants. This information is reported to other parties, such as Ministers and BIZ Providers through BIZ News.⁹

The following outlines the information that is required under each of the two categories described above.

Compliance Reporting

Providers are required to supply information about each workshop, seminar, course, network meeting, or one-on-one service, including:

Its name;
The location in which the service was delivered;
The date of delivery; and
The sex and ethnicity of each of the clients.

(An example of this template is attached as Appendix 1.)

Providers are required to supply information in quarterly reports in which they describe their actual performance compared with their performance standards (as per the contracts). This is to ensure that they are providing what they have been contracted to deliver. Explanations are required when performance standards as specified in the contracts are not met.

(An example of this template is attached as Appendix 2.)

Statistical Reporting

Providers are required to report on:

The number of places filled by individuals attending training/networking/one-on-one sessions;
The number of unique Small/Medium Enterprises (SMEs) represented at training/networking/one-on-one sessions by individuals, including:

The length of time that each has been in operation;
The number of people employed in each; and
Their Business Industry Classification Codes.

So if several people from one SME attend a course, they will each be counted in the number of places filled, but the SME will only be counted once in the other fields.

Note that at present, providers are not contractually obliged to provide the Ministry with this information.

(An example of this template is attached as Appendix 3.)

Compliance Reviews

In addition to the reporting requirements, every provider undergoes a compliance review at least once during the course of the contract period.¹⁰ The compliance reviews are carried out by auditors from the Ministry. In instances where problems are uncovered or where the Ministry has concerns about particular providers, the reviews may be undertaken more frequently.

Reviews include the following areas:

Accuracy of information supplied by providers to the Ministry;
Accuracy of financial information supplied to the Ministry, and value for money of the services delivered;
Quality of materials, mentoring and courses delivered;
Delivery against contractual delivery requirements;
Needs assessments and target groups; and
Performance monitoring systems.

(A copy of the compliance assessment programme is attached as Appendix 4.)

In addition, providers can be required to allow the Ministry, or any third party on its behalf, to observe their delivery of services.

Other Reporting

As well as the standard information required on a quarterly basis, providers may also supply other information about performance. This information is specific to each provider; mostly in the form of measures of client satisfaction; and is typically in terms of its original contract, where not supplanted by the standard quarterly reporting.

2.2 The Relevance and Usefulness of the Information Required and its Adequacy for Managing Contracts and Ensuring Delivery of Quality Services

Compliance and Statistical Reporting

The compliance component of the quarterly reports is required to enable the Ministry to determine whether providers are meeting their contractual obligations. The statistical requirements are necessary to give the Ministry's Economic Development Policy Unit a profile of BIZ programme participants, and therefore to make inferences about the effectiveness of the programme.

As well as the Ministry's requirements, it is important to be conscious of the burdens that reporting can place on providers, and we note that many have complained about these requirements.

Although the complaints are not specific on this point, we would expect that the programme information statistics relating to SMEs are the main source of difficulties, as these require more processing of client information than simple head-counts of people attending courses.

On balance, we do not consider that the current reporting requirements are excessive or too difficult to comply with. While gathering some of the data presents problems, the total amount of data does not appear excessive, providing suitable systems for capturing it are in place.

The Ministry has gone to some lengths to make the reporting as simple as possible by, for example, providing the standardised templates against which all providers can send back to the Ministry electronically. In order to facilitate the reporting process the BIZ Unit has given providers guidelines on how to produce and classify the data properly. Also, the BIZ Unit can be contacted if providers are having problems in this respect.

The BIZ Unit says that in addition to the current required information, it would like providers to include in their reports, the names of each participating SME. This would enable the Ministry to track each SME through the programme, to assess the level of follow-on training, etc.

This would require more work from providers, but should not be too onerous, as presumably they would need to record the names in order to produce information about SMEs. It should be noted, however, that providers have almost unanimously stated that they don't like changes to the reporting requirements (and apparently minor changes can necessitate significant modifications to information systems).

The use of this additional information would appear to be for statistical reporting and policy purposes rather than for contract management. Before proceeding with such a measure, the Ministry needs to be confident that the benefits arising from the additional information would be great enough to justify the associated compliance costs. It does not appear to us that the Ministry is sufficiently clear as to how it would use this information to justify proceeding at this time.

(We do note that direct contact between the Ministry and BIZ clients would occur through the use of enhanced satisfaction measures discussed in section 3.2. This would not of itself require the Ministry to maintain a database of BIZ clients.)

Other Reporting

With regard to the current reporting requirements, it should be noted that these only measure the quantitative aspects of service quality - i.e. the number of persons and SMEs to whom the services were provided. They do not enable judgements to be made (at the level of individual providers or the BIZ programme as a whole) about the quality of services delivered - in terms of whether the services were delivered professionally and provided clients with skills and information which enhanced their management capability - and do not distinguish between satisfactory, unsatisfactory and excellent services.

Measures of the quality of services are partially given through the compliance review process (but only to the extent that it ensures that minimum requirements are met) and the client satisfaction measures as reported to the Ministry by providers (about which we have considerable reservations, as they provide numerous avenues for introducing biases).

Nor are useful measures produced about costs of administration, and unit costs of services delivered.

We discuss additional reporting measures in section 3.2.

2.3 Whether Providers Are Meeting Performance Standards, Targets and Guidelines; and Differences in Performance among Providers and in the Uptake of Services and Possible Reasons for This

The quarterly reports are the key tool through which providers communicate to the Ministry about whether they are meeting their performance standards and targets.

In terms of performance standards, ethnicity targets are the aspect that providers are having the greatest level of difficulty in meeting. They are generally meeting the reporting requirements adequately (see section 2.5) and the client satisfaction targets, although as described in section 3.2 we do not consider this to be an effective measure of provider performance.

Fulfilment of Performance Standards Relating to Client Targets

The targeted minimum number of clients to attend all BIZ courses, seminars and workshops between the commencement of BIZ contracts and 21 January 2000 was 32,314¹¹. The actual number obtained was 30,939 - a shortfall of 4.3%.

Eleven out of the 43 providers included in the BIZ Unit's database had shortfalls of at least 12% below minimum targeted numbers of clients. The only apparent pattern amongst this group was that a relatively high proportion were providers that delivered predominantly to Maori (four out of 11). There did not appear to be any patterns in terms of regional coverage or nature of the service offered.

According to the Negotiation Report,¹² a total of $20.0 million was originally allocated for the BIZ programme for the first round of contracts, including $6.1m for Maori, $1.2m for Pacific Peoples and $3.5m for women.

There are no equivalent targets for the proportion of clients from each target group over the programme as a whole, but a number of providers have such targets in their individual performance agreements.

We have divided the providers into groups that delivered to the target audiences of Maori, Pacific Peoples and Women.

Providers classified as deliverers to Maori audiences include those that are Maori owned and/or operated who are delivering (although not exclusively) predominantly to Maori; and those that have Maori client target levels of 40% or above. Ten providers fell into this category.

Actual client numbers in courses delivered by this group were 1.3% lower than targeted during the period from the commencement of the programme to 21 January 2000.

While this shortfall is lower than the overall shortfall across the BIZ programme (4.3%), out of the 11 providers with client levels at least 12% below their targeted levels, four were providers classified as being deliverers to Maori audiences.¹³

From the commencement of contracts to 31 March 2000, Maori represented 24% of attendees at skills training sessions.¹⁴ Note that this includes Maori attending courses provided by both Maori and other providers.

We classified providers as being deliverers to Pacific Peoples according to the list contained in the Evaluation Report¹⁵of providers with a "Pacific Peoples target group capability". Eleven providers fell within this category.

In this group, actual client levels for the period between the commencement of contracts and 21 January 2000 were 16.8% below minimum target levels.¹⁶

From the initiation of the programme to 31 March 2000, Pacific Peoples represented 9% of attendees at skills training sessions, including those supplied by these 11 providers and others.¹⁷

Several providers with significant Pacific Peoples client targets that were interviewed in the course of this evaluation indicated that they had set their ethnicity targets based on the demographics of the population in their area. This may have led to unrealistic targets in that the percentage of Pacific Peoples-owned businesses is generally lower than their proportion in the population might suggest.

As BIZ is aimed at people already in businesses, this shortfall does not necessarily mean that the programme is failing to deliver to Pacific People - rather, different interventions may be needed to increase their participation in business activity.

Providers were classified as having an emphasis on women target groups if over 50% of their total programme was devoted solely to them. Eight providers fell within this category.

The actual number of clients obtained by these providers was 12.3% below the targeted minimum number of clients.

However, over the whole programme female participation is higher than male participation, when taking into account women who attend components of the programme not strictly aimed at them. In the period between the commencement of contracts and 31 March 2000, 61% of all skills training participants were female.¹⁸Therefore, this shortfall should not necessarily be interpreted as a lack of interest by women in the BIZ programme.

Possible Causes of Differences between Providers

Differences in performance have presented themselves in terms of differences in the ability of providers to meet the targeted number of clients. Other differences can be presented in terms of differences in the quality of the services that providers are delivering. There are no data available on service quality differences between providers as there are no systems in place to measure this¹⁹.

Providers offer different courses with different methods of delivery, to different target group audiences in different regions. Given the lack of consistency between what providers deliver, variations in their experience, and the lack of monitoring of the delivery content, significant differences in performance are not surprising.

The information that we examined does not provide any explanations as to the reasons for variations. However, a number of possible reasons can be given.

One possible reason for variations in performance is differences in the effectiveness ofmarketing. Each provider is responsible for its own marketing and generating clients for its BIZ services. Depending on the success of its marketing, and understanding of and linkages to target audiences, some providers would be more effective than others in filling courses. Indeed, many may have misread target audiences (as described earlier). These marketing problems could present themselves in variations between providers in their abilities to meet client number targets.

Another reason for differences in performance is provider capability, in that insufficient emphasis was placed on examining the capability of providers to market and deliver quality services during the tender process. The Ministry concedes that short implementation time frame prevented it from being able to perform this task as well as it would have liked (see section 3.4). As a result, the tender process may have contributed to variations in the quality of course providers, and therefore in the quantity and quality of the services delivered.

Providers also differ in experience in providing services like BIZ, and in the delivery of government-purchased outputs. For example, it would be expected that a polytechnic should be able to put together and deliver a training programme with relative ease, given that the provision of courses forms its core business activity. Some providers, however, did not have previous experience in the delivery of training programmes and could therefore be expected to experience more implementation and delivery problems.

Other differences in performance could have resulted from variations in the competence of key staff delivering the programmes. Loss of such staff during the duration of contracts could have affected performance, and indeed whether services could be provided at all.

2.4 The Proportion of Funding Spent on Provider Administrative Costs Compared to Funding Spent on the Provision of Services

We understand that the concern about administrative costs arose from a view that these had been excessive in some of the predecessors of the BIZ programme.

One of the BIZ Unit's objectives is to incur internal administration costs (policy advice and contract management) of less than 15% of total programme costs. The operating budget for the BIZ Unit is $1.496 million in 1999/2000, which represents 10.8% of total programme costs including payments to providers.²⁰

There are also administrative costs incurred by providers in operating their services. To assess these, we recorded the proportion of funding that each provider allocated towards administrative costs, where this information was available. 34 providers had information relating to administration costs in their contracts. The following is based on these 34 providers.

These figures were obtained from the original contracts and do not take account of changes in the proportion of funding spent on administration that may have occurred due to contract variations. However, such changes are likely to have been small.

According to the payment schedules contained in each of the contracts, the average proportion of funding spent on administration costs by providers is around 13.5%. Amongst providers, these figures ranged from 2.7% through to 44.3%. There does not appear to be any correlation between the types of services delivered and the proportions of funding spent on administration.

In terms of providers with target group audiences of Maori, Pacific Peoples and women, Maori providers had an average level of administration costs of 13.2%, Pacific Peoples providers 10.4%, and women providers 23.8%. (Note that only four of the providers targeting women had the relevant information contained in their contracts. One of these providers had particularly high administrative costs, so this sample may not be a reliable indicator.)

From the available information for women providers, we have been unable to determine the reasons for variations in these costs. Differences could be the result of, but not limited to, differences in efficiency among providers, differences in accounting allocations, and variations in the effort and cost put into development of course materials.

In particular, the figures given by providers in their contracts relating to administrative costs may not be particularly reliable. It is possible that they are using different definitions of "administrative costs" in their accounting allocations, therefore reducing the comparative value of the data.

The tender evaluation process was designed to ensure that the funding would not be given to providers that had comparatively high administration costs. The fact that there are large discrepancies in the amount of funding allocated to administration among providers could be an indication of a partial failure of the tender evaluation process to perform one of its intended functions.

The Ministry says that it negotiated lower administration costs with some providers. However, it is possible that this negotiation phase was not carried out consistently or as comprehensively as intended due to the compressed time frame for implementation of the BIZ programme.

Since the tender round, no action has been taken over provider administrative costs.

Our view is that monitoring provider administrative costs is less important than obtaining reliable data about unit costs of services delivered (see section 3.2). The Ministry needs to be satisfied that it is getting value for money from the total costs of the services which it purchases, and if it can be satisfied about this, the composition of the costs is a secondary issue.

For future tender rounds we consider that, if administrative costs are seen as a matter of continuing concern, the Ministry should provide clear definitions of administrative costs (for proposals and reporting), guidelines as to what it deems to be an acceptable range of costs, and require proposals to remain within that range or to justify variations. This should be a relatively straightforward exercise for the Ministry to undertake given that it now has benchmark administrative costs in place.

2.5 Whether Providers Are Meeting Reporting and Data Requirements Adequately

Reporting Performance

This can be assessed in terms of the timeliness, accuracy and completeness of the data supplied by providers in the reports. Overall, the BIZ Unit claims that providers are generally meeting the reporting and data requirements adequately on these three measures.

While there are differences between providers in terms of meeting reporting deadlines, the BIZ Unit states that there are no apparent patterns to the groups that send in late reports, i.e. they do not appear from a particular region, nor to deliver to a particular target group audience, nor to deliver particular types of services.

In terms of timeliness, the number of late reports has dropped substantially since the BIZ programme commenced, and the majority of providers are now meeting reporting deadlines. This is probably because reporting requirements have become much clearer (see section 2.1), and because providers now have greater experience in preparing reports.

Although all reports arrive at the Ministry eventually, a small number (approximately 6 or 7 each quarter according to the BIZ Unit) do not arrive by deadlines and the Ministry is required to undertake a series of follow-up measures. If the Ministry wanted to further reduce the number of late reports, it could expand the coverage of performance standards contained in the contracts to include quarterly reports being sent to the Ministry by set deadlines.

The BIZ Unit says that a small number of providers are failing to fill out the section of the standardised reporting template relating to client details, i.e. the Business Industry Classification Codes of the clients. It is unlikely that this is due to the forms being too difficult to fill out, as the Ministry has produced guidelines on how to complete the forms. However, these data are more complex and difficult to gather than others required for reporting, and the most likely explanation is that some providers are unable or unwilling to set up their information systems to generate them. However, the number of providers that do not fill out the forms correctly has fallen over contract period, and most are now completing the templates.

Note that at present, providers are not contractually obliged to provide the Ministry with this information. Problems in this area of reporting could be solved by having contracts require providers fill out these forms completely as one of their contracted performance standards.

Reliability of the Data

The BIZ Unit works on the assumption that the information it receives from clients is reliable. Compliance reviews are critical for establishing whether this is the case.

However, the compliance reviews recently carried out by the Ministry (which are to be carried out at least once throughout the course of the contract period) have highlighted a number of instances where the information given by providers to the Ministry has not been accurate. In most of these cases, the providers did not intentionally provide inaccurate information. Instead, the providers did not have their information systems set up in a way that captured the relevant information and translated them into reports correctly.

It was recently discovered that one provider had been intentionally falsifying information given to the Ministry. In response to this, the Ministry terminated the provider's contract and reported the matter to the Police.

In future, the Ministry could include timely, reliable, accurate and complete reporting as a contractual term within the required performance measures (with contractual remedies for providers failing to meet these requirements). This is discussed further in section 3.5.

⁹BIZ News is an internal newsletter for BIZ providers, produced and distributed by the BIZ Unit.

¹⁰Most BIZ provider contracts have a period of approximately two years.

¹¹These figures were obtained from the database maintained by the BIZ Unit, as described in the methodology section. Note that the database only includes information relating to courses, seminars and workshops. The actual number of clients contracted for overall would be greater than this, given that the programme also includes mentoring, networking, etc. The overall difference between clients contracted for and clients participating would also probably be different.

¹²Ministry of Commerce (1999a)

¹³Two providers that fell within this category were not included in these statistics as the providers do not offer training courses, and the database from which this information was obtained only holds information relating to training courses.

¹⁴Ministry of Economic Development (2000)

¹⁵Ministry of Economic Development (1998c)

¹⁶Two providers that fell within this category were not included in this analysis as they did not hold any courses, workshops or seminars and therefore were not included in the database from this information was obtained. The services carried out by these providers included mentoring, networking, and one-on-one diagnostic sessions.

¹⁷Ministry of Economic Development (2000)

¹⁸One of the providers that fell into this category was not included in this analysis as the course was ongoing and had not been completed by 21 January 2000.

¹⁹The compliance reviews do not make comparisons between providers.

²⁰Ministry of Economic Development (2000). Note that costs of policy advice provided within the Ministry itself are not included in the above figures, but are unlikely to push the administrative proportion above 15% of the total.

Jump to main page content.

Printed: 26/11/2009