Executive Summary
In the body of this report we provide a summary of the methodology employed and the information obtained during our analysis. The following is a summary of our key findings. The references after each title below are to the relevant section of the report.
Analysis of BIZ Provider Reports
(Section 2)
Relevance of Reporting Information
(2.2)
We consider that all of the standard provider reporting requirements for the programme, in terms of both contract compliance and statistical reporting, are necessary. While conscious of the costs to providers meeting reporting requirements, we do not consider the current requirements to be excessive.
The main weakness associated with the current reporting regime is that the reports provide information only about the quantitative aspects of service delivery - i.e. how many skills training sessions were provided to how many individual people and SMEs. They do not enable judgements to be made about the quality of services delivered; the unit costs of services; and client satisfaction levels. These measures are discussed further in section 3.2.
Performance against Targets
(2.3)
Over the period from the commencement of contracts to 21 January 2000, there has been a small shortfall of 4.3% between the number of number of clients on courses, seminars and workshops targeted for and actually provided.
In terms of the targets set for particular groups of clients, a number of providers targeting Maori, Pacific Peoples and women fell below their contracted minimum target client numbers.
Maori accounted for 24% of participants at skills training sessions, and Pacific Peoples 9%.
In terms of delivery of BIZ services to women, there have been significantly more women attending courses (61% of the total) than men.
The greatest shortfalls arose in service delivery to Pacific Peoples. The shortfalls may be a reflection of the fact that Pacific People are less likely to be in business than the rest of the population, and therefore that the initial targets were unrealistic.
The providers are a very diverse group of organisations, so significant differences are not surprising. Possible explanations for the differences in performance between providers include: differences in marketing and linkages to target audiences; insufficient scrutiny of the capability of providers during the evaluation phase; differences in experience in providing these types of services; and variations in the quality of key staff.
Administrative Costs
(2.4)
The BIZ Unit's budget for 2000/2001 is $1.5 million, 10.8% of total programme costs.
We estimate that of the 34 providers that had the relevant information contained in their contracts, 13.5% of their total funding is spent on administration. There is considerable variation between providers, ranging from 2.7% to 44.3%, and we are not confident of the reliability of the data.
Differences in administrative costs could arise through: efficiency differences among providers; differences in accounting allocations; and variations in the efforts and costs put into development of course materials.
We note that there is no benchmark as to a reasonable level of administrative costs, and (other than some negotiations at the initial contracting stage) the Ministry to date has not acted upon this information.
We do not consider monitoring provider administration costs to be a priority for action in the future. However, if this is seen as a matter of concern, the Ministry should establish a benchmark range for administration costs, and in future tender rounds indicate to potential providers that it will not fund costs outside this range.
Reporting Performance
(2.5)
Most providers are providing reports on time and with the required information. However, a small number of providers do not meet reporting deadlines and/or fail to provide all the information requested by the Ministry (generally data related to SMEs, which providers are not contractually obliged to provide).
Moreover, some inaccuracies have been uncovered in the course of the Ministry's compliance reviews. In such cases the Ministry requires that the relevant providers undergo more frequent reviews until it is confident of the integrity of their systems.
In future, the Ministry could include timely, reliable, accurate and complete reporting as a contractual term within required performance measures.
Analysis of Management and Administration of the Programme
(Section 3)
Contract Management
(3.2)
In general, the systems operated by the BIZ Unit are satisfactory in terms of competence of staff, documentation of systems and procedures, and access to records.
The main areas for improvement include:
- additional resourcing for the BIZ Unit - an additional contract manager, with the Unit manager focussing on the review and monitoring of contract managers.
- introduction of measures of client satisfaction which are standardized, professionally designed and operated, and with providers left out of the client assessment stage.
- development of systems to assess the quality of services delivered.
- review of available data on unit costs of services provided, with the objective of deriving meaningful benchmarks of unit costs for comparable services.
- removal of the ambiguity in the eligibility criteria for use of BIZ services, possibly including a positive definition (rather than the current definition of who is ineligible), and a standardised application form for all prospective BIZ clients.
Programme Implementation
(3.4)
The timeline of the tender process in 1998-99 was extremely compact, which we believe detrimentally affected programme. Consequences of this included:
- an inadequate process for assessing need and demand for BIZ services;
- a truncated bidding and contract negotiation process, with high risk of inappropriate decisions;
- insufficient standard contract provisions;
- insufficient planning for the operation of the BIZ Unit; and
- insufficient consideration of other models of public sector contract management.
Some of these problems have been rectified since that time. Others can be addressed in future tender rounds. It is critical that these proceed with a more structured process and sufficient time to undertake the processes properly.
Future tender rounds also need to:
- provide signals as to the services the Ministry wishes to purchase. This may necessitate consultation with local business communities (business development bodies, economic development agencies, business leaders, providers etc) in order to form a joint assessment of local demand; and
- provide the opportunity for the entrance of new providers and for new approaches to service delivery.
We do not consider this as representing any fundamental shift in the philosophy of the BIZ programme, which was about providers determining local needs. Rather, we see this as enhancing the process in order to ensure a balance between local demand, provider capacity and available funding.
Appropriateness of Contracts
(3.5)
Problems which are apparent in the contracts relate largely to the number of non-standard provisions, particularly in relation to performance measurement. Further standardisation is desirable with respect to:
- standardised reporting on all aspects of performance measurement, including client satisfaction and service quality;
- requirements for reporting within the timeframes set by the Ministry;
- requirements for providing all information required within reports; and
- satisfactory reports (e.g. about information systems etc) from compliance reviews.
Furthermore, contractual provisions are needed to manage non-compliance, including remedies when termination would be a disproportionate response, such as:
- suspension clauses - where payments could be suspended under explicitly-specified circumstances; and
- contingent payments, where a small proportion of the contractual payments would depend on compliance with contractual provisions about reporting.
There will be a continuing need for flexibility and variations in contracts according to local demand. The BIZ Unit's current process for dealing with contract variation requests appears to be effective, provided the Unit follows the procedures set out in the Key Procedural Manual.
Provider Performance
(3.6)
Considering future enhancements to the relationship between the Ministry and providers, the emphasis should now move to improving the competencies of providers. This should be addressed in a structured and planned fashion through a provider development agenda.
This approach needs "buy-in" from providers, rather than the threat of contractual sanctions. It should build on the experience gained in the BIZ programme to date; the positive relationship between providers and the BIZ Unit; and the additional resources available through enhanced contract management systems (and possible an additional contract manager).
Implementing "quality of service" performance measures (e.g. relating to course design, standard of materials, quality management systems, competence of trainers etc) will provide an important part of this programme, with development undertaken jointly with providers, so that there is common understanding and expectations about the use of these measures, and feedback to individual providers about their own performance relative to others in the programme.
Enhancements to performance by individual providers can be best promoted through extended networking and exchange of information amongst them about "best practice". While the success of such activities will depend upon the providers themselves assuming responsibility and "ownership" for the outcomes, the BIZ Unit will be able to provide positive leadership, with the objectives of a provider development agenda providing a focus and a framework for these activities.
Back to Top