Executive Summary
The purpose of this scoping study, commissioned from the New Zealand Institute for the Study of Competition and Regulation by the Ministry of Economic Development is twofold:
- to conduct an analysis of the state of Electronic Commerce performance measurement throughout the world; and
- to identify areas where investment in Electronic Commerce performance measurement research in New Zealand can be made, in order to provide a sound foundation for assessing future policy development, and infrastructure investment.
While the brief for this report is to examine Electronic Commerce performance measurement, we have not limited our analysis solely to commercial applications, as it is recognised that in a modern, information-based economy, the distinctions between:
- individuals, firms, markets, states and global communities of interest,
- business and personal transacting; and
- business locations and mobile transactors,
are becoming increasingly harder to define. Thus, the performance measures we examine are analysed not in isolation, but as integrated components of an entire economy, both electronic and non-electronic. This is because we consider it important to understand not just what new technologies alone are doing, but also to examine how they substitute for the functions and products they are replacing, and integrate with those they are complementing.
The report recognises that, while the Internet has come to dominate recent thinking about Electronic Commerce, the foundations for this phenomenon go much deeper. To this end, we have adopted the wide, inclusive definition of Electronic Commerce promulgated by the OECD:
"Electronic commerce refers generally to all forms of transactions relating to commercial activities1, including both organisations and individuals, that are based upon the processing and transmission of digitised data, including text, sound and visual images."2
Further, we take as our underlying foundation the premise that while technologies provide a vehicle for its transfer, the fundamental "good" of a modern economy is information. All transactions require information as a component. While this has always been true of all commercial and non-commercial exchanges, the fundamental changes wrought by modern information technologies are the different ways in which information is procured, processed, transmitted, stored, utilised and enhanced, and the costs of carrying these processes out. These are the changes upon which Electronic Commerce activities are predicated, and thus become the foundation for our analysis. We acknowledge also that the special properties of information (intangible, many public good and network good properties, an "experience" good, multiple functions and increasing returns to scale) necessitate that a different set of analytic considerations is used to examine its trade and exchange than those by which trade in the more typical rival and excludable goods are judged.
What Is Being Done
The extensive literature review undertaken for this report uncovered a vast range of Electronic Commerce performance measurement exercises. These include:
- international benchmarking undertaken by umbrella organisations (principally the OECD);
- national statistical agency monitoring of existing economic activity, with specific Electronic Commerce examinations (e.g. United States Bureau of Economic Analysis);
- specifically-commissioned governmental agencies examining particular national and regional Electronic Commerce activities (e.g. NOIE in Australia);
- ad-hoc government-funded national research (e.g. the New Zealand FORST-commissioned study undertaken by the Waikato University Business School);
- university-based research measuring activities (e.g. University of Texas Center for Research in Electronic Commerce);
- firm-based research undertaken by management consultancies (e.g. Boston Consulting Group, PricewaterhouseCoopers, Deloittes); and
- wider population-based sampling by market research companies (e.g. Nielsen, Forrester).
The predominant theme of all of these measurement exercises is their emphasis on technologically-based reporting of "what is being done", with little critical analysis of "what this actually means" in respect of longer-term economic outcomes. Almost all focus exclusively on Internet applications, and are underpinned by the mostly unstated and uncritical assumptions that "more Internet activities" are better than fewer, and "any electronic" is better than "any manual or non-electronic". Further, the metrics generally used to measure potential are available bandwidth capacity, numbers of ISPs and proxies for numbers of people connected to the Internet, while activity is benchmarked by numbers of secure servers and the "value of trade brokered on the Internet". The principal exceptions to this trend are national statistics, which still record overall economic performance of economies irrespective of trading mechanisms, although some break down performance measures into computer-using and producing sectors for comparative purposes.
Limited Usefulness and Relevance
While these exercises provide useful snapshots of "what is being done", and benchmark progress of technology uptake over time, the analytical foundations of what these measurements mean in terms of wider social and economic benefits are far less soundly formed. The focus on technology (e.g. computers, Internet, mobile phone connections, bandwidth capacity) and applications (e.g. B2B exchanges, supply chain management systems, on-line payment capacity) measurement obscures the activities that these conduits serve: the creation, storage, transmission, exchange and utilisation of information. As it is new uses of information and reductions in the costs of procuring information that are the ultimate creators of new value in a modern economy, the effectiveness of these technologies and applications can only be assessed in respect of how well they achieve these new value-creating aims.
Although each technology has an individual impact, the network properties of information result in system effects from their interactions that impact far wider than each individual application. It is the consequences of these interactions that provide real challenges for policy-making and investment decision-making. Yet it is the apparent absence of system-based analysis that characterises almost all of the literature on Electronic Commerce performance measurement, largely because each study is conducted in isolation from the wider economic and social environments in which it is occurring. Emphasising the Internet alone overlooks the impact of and understanding embedded in (for example) both:
- other long-standing forms of electronic and computer-based utilisation (e.g. EDI3, microprocessor-based production); and
- the wider effects of information as a factor of all production and transacting activities.
There is much still to be gained from understanding the impact of information in these areas that can inform understanding of the impact of the Internet, given that functionally the Internet is principally a tool enabling a widening of electronic communication networks that have already existed within and between firms.
The Measurement Problem
The relative newness of electronic technologies and the speed with which these applications have been adopted, have strained the ability of theoretical research to keep abreast of changes and their impacts. This has created pressure to measure something, even though the relevance of what is being measured may not be fully understood. This limits the usefulness of what has been measured. Demands for comparative benchmarking have driven research agendas, such as that of the OECD, to create standardised measures of tangible symptoms. But these statistics need to be interpreted with caution, as there are as yet few unequivocal associations with specific technology uptake and economic and social benefit. It is this area where research effort is still scant. This indicates that a key research priority is to take individual statistics of the technologies individuals and firms use to interact to interact (how they communicate), and expand understanding of what, why or how often they communicate, with whom they communicate, and what the social and economic outcomes are.
The Challenge for Research Agendas
In summary:
- there are many things that need to be measured to assess the economic and social consequences of electronic commerce (e.g. technologies, skills, capacities, information exchanges and uses, substitution and complementarity patterns, etc.);
- many of these things cannot as yet be measured, either because they are intangible, or because we do not as yet know or understand the causal relationships between things we can see and measure, and the outcomes they engender; and
- many of the things that are currently being measured are of dubious relevance to overall economic and social performance, yet they are currently being used as proxies in a relatively unquestioning manner in the pursuit of policy development and research investment.
Similar challenges face policy and decision-makers in all economies. For example, these same research shortcomings have been noted by NOIE in Australia. This finding is extremely insightful, given that the efforts of NOIE to benchmark electronic technology uptake represent probably the most comprehensive national statistical undertaking based upon the OECD electronic commerce performance measurement methodology.
The key appears to be in understanding how transacting entities use information, how new technologies change the costs and ways in which they use information, how these change the value created, and the distribution of that value. This requires research at the firm and transactor level in addition to higher levels of aggregation, to better understand information as the primary driver of value creation. Technologies of all forms (human, manual, electronic, etc.) are involved, necessitating an understanding substitution and complementarity in addition to individual processes and interaction. Only then will it be possible to make sound policy and investment decisions to promote (or discourage) the use of specific electronic technologies.
Recommendations for New Zealand
Performance Measurement Exercises
Despite the limitations of technology-related performance measures, they have become the de facto standard by which national economies are being judged. Currently, the sole systematic New Zealand-wide electronic commerce performance measurement exercises are those undertaken by Waikato University for FORST, and those of the management consultancies and market research companies. There is no systematic national reporting such as that undertaken by NOIE on a quarterly basis, and other government-commissioned reports have been isolated (e.g. MED/BRC (2000)).
Consequently, the figures quoted by foreign agencies such as the OECD, and organisations such as the Economist Intelligence Unit rely upon out-of-sequence, sporadic and self-reported findings, which are interpreted often without the benefit of the context in which they were collected. The result has been some unfortunately misleading rankings of New Zealand, relative to other countries. For example, the EIU Electronic Commerce Readiness Indicators moved New Zealand from 17th to 20th between 2000 and 2001, at the same time as Australia moved from 16th to 2nd, despite New Zealand having had a consistently higher uptake of electronic technologies than Australia, and the New Zealand policy environment becoming more similar to that of Australia's over that period as a result of the implementation of the telecommunications and competition law policies of the Labour-Alliance coalition. The only significant difference appears to be the level of reporting provided by NOIE on Australian uptake. These reports are fundamental inputs into the decisions of foreign operators when deciding on potential investments in New Zealand. Further, these findings are often made in ignorance of the purchasing and utilisation habits of New Zealanders, leading to comparisons based upon either prescribed bundles that do not reflect actual usage patterns, or posted prices rather than those actually paid. Absence of actual and reliable information and interpretation biases New Zealand's relative position in these rankings.
At the very least, it is recommended that New Zealand undertake a regular, standardised reporting of the type conducted by NOIE (see Appendix 3) to avoid these negative consequences. It is likely that membership of the OECD will ultimately require such reporting anyway, but a proactive approach would be welcome. It is vital, however, that these statistics report New Zealand's uptake of technologies in the context of the New Zealand environment, and benchmarked against other countries in full knowledge of the similarities and differences that may influence the interpretation of such statistics. This will require detailed research into the meaning and interpretation of each of the uptake statistics, and an understanding of how or why New Zealand observations may differ from those of other countries. Research of this type has already begun (e.g. Arnold and Evans (2000), and co-operative research with other OECD members will further inform this process. Analysis of the currently-used metrics in sections 2, 3, and 4 of this report provide a starting point for a more critical approach to interpreting specific technology uptake metrics.
Recommendation 1: Implement a New Zealand-based systematic performance measurement process.
Theoretical Research
As the New Zealand Government has only a limited budget to spend on Electronic Commerce research, it is important that research effort should build on, rather than replicate, the research undertaken elsewhere. Unless there are reasonable grounds to presume that the findings from such replication in New Zealand will be substantially different from those obtained elsewhere, there are strong grounds to support the conduct of unique research that adds to the body of world knowledge, while simultaneously drawing on that world-wide body of research as it becomes available.
The shortage of research based upon information usage, and the systematic interactions of information exchanges throughout an economy have been identified as key areas for future research. New Zealand has an opportunity to lead world research in these areas. The relatively small size and limited number of trading sectors in the New Zealand economy provide a viable sample for investigating these phenomena. Further, New Zealand's history as an early adopter and high uptaker of computerised technologies such as EDI, the Internet and mobile telephones mean that there is a mature information base available in some sectors to facilitate analysis. Specific economy-wide applications that could be researched include (but not exclusively):
- the role of information in the creation of the central banking network
- ubiquitous EFTPOS in New Zealand
- effects on information uses and communication resulting from the restructuring of information-intensive industries such as the Crown Research Institutes and the health sector
- information-based uptake of email and websites in rural New Zealand.
Recommendation 2: Invest in systems-based research of information uses in New Zealand, in order to relate observed symptoms and measurements to wider outcomes
In order to undertake research into information uses, it is recognised that the problem of the intangibility of information must be addressed. This research must also address the vexed issue of creating appropriate proxies that capture the relative values of these intangibles in order to conduct meaningful analysis. While this may not be a straightforward task, beginning to address it provides a useful refocusing from the past emphasis on technology to the new emphasis on information.
Recommendation 3: Invest in research to develop suitable proxies for the value created from intangible information variables
It is noted that the research currently being undertaken into uses of specific technologies within companies is important as a complement to recommendations 1 to 3. The relationships established as a result of this research provide valuable insights into the operation of firms in New Zealand, a litmus test of the validity of the findings of systems-based information research, and valuable understandings of where New Zealand technology uptake observations vary from those of similar countries. Thus the systems-based information research proposed is in addition to that research already commissioned.
Recommendation 4: Continued research into electronic technology impacts is important.
International Co-Operation
Recognising that information is a good exhibiting network property and that research is an information product, there are economies of scope to be gained by participating in joint research wherever possible. These economies will be greatest when joint research is undertaken with an entity or organisation whose environment most closely resembles that of New Zealand. The obvious contender for a joint research partnership with New Zealand is Australia, given that NOIE officials have expressed dissatisfaction with the limitations of technology uptake measures, and an interest in investigating other information-based applications. Such research could be for the mutual benefit of both countries, and will enable access to a wider base for data gathering and hypothesis-testing. Furthermore, joint research may carry greater credibility on the world stage, given that an agenda of information-based rather than technology-based research may be at variance with much of the body of technology-focused research conducted elsewhere. Therefore, it is strongly recommended that a research partnership with NOIE be investigated.
Recommendation 5: Investigate the possibility of a research partnership with NOIE
Summary
These recommendations provide a research agenda, but there is no guarantee that the outcomes of such research will provide instant answers to the taxing questions of how to develop policies to change outcomes in New Zealand. There is still much that we do not understand about the operation of the economy we have presently, let alone how technology or information-specific policies may change it. The only guarantees are that such research will add to our understanding of things that in the past have been intangible.
There is no "magic bullet" to be found - but there is a great deal to learn.
Back to Top