5 .Conclusions
We have compared a number of key variables, including subjective and objective measures of firm performance, drawn from the first two years of the Business Operations Survey with IRD tax returns and financial accounts information. There is much commonality in the picture we see using either administrative (tax) or quantitative survey data, giving us some comfort that the data, while not collected for statistical purposes, serves us well as a tool for measuring firm performance. This is not a trivial result – survey data may be considered superior to tax data because questions are designed to collect the right conceptual variable while, conversely, tax data may be considered superior because, for example, firms could be made subject to audits with penalties for inaccurate filing. The fact that we find some concordance across our quantitative data sources suggests either that (a) these pros and cons balance out, or that, (b) the data comes from the same source (financial accounts). Demonstrating the usefulness of the tax data for economic research enables us to confidently construct longitudinal performance measures for survey respondents (see Fabling et al., 2008 for more on this subject).
The one exception to this is profits. Total taxable profits reported in the financial accounts IR10 form and operating profits calculated from the BOS are clearly measuring different things. There is a considerable difference in the levels of profits between the two sources, although they are significantly correlated. Our suspicion is that this is not due to cyclical write-off behaviour (although we cannot be certain without a longer time series of data). It may in fact be due to something as simple as the wording of the examples provided in the definition of "other expenditure" in the BOS. However, our decomposition is only indicative and we suggest that investigation of a longer time series of IR10 data and/or additional cognitive testing with the BOS are required before a more definitive answer can be provided.
Another exception is our counts of working proprietors. Part of this is because it is not entirely clear precisely who should appear in such counts and that both measures are rather incomplete. However, the impact on total employment is minimal. Whilst there is a levels difference between the measures of employment, the measures are highly correlated, and we suspect that the difference is due to what the different sources measure. Whilst a single point in time measure (as in the BOS) may be better in terms of the quality of individual responses, it may be that rolling mean employment is a more appropriate measure of employment when analysing annualised data. Use of either measure is likely to lead to classical measurement error in the subsequent estimation of models (the effects of which are relatively simple to understand) rather than impart more complex bias (although one exception may be in making between-industry comparisons where there are differences in the cyclicality of employment).
Despite the differences in the level of employment, measures of productivity obtained from the BOS and a combination of BAI/IR10 and LEED are significantly correlated. The BOS based "objective" measure of productivity tallies fairly well with firms' report of their productivity relative to their competitors, as does the measure of relative profitability. This is despite ambiguity that surrounds the questions such as: what do we mean by productivity (labour or multi-factor productivity)? and who are the firm's competitors (does it correlate to a particular ANZSIC classification)? These ambiguities create problems for what cognitive psychologists call the comprehension of the question's semantics (identifying a question's focus) and pragmatics (linking the terms used to actual concepts), as well as their ability to retrieve any information – they may be able to find information on their own profitability and productivity (either from financial accounts or introspection) – but from where do they obtain reliable knowledge of their competitors?47
One implication of our results is whether financial questions are required in the BOS. There are always calls to reduce respondent load. There are at least two reasons for this. First, because we do not want to place an unnecessary burden on firms. Second, because data quality may be reduced if respondents feel that filling in such firms is an unnecessary burden. One way to reduce the burden is to reduce the number of questions asked of firms, by removing questions on financials and employment, and replacing the information with data from administrative sources.
There are a number of complex issues that need to be considered in addition to data quality issues. One issue is legal. We have had access to confidentialised data under strict guidelines in the DataLab of Statistics New Zealand. It is unlikely that these can or will be relaxed in the future and this may restrict the individuals who are able to access such data at the unit record level.
Not withstanding these issues, we can say something about the data quality if these are overcome. There are a number of reasons to be cautious about replacing the financial and employment data in the BOS with data from administrative sources. At a general level, one should be wary of using data from a number of different sources. Definitions and frames of reference may be crucially different. Our comparison of sales data showed that we only have IR10 data for around 60% of the cases for which we have BAI and BOS data and availability is a function of sales, i.e. firms that do not return IR10 forms appear to be larger than average. Whilst sales data sourced from the BOS and BAI are for most practical purposes equivalent, one caveat is the potential for sales data in the BAI to be contaminated by capital sales. Because of its stratification methodology, the Annual Enterprise Survey does have a full coverage strata made up of large units with significant economic activity within their industry group and has been used successfully to patch holes in BAI/IR10 data (e.g. Maré, 2008; Fabling and Grimes, 2008).
Back to Top