• Home
• About MED
• Publications
• Occasional Papers
• 08/04
• 4. Results

• Publications

• Occasional Papers

• 09/01
• 08/09
• 08/08
• 08/07
• 08/06
• 08/05
• 08/04
• 08/03
• 08/02
• 08/01
• Archive

4.1 Comparing Self-reported and Official Quantitative Data

We begin our analysis with a comparison of financial data reported in the first section of the BOS with that available from administrative sources. The first section of the BOS contains information on the financial position of the firm, including operating revenue and expenses, assets and liabilities, proportion of sales exported etc. Whilst this data provides a useful picture of the NZ economy, there are other sources for this information. The primary purpose of collecting this information is to provide context for information obtained from later questions. For example: how does exporting or innovation behaviour vary by firm size? are firms which undertake certain business practices larger or smaller than those that do not? and are they more profitable and or productive? This analysis can take the form of cross-tabulations (as seen in the "Hot of the press" publications produced by SNZ), publications like Knuckey and Johnston (2001) and SNZ/MED (2005), or more sophisticated econometric analyses (e.g. Fabling and Grimes, 2007). In order for these analyses to be robust, they need good measures of financial information with which to investigate the determinants and impact of variables of interest to researchers and policy makers, like competition, R&D, employment and management practices, use of ICT, access to finance, or exporting.

4.1.1 Sales

Our first comparison is sales of goods and services. Some work is required to ensure consistency across the data. Respondents to the BOS were asked to supply GST exclusive amounts when supplying financial information. Where respondents have indicated that the figures do in fact include GST, GST exclusive figures are computed by removing the component that is exported (and thus not liable for GST) and multiplying the remainder by 8/9 (and then adding back exports). This only affects a very small proportion of observations. The data in the BAI are GST inclusive and those in the IR10 are GST exclusive.¹⁷ Because of this, in what follows we also adjust the figures taken from BAI returns.¹⁸

Table 1 and Table 2 show the results of our comparison of sales across the alternative sources.¹⁹ We can see from the first set of columns in Table 1 that we are less likely to have sales figures from IR10 returns (we only have IR10 data for around 60% of the cases for which we have BAI and BOS data).²⁰ Moreover, firms that do not return IR10 forms appear to be larger than average (for our sub-sample of firms taking part in the BOS). Because of this, and analysis that suggests that BAI purchases data are more appropriate than IR10 purchases data for constructing productivity measures (Cox, 2007), we concentrate on BAI sales data from here on in.²¹

The second set of columns provides data on sales for firms for whom we have all sets of data. In the top half of the table this refers to all three sources. In the bottom rows we report figures for firms with both BOS and BAI data (because of the number of firms for whom we do not have IR10 forms). The figures in the BOS are slightly higher than those in the IR10, but lower than those in the BAI. We can consider this more formally, using a Wald test of the significance of the difference between the means of the pairs of variables. The final columns report the results of our test of the hypothesis that the IR10 and BAI figures are significantly different from the BOS. We cannot reject the hypothesis of equality between the two alternative administrative sources of sales data and that obtained from the 2005 BOS. When we look at the 2006 data, we can accept the hypothesis that the sales data from the IR10 are lower than those reported in the BOS (at the 10% level), but not the BAI. When we look at the combined years' data, the difference between BOS and IR10 sales is significant at the 5% level. However, these differences are relatively small, being less than 5% of sales.

When we restrict ourselves to the, much larger, sample where we have both BAI and BOS data, the most striking change is that the mean value of sales is much larger. The relative differences between the figures from the different sources are much smaller and are nowhere near significance in the statistical sense.

• Figures based on sample strata and weights (except the observations which relate to unweighted data)
• All figures exclude GST
• n rounded to nearest five for confidentiality reasons
• Figures for GST inclusive BAI sales are brought into line with GST-exclusive BOS figures by multiplying non-zero-rated GST sales by 8/9

The correlations between the three measures of sales can be seen in Table 2. These are pair-wise correlations and so are based on all firms for which we have data on the two respective measures. There is a high degree of correlation between the three measures of sales. The correlation with the sales reported in the BOS is slightly higher for the IR10 data than for the BAI, although the rank-correlation is almost identical.

• Whole sample (i.e. both years)
• *Figures based on sample strata and weights

From our analysis, we conclude that the sales data sourced from the BOS and BAI are for all practical purposes equivalent. We must be more wary with the IR10 data, because of (a) lower reporting rates, and in particular with the probability of reporting appearing to be a function of sales; and (b) the level of sales reported in the IR10 tending to be lower than that reported in both the BOS and the BAI. We must mention one caveat, however. That is the potential for sales data in the BAI to be contaminated by capital sales. In the BOS, respondents are specifically requested not to include proceeds from the sale of fixed assets or gains on the sale of fixed assets. These may be present in the BAI data. For more on this subject see Fabling et al. (2008).

4.1.2 Profits

The term "profit" can be interpreted in a number of different ways. Tax accountants have one interpretation, with distinct legal and behavioural implications, and economists have another. In simple terms, firms have an incentive to minimise the former and maximise the latter, ceteris paribus. Our ability to make comparisons is subject to data availability. GST-based data does not contain information to allow a comparison with the BOS. For a comparison of administrative and survey measures of profits, we use data from IR10 returns. One disadvantage of this is that it reduces our sample size (because of the lower response/submission rates for iR10 returns). The administrative source of profits is "total taxable profit" from the IR10 return.²² We calculate operating profit from the data in the BOS as total operating revenue less operating expenses.²³

It is clear from Table 3 that total taxable profits from firms' IR10 returns and operating profits from the BOS are rather different things. Taxable profits from the IR10s are considerably smaller than operating profits calculated from the BOS. This is not due to the larger average size of IR10 non-respondents (which we saw from the sales figures in Table 1), although the figures are a little closer when we consider the common sample. Even in the sample of firms for which we have both sources of profits, those calculated from the BOS are on average three to five times larger than the sum firms report as taxable profits.²⁴

• Figures based on sample strata and weights (except the observations which relate to unweighted data)
• All figures exclude GST
• n rounded to nearest five for confidentiality reasons

Despite the difference in the levels of profits, they are significantly correlated (Table 4). In particular, note the higher rank correlation. It appears that one profit definition is a monotonic, non-linear transformation of the other. Thus, analysts using these different profits might come to some similar conclusions, but this is by no means certain, especially where the raw data (rather than rankings, or groupings such as deciles or quartiles) are used.

• Whole sample (i.e. both years)
• *Figures based on sample strata and weights

One explanation for the difference between the two measures of profits is that the two should be measuring different things, because of different definitions (for example, the – on average – minor effect of changes in stocks). Another is that there are clearly incentives to reduce the amount of profits liable to taxation. Whilst the IR10 is not used for calculating tax liabilities, it is collected by the Inland Revenue Department and so one would expect firms to wish to present a picture that is consistent with tax returns.²⁵ The incentive to reduce taxable profits may have a temporal dimension – for example, firms may shift profits or write-offs across years for tax or other purposes – and we may have just chosen two years when firms have tended to over-report expenditure in their IR10s.

One thing we can do, however, is consider the components of profits in more detail. Revenue in the BOS is broken up into two components: that from "the sale of goods and services" and that from "all other operating revenue". The survey provides notes for respondents as to what to include in "all other operating revenue". Respondents are asked to include: "renting and leasing income", "government grants received for operating purposes", and "interest and dividend revenue". They are asked to exclude: "proceeds from the sale of fixed assets" and "gains on the sale of fixed assets". In the IR10 form, income includes "gross income from sales and/or services", "interest received", "dividends", "rental and lease payments", and "other income".²⁶ The instructions in the BOS closely match the boxes in the IR10. The only exception is that the BOS mentions "government grants received for operating purposes". No explicit mention of government grants is made in the notes on the final page of the IR10 form. The notes for "other income" ask filers to: "Include all other sources of income that would be shown in the trading or the profit and loss account. This includes, for example, subvention receipts, depreciation recovered, deferred income assessed this year, income spread forward into this year and Rural Bank suspensory loans forgiven". There is also no mention of grants in the Guidelines for completing the IR 10 document (IR10G) that IRD produces.

The top section of Table 5 shows income/revenue in the BOS and IR10, broken down into sales and other income/revenue (for the IR10, this is the sum of the four non-sales items). We can see that each of the three items is recorded as being higher in the BOS than in the IR10. However, our F-tests suggest that we can accept the hypothesis that firms report higher sales in the BOS (at the 5% level), but not other income. In part, this latter result is due to the much higher variance in reported "other operating expenditure" in the BOS. This could be taken as prima facie evidence for greater uncertainty among BOS respondents as to what they should include in their answer to this question. Nevertheless, other income/revenue is on average twice as high in the BOS as it is in the IR10. This accounts for a large proportion of the additional total income/revenue reported in the BOS. The test for the difference in means for total revenue is just significant at the 10% level.

• NB Comparisons are made over total firms for which all data are available
• Number of observations = 6,500 (rounded to nearest five for confidentiality reasons)
• Figures based on sample strata and weights

The bottom sections of Table 5 show the components of expenditure. The BOS breaks this down into four groups, the amount the business paid in: "salaries/wages", "interest", "depreciation and amortisation" and "all other operating expenditure". The notes in the BOS ask respondents to include in "all other operating expenditure" the following: "purchases of goods and services from suppliers" and "renting and leasing costs". Respondents are asked to exclude the following expenditures: "salaries and wages", "purchase of fixed assets", "interest and finance costs", "depreciation or amortisation", and "losses on sales of fixed assets". The IR10 form has sixteen expense categories (these are detailed in the appendix, with descriptive statistics in Table 21). It has categories that appear to exactly mirror the three specific categories in the BOS. One slight difference is that respondents to the BOS are asked to include employee ACC contributions in the total amount the business paid for salaries and wages. In the IR10, respondents are asked to include ACC levies in "other expenses".

For the specific categories in the BOS where we can make a direct comparison, firms report slightly higher amounts for expenses than in the IR10. However, the only category for which this is statistically significant is for salaries and wages. This is in part due to the additional ACC costs included in the BOS salaries and wages. However, this is not sufficient to explain the majority of the difference.²⁷

Despite the similarities between the specific categories of expenses – and indeed the tendency for firms to report slightly higher values in the BOS than in their IR10 – overall expenses are much higher in the IR10 returns than in the BOS. As we can see from the table, the majority of the remainder of expenses is made up of purchases. Note that if we sum the IR10 categories corresponding to the two examples of "other operational expenditure" provided in the BOS notes – "purchases" and "rental and leasing costs" – the figures are much more similar than the overall total of other expenditures. Indeed, we cannot reject the hypothesis that they are the same in our Wald test.

This raises the question of how respondents deal with "other"-type categories in such forms. One hypothesis is that they have an actual figure for the total amount. They then remove the components they are asked to supply and put the figure for the remainder in the "other" box. We call this the "top down" response. Another is that respondents consider the total as the sum of the parts. We call this the "bottom up" response. In cases where the totals are clearly defined in the eyes of the respondents, one would expect respondents to calculate the "other" category using the top down approach. If there is some ambiguity – in either the total or the other component categories – respondents may employ the bottom up method, or a combination of the two. In the top down case, the instructions for what to include in the other category are surplus if what should be included in total and the other component categories is clear. In the bottom up case, respondents rely much more on the instructions in the questionnaire. It is possible that respondents see the examples of "other operational expenditure" as an enumeration of the contents (i.e. a complete listing of all the items that should be included in the category). The results of our analysis presented in Table 5 show that we can reject the hypotheses that either the "total" or the "other" expense categories in the BOS and IR10 are the same across firms. However, to reiterate the previous paragraph, we cannot reject the hypothesis that respondents report the same value in the "other expenditure" category in the BOS as they do in the IR10 categories that are specifically itemised as examples of BOS "other expenditure". This is prima facie evidence for the "bottom up" approach, or what we might call an "enumeration effect".²⁸

One potential culprit that we noted above for differences between the profits is the writing-off of bad debts. Firm might choose to write-off bad debts in good years in order to offset them as expenses against profits. These will show up in the financial accounts in the balance sheets as current assets until they are written-off, when they appear on the profit and loss account as expenses. Because of the tax incentive to write bad debts off in good years, one might expect the writing-off of bad debts to be pro-cyclical. This may be the case, but the size of bad debts expenses is rather small compared to the difference between total expenditures as recorded in firms' BOS and IR10.²⁹ According to the IR10, over half (55%) of operating expenses is made up of purchases. The next largest proportion (18%) is made up of salaries and wages. Of the remainder, around half is made up of fourteen expense categories named in the IR10, none of which contribute more than 3%.³⁰ The other half is the ubiquitous "other expenses" category.

Our analysis suggests that the majority of the difference between total taxable profits, as recorded in the IR10, and operating profits in the BOS is due to the much higher amounts recorded as expenses in the IR10. The obvious explanation for this is that there are incentives for firms to reduce their tax liability by ensuring expenses are as high as reasonably possible and income as low. Note that the IR10 is "designed to collect information for statistical purposes" (SNZ: IR10G: IR10 Guide) and not for calculating tax. Another potential explanation for the difference is that there are many more expense categories listed in the IR10 than in the BOS and this may stimulate respondents to include more expenses in the IR10 (what we have called the "enumeration effect". In the language of Tourangeau et al. (2000), this may cause problems with comprehension, causing respondents to either retrieve the wrong information or judge that some of the information retrieved does correspond to the data required. When we have compared the sum of the two expenses listed in the BOS notes as components of "other operating expenditure" – purchases and rental/ lease payments – with equivalent entries in the IR10, the results are quite similar.

4.1.3 Employment

Employment raises rather different issues to the financial variables. Employees can be full or part-time; they can be temporary or permanent; they can be employed for the whole of the year or part of it; some staff may not be employees (e.g. working proprietors). In this paper our administrative measure of employment is made up of two components: employees and working proprietors. Our measure of employees is defined as an average of twelve-monthly PAYE employee counts in the year (known as rolling mean employment, or RME).³¹ This takes into account some of these complications (e.g. part-year working), but not others (e.g. variations in hours worked, such as the difference between full-time and part-time workers). Our measure of working proprietors also comes from the LEED, but is rather more complex. It is a count of the number of self-employed persons who are paid taxable income during the tax year. This is based on a number of IRD forms and is calculated on a March year-end basis. For more information on the calculation of this figure, see the data appendix.

Employment in the BOS is broken down into full-time (working 30 hours or more per week) and part-time working proprietors and employees. Respondents are asked to exclude contractors from employees. In the 2006 survey respondents are also asked to include the total headcount (FT and PT) of workers.³² We calculate two measures of employment from the BOS. Our headcount measure is simply the sum of FT and PT workers. Our FTE measure assumes that PT workers work half the hours of FT workers (i.e. FTE=FT+0.5PT).

The comparison of employment from the LEED and BOS employment is presented in Table 6. Looking first at working proprietors, the BOS headcount measure is higher than the equivalent LEED count, with the BOS FTE measure somewhere in between. This result is fairly independent of whether we consider the separate sample or focus on the set of firms for whom we have both figures. In general, firms tend to include around half an extra working proprietor in their responses to the BOS that one would expect given the LEED data. This result is statistically significant at the 0.1% level. This may in part be due to the fact that some of the working proprietors included in the BOS do not draw an income from the firm during the year. However, on the contrary, it is also possible for individuals to be included in the LEED measure of working proprietors because they receive non-wage income. One might expect some of these to be excluded the BOS working proprietor count. Another explanation is that some respondents to the BOS included some contractors (they may not have realised a staff member was a contractor, or may have misread the instructions).

• NB "Test of inequality" is with LEED figure
• Figures based on sample strata and weights (except the observations which relate to unweighted data)

Turning to the employee numbers, we can see that there is something slightly unusual going on in the BOS sample for 2005. The variance (and thus the standard error of the mean)³³ is considerably higher for BOS employee numbers in 2005 than for either the same question(s) in the BOS 2006 or the LEED RME counts in both years. It should be noted that the question in the BOS changed slightly between 2005 and 2006. The changes were mainly formatting, with boxes and notes moved slightly. One major change was that in 2006, respondents were asked to include a total in addition to the numbers of part-time and full-time working proprietors and employees. These changes were made in response to problems survey respondents appeared to have with the employment questions.³⁴ Because of the higher variance in responses to the 2005 survey, we cannot quite distinguish statistically between the BOS headcount and the LEED RME figures (at standard statistical levels), despite the fact that the respondents to the BOS appear to record 25% more employees than one would expect given the LEED figure. For 2006, the standard error of the mean of BOS employment drops to something much more akin to that for the LEED figure. Furthermore, the mean estimate of employment also drops to something closer to the LEED figure. Nevertheless, respondents to the BOS still report employment around 10% higher than the figure obtained from PAYE records; a difference that is highly statistically significant. The BOS FTE measure of employment is much closer to the LEED figure in both years, although this difference is still statistically significant in 2006.

As one would expect, given the relative numbers of working proprietors and employees, the results for total employment are similar for those for employees. One difference is that now the difference between the BOS headcount and LEED RME figure for total employment is statistically significant at the 10% level (although not the 5% level).

Why are the BOS headcount figures higher, but the FTE figures close to the LEED RME figures? Is there a tendency to over-report temporary workers in the BOS, or is there some other explanation? There are a number of potential explanations, relating to the individuals who are included, the time frame over which individuals are counted and the type of employment. LEED has a very specific population: those for whom a PAYE form is submitted. Individuals will not be included in the PAYE records if they work without pay. Also students earning less than $20 per week, or no more than $1,400 per year may not be included (although this depends on the practices of their employer).

The BOS figure is a spot estimate of employment. Respondents (either from the Human Resources department or if they do not have one, the General Manager) are asked: "As at the end of the last financial year, how many staff worked for this business?" LEED RME is on the other hand an average of monthly spot estimates over the whole year. There are at least two reasons why this might lead to a higher value being reported in the BOS than in the LEED. The first is that BOS respondents answer exactly the question they are asked, and provide a true estimate of staff numbers at that particular point of time, but employment tends to peak at the end of the financial year. There is some reason to expect this to be the case as most financial years end in March and so this might include seasonal employment at the end of summer. The second is that BOS respondents suffer a recollection bias and do in fact report their recollection of staff (either individually or the numbers involved) over the year and misattribute the times over which they are employed. This is qualitatively similar to the concept of "telescoping" in the survey literature. An example of this is when respondents misreport times for which they were employed or unemployed (Akerlof and Yellen, 1985; Mathiowetz, 1986). The result of this consideration of total staff over a longer period (e.g. the whole year) is to overestimate the proportion of staff that were employed at the end of the financial year.³⁵

The correlations between the different measures of employment are all highly statistically significant (Table 7).³⁶ The correlations between BOS and LEED employment are little different when we consider the BOS FTE and headcounts measures, although the difference is slightly larger in 2005 than in 2006.

There is a relatively low degree of correlation between the survey and administrative measures of working proprietors, particularly in 2005, although the much higher rank correlation suggests that the relationship between the two is a non-linear but fairly monotonic transformation (i.e. it affects different parts of the distribution differently, but has much less of an impact on their ordering within the distribution).

There is a much higher correlation between the BOS measures of employees and total employment in both years. Once more it is much lower in 2005 than in 2006, although the rank correlations are almost identical.

Overall these results suggest that the two sources of data are very similar in 2006, with a tendency for the BOS figures to be slightly larger than those from LEED. There is clearly quite a difference between the 2005 and 2006 responses. Our reading of the results, in combination with discussions with SNZ staff, suggests that there were problems with the responses to the employment questions in 2005.

4.1.4 Productivity

Productivity is an important determinant of the wealth and welfare of economies (Prescott, 1998; Kneller and Stevens, 2002). Because of this, it is a key variable of interest to economists and policy-makers alike. When we consider the impact of factors such as competition, innovation or management capability on the economy, one of the key indicators is productivity. Having good measures of productivity therefore is extremely important.

In this paper we shall focus on the log of labour productivity,³⁷ equal to the log of value-added minus the log of employment. We construct a measure of labour productivity from both administrative data³⁸ and from the BOS.³⁹ Our measure of productivity is essentially made up of three variables - sales, purchases and employment – each of which may be measured with error. The impact of these individual errors on the measurement of productivity – and hence our comparison of the two measures – is ambiguous, as it depends on the correlation between the errors, as well as their size. Nevertheless, our expectation is that overall measurement error will be below that of the sum of the individual components, and hence the BOS and administrative data will be more similar.

As we can see from Table 8, the two measures are significantly correlated with each other, but the value of productivity obtained from the BOS is statistically significantly higher than that obtained from the BAI. One reason for the difference might be the contamination of sales and purchases data in the BAI by capital sales and expenditure (c.f. Fabling et al, 2008). However, it is not certain this will bias the sales data in either direction. Indeed, on average one would expect sales and purchases of capital goods to approximately even out.⁴⁰ In the BOS, respondents are specifically requested not to include the purchase of fixed assets or losses on sales of fixed assets. Because of this, one would expect value-added – and hence productivity – to be higher in the BOS measure, ceteris paribus. However, our measure of purchases in the BOS is "all other operating expenditure" and so may include other expenses (although our discussion of Table 5 suggests that many of the other expenses are not included). Turning to the employment side of the productivity calculation, recall from our discussion of Table 6 that our headcount measure of employment is significantly higher in the BOS than in the LEED, which would tend to reduce the measure of labour productivity calculated from the BOS financials relative to that from the BAI/LEED.

• Weighted and stratified (except Spearman correlations)

Whilst we might be worried that labour productivity calculated using the BOS is significantly different from that calculated from the BAI data, we should caution that we are seldom interested in the level of productivity per se, except to make comparisons between firms. We are interested mainly in relative productivity, i.e. in understanding why some firms are more productive than others. The fact that they are significantly correlated – despite the definitional differences – and in particular the high rank correlation suggests that any policy prescriptions based upon either measures will not give results that are too far different from those based on the other. However, it would be useful to understand the differences in more detail.

4.2 Comparing Self-reported Subjective/Qualitative and Quantitative Data

In this section we turn our attention to subjective measures of firm performance. Such measures are often the only information analysts have at their disposal and so their correlation with more objective – but harder to come by – measures of firm performance is of considerable interest. In a manner similar to Forth and McNabb (2008a) we compare self-reported qualitative, subjective measures of performance with "objective" measures derived from financial information reported in the BOS.

• Table refers to the BOS Question 38: "How do you think this business compares to its major competitors on each of the following? Profitability; Productivity"

The two subjective measures of performance we consider are relative profitability and productivity. Question 38 of the BOS asks firms how they think their business compares with its major competitors against both these metrics. Taken at face value, Table 9 confirms our suspicion that respondents tend on average to consider themselves above average. Almost half of the respondents in both years and for both questions think of themselves as "on a par with competitors". Around twice as many firms reported themselves as more profitable than their competitors than less,⁴¹ and around six times as many feel they are more productive than less. This is consistent with the results for output per head and value-added per head in the Workplace Employment Relations Survey (WERS) reported in Forth and McNabb (2008a), although the respondents to the WERS have five choices ("A lot below average", "Below average"; "About average"; "Above average"; and "A lot above average"). The majority of (weighted) responses (over 80%) in Forth and McNabb are either "About average" or "above average" (with the split being roughly equal between the two).⁴²

One explanation for this apparent upward bias is the large number of firms that answer "don't know" to this question (Fabling and Grimes, 2007, find striking similarities between the "don't know" group and a combined "low/average" group). For example, firms that reported that they did not know might be less profitable or productive, and either (a) the factors that prevent them knowing are correlated with their low competitiveness (i.e. managers who cannot see how well the firm is doing are not good managers, which makes the firm perform less well)⁴³ or (b) they know the truth and cannot bring themselves to admit it in writing. We can get a better understanding of whether the second explanation is true, by comparing the subjective measures of profitability and productivity with objective ones. This we now do.

4.2.1 Productivity

Considering productivity first, we can compare the subjective estimate of (relative) productivity with a labour productivity measure from the financial information from Part i. of the BOS. There are a number of ways in which respondents could be answering subjective questions with respect to productivity. First, there are issues about the appropriate denominator, i.e. whether firms are considering labour or some type of multi-factor productivity. Second, there is the issue of the appropriate numerator, e.g. sales or value-added. Not all respondents will have economics degrees and so this question is open to multiple interpretations. Earlier evidence (Forth and McNabb, 2008a) suggests that managers' reporting of subjective measures of profitability are closer to objective measures than they are of productivity. Forth and McNabb suggest that this reflects greater clarity as to what is being asked about profitability than productivity. However, as we have noted in footnote 41 above, Forth and McNabb do not report the proportion of WERS respondents that are in the "no comparison possible", "relevant data not available" and "not answered" groups. Forth and McNabb (2008b) do for productivity and 4.6% of respondents state that no comparison is possible, 2.5% say that the relevant data are not available and 4.5% did not answer (weighted figures). In the case of the BOS more firms respond that they do not know their profitability than report they do not know their relative productivity. It is interesting to note that more respondents report that they do not know their relative productivity in the BOS than do in the WERS. This may be because the WERS does not contain financial questions requiring the respondent to either have access to the firm's accounts or some knowledge of them, as the BOS does. It may, therefore, be clearer to WERS respondents that such a question is an estimate than it is for BOS respondents that have just filled in several pages of information on income, expenditure, assets, liabilities and employment.

The mean values of labour productivity for each group, their standard errors and 95% confidence intervals are presented in Table 10. Those who report their productivity is lower than their competitors clearly have the lowest labour productivity, although the difference is larger in 2005 than 2006. Those who believe they are more productive than their competitors indeed appear to be more productive than those who believe they are on a par with their competitors. Firms who report that they do not know how their labour productivity compares with their competitors tend to be around or just below the "on a par with" group.

We can make this comparison more rigorous by performing Wald tests of the equality of these estimates. As we can see from Table 11, we can accept the hypothesis that the means of productivity of the three groups are different. We cannot distinguish the productivity of firms in the "don't know" category from the "lower than" or "on a par with competitors" groups, but we can from the "higher than competitors" group.

• Table shows values of the log of labour productivity by subjective relative probability
• Weighted and stratified

• Top section of table reports two-way Wald F-Test of inequality of means of productivity between groups along with probability that the difference is not significantly different from zero
• Bottom section joint test that three categories of relative productivity are equal and that these are jointly equal to the "don't know" category, respectively
• Weighted and stratified

This result is confirmed when we perform an OLS regression of labour productivity on dummy variables representing the subjective relative productivity groups and the "don't know" category (with "On a par with competitors" as the baseline group) (Table 12). The significant, negative coefficient on "lower" confirms that firms that report their productivity is lower than competitors indeed tend to have lower productivity than firms that think they are just as (or more) productive than their competitors. Likewise, firms that believe that they are more productive than their competitors indeed tend to be so. Our results therefore support the thesis that the subjective data have some predictive power and correlate with objective measures.

• Table reports results of a linear regression where the dependent variable is the log of labour productivity. Independent variables are dummy variables representing the subjective relative productivity groups and the "don't know category" (with "On a par with competitors" as the baseline group).
• Observations rounded to the nearest five for confidentiality reasons
• Weighted and rounded

One of the great strengths of the LBD is its breadth. Because we have data from the majority of economically significant firms in New Zealand, we can consider a more interesting alternative measure of relative productivity – the productivity of the firm relative to all off its actual competitors. The key question is how we define competitors. This is no simple matter.

There are a number of reasons why one would suspect that it is difficult to define the market within which firms operate and hence identify its competitors. Firms may operate in more than one product market. They may compete locally, nationally or internationally. It may be the case that some ANZSIC codes do not necessarily correspond to definitions of a "market" and this may vary not only by product type, but also by firm. Previous work suggests that there is considerable heterogeneity in how firms respond to questions where they are asked to compare themselves to their competitors (e.g. Mason, 2005). Part of this is due to the nature of the products or services firms are offering, part is due to the fact that firms often only compare themselves to firms with which they fell they can compete. Firms producing a much higher or lower quality product are frequently considered to be in a different market, despite the fact that the ANZSIC classification of their operation is the same.

Notwithstanding these caveats, however, it is possible to consider firms with similar industrial classifications. We examine two alternatives – defining the group of firms in competition with the firm as all those in the 3-digit or 4-digit ANZSIC industry in which the firm is situated. We calculate these measures in two ways. In the "unweighted" calculation, we compare the firm's productivity relative to the simple mean of all firms in the 3- or 4-digit industry. In the "weighted" version, we compare it to the mean of all firms in the relevant industry, weighted by sales.

• Standard errors in parentheses
• ^* significant at 10%; ^** significant at 5%; ^*** significant at 1%
• Note that all results are weighted and stratified, "Unweighted" and "Weighted" column headers relate to method of calculating industry average productivity

The results of this exercise are presented in Table 13. The first thing to note is the consistency of results across the specifications. The coefficient on the "lower than" variable is of the expected sign (negative), but is statistically insignificant across all specifications. The coefficient on the "higher than" variable is also the expected sign (positive) and is statistically significant. The coefficient on the "don't know" variable is even less significant that that on the "lower than" variable. These results are similar whether we use the 3- or 4-digit industry measure. The coefficient on the "higher than" category is higher when we use the weighted calculation.

4.2.2 Profitability

The picture is similar when we consider profitability (Table 14 to Table 16), where profitability is defined as profits divided by sales.⁴⁴ Profitability certainly rises over the groups of firms in each of the self-reported profitability groups. The picture is consistent across years. Those in the "lower than competitors" group have a profitability that is around zero and even have slightly negative profitability on average in 2005. This difference between groups is statistically significant and at standard levels of significance (Table 15). The "don't know" group appears to be equivalent to (or slightly lower than) the "average" firm. The mean value of profitability is statistically indistinguishable from the "on a par group", even at the 10% level. It is distinguishable from the "higher than" group at the 1% level, but only at a lower (7%) level from the "lower than" group. These results again hold when we take a multiple regression approach (Table 16).⁴⁵

• Table shows values of profitability by subjective relative profitability
• Weighted and stratified

• Top section of table reports two-way Wald F-Test of inequality of means of productivity between groups along with probability that the difference is not significantly different from zero.
• Bottom section reports joint test that three categories of relative productivity are equal and that these are jointly equal to the "don't know" category, respectively.
• Whole sample (i.e. both years)

• Table reports results of a linear regression where the dependent variable is profitability. Independent variables are dummy variables representing the subjective relative profitability groups and the "don't know" category (with "On a par with competitors" as the baseline group)
• Observations rounded to nearest for confidentiality reasons
• Weighted and stratified

4.3 Perceptions of Changes in Performance

Up until now we have considered BOS responses to questions about the current financial year. Respondents are also asked about how they believe their business performance changed over the last financial year (with regard to their sales, profitability, productivity or market share). In this section we consider firms' perceptions of such changes with alternative objective measures. Using the data from the BOS, BAI and IR10s, we can analyse the changes in these variables in each of these groups. The relevant question in the BOS is Module A Question 39 "Over the last financial year, did the following items decrease, stay the same or increase for this business?"

The only variable that we shall be considering that we have not already described is market share. We calculate two market share variables, differing only in their definition of "market". In these we define market share as the share of total (BAI) industry sales at the 3-digit and 4-digit level (ANZSIC).

The results of our comparisons are presented in Table 17 and Table 18. Table 17 uses BOS data exclusively and considers the respondents' responses to the qualitative question of whether sales, profitability and productivity stayed the same or increased over the last financial year. Because we are using financial information from the BOS, the table only relates to the qualitative question asked in the 2006 BOS.

The first four columns in the table present the mean value of the change in each of the variables, with its linearised standardised error. The means of the "decreased", "increased" and "don't know" categories have asterisks appended if we can accept the hypothesis that they are different from the "stayed the same" category.

Overall, whilst there is some correlation between the quantitative and qualitative measures, the results are not uniformly strong. The mean of the change in each of the quantitative performance variables is lower for firms who report they decreased than those who reported that they stayed the same. However, this difference is generally not statistically different. We can, however, accept the hypothesis that firms that report their sales have increased have experienced higher sales growth than the "stayed the same" (and indeed the decreased) group.

The mean value of each of the (change in) objective performance variables are insignificantly different from zero for firms who put themselves in the "stayed the same" group. The mean of the change in the variables for firms in the "don't know" category is insignificantly different from both zero and from those in the "stayed the same" category for all of the variables. This is consistent with the hypothesis that answering "don't know" is essentially random with respect to the changes the firms experienced themselves. Finally, we can accept the joint hypothesis that the mean values of sales in each of the qualitative response categories are different from each other (both excluding and including the "don't know" category).

• Figures relate to change
• Figures in parenthesis: Linearised standard error of mean in columns 1-4, p-value of Wald test in columns 5 & 6
• Asterisks relate to results of F-test of difference of means of group with "stayed the same" groups, *** significant at 1% level, ** significant at 5% level, * significant at 10% level
• Weighted and stratified

Turning to the objective measure of change taken from the BAI and IR10 data, we can add two more comparisons relating to market share (using the 3- or 4-digit ANZSIC code as the classification of "market"). We can also use responses to the subjective measure of change from both BOS 2005 and BOS 2006.⁴⁶ The results for the first three variables are similar, but stronger. We can once more accept the hypothesis that the mean of the change in sales for firms who state that their sales have increased is indeed higher than those who stated that they stayed the same (at the 1% level). We can now also accept the hypothesis that the change in sales for those who said they decreased is lower (significant at the 5% level). Furthermore, we can now distinguish the change in productivity and market share of two of the groups. The change in productivity for those who said it has increased is higher than that for those who said it stayed the same (significant at the 1% level). We can also say that the change in market share, defined according to 4-digit industry, for firms that stated that their market share had fallen was indeed lower (statistically significant at the 5% level).

• Figures relate to absolute change
• Figures in parenthesis – Linearised standard error of mean in columns 1-4, p-value of Wald test in columns 5 & 6.
• Asterisks relate to results of F-test of difference of means of group with "stayed the same" groups, *** significant at 1% level, ** significant at 5% level, * significant at 10% level
• Weighted and stratified