Service Performance
1. International Comparisons
A benchmarking study of the TRS with other countries, such as Australia, USA, Sweden and the United Kingdom, compared charges to the user and the suitability of service quality measures.
This study, conducted by Consultel Associates Ltd for the MED, found that New Zealand compares favourably to these comparison countries in terms of the charges to users because all local and national relay calls in New Zealand are free of charge. In Australia, the UK and Sweden, users are charged for the relay call segment between the relay centre and the called party.
Another element of cost to users is the textphone equipment. In New Zealand a range of textphone equipment is heavily subsidised by the Crown. The rental is $15 per month (including GST) and is charge-free for people who meet the income thresholds to be eligible for a community service card.
Australia offers a more limited range of textphones through the Telstra and Optus Disability Equipment Programme. In the UK and Sweden, users must purchase their own equipment. In the USA, most states subsidise wire-line but not wireless textphones through an Equipment Distribution Programme.
At present, under the TRS TSO Deed and Addendum One to the Deed, New Zealand uses equivalent Service Quality Measures (SQMs) to those in Australia, UK, USA and Sweden with one exception: SQM 5 "Dialling Response for Outward Calls". This is discussed further in the section below.
2. Monitoring Service Performance
The TRS provider Sprint must comply with the Service Quality Measures specified in the TRS TSO Deed referred to earlier in this report. Sprint reports a number of additional quality assurance measures that it takes to monitor and enhance the services offered to users.
As at 15 December 2005 the Relay Centre employed 18 Relay Assistants. Comprehensive Relay Assistant training programmes and performance assessment programmes aim to continually improve Relay Assistant call handling performance. Programmes are based on performance standards developed from customer satisfaction surveys, customer input and contractual obligations.
Other quality assurance efforts include: certified trainers explain policies, practices and call procedures; and management and resource staff located in the work centre with Relay Assistants provide guidance, motivation, training and performance reviews.
The New Zealand Relay Advisory Group (NZRAG), established in March 2005, also provides feedback and raises issues affecting the performance of the TRS, and makes suggestions for enhancing services. Members of NZRAG represent various user groups. A Charter for NZRAG has been developed and agreed by members that embodies the experience of successful equivalent groups in the USA and specific New Zealand requirements.
Service Quality Measurement
As already mentioned, a number of Service Quality Measures (SQMs) are applied under the TRS TSO Deed for monitoring the TRS performance.
The service quality measure "Dialling Response for Outward Calls" (SQM 5) is computed on a quarterly basis for each financial year of the TRS Provider. The required Performance Standard is that at least 85% of outgoing calls from the TRS system will be dialled by Relay Assistants within 15 seconds of receiving call destination information from the call originator (quarterly figures). This service measure requires sampling for it to be applied in practice.
Service Quality Measures by their nature must be verifiable. Statistical sampling methods are open to interpretation and dispute as to whether an SQM has been met or not.
The Commerce Commission monitors Sprint's compliance with the SQMs under the Relay Service TSO Deed on behalf of the Crown. For SQM 5 the Commission requires a statistically rigorous process that exceeds Sprint's normal observational tests (which Sprint carries out as an internal quality assurance measure under its 31 contracts in the USA and in New Zealand). Sprint's current testing regime is that each Relay Assistant is assessed twice monthly by an observer that is physically present. The Commission does not consider that Sprint's testing is blind or random.
The Commerce Commission requires a blind, random test that needs at least 196 random test calls generated to the Relay Centre per quarter. The results must then be statistically weighted according to the traffic that the Relay Centre normally experiences on that day of the week and time, seasonally adjusted. The cost of these test calls would be borne by the TSO Liable Persons. In addition to the cost of the test calls the statistical weighting requires considerable manual work by Sprint, auditing by KPMG and finally examination by officers of the Commerce Commission.
In hindsight and with the benefit of one year of operational experience it is concluded that the measure serves no useful purpose in terms of the TRS user experience. The stakeholder review and the user survey carried out by the NRB did not identify any significant user concerns with the dialling response for outward calls. It is not a service quality measure that is used by Australia, the UK, the USA or Sweden. It has also become clear that unless a measure can be captured electronically the verification costs are unreasonably high. A further Addendum to the TRS TSO Deed would need to be negotiated to discontinue SQM 5.
Recommendation
The service quality measure "Dialing Response for Outward Calls Service" (Quality Measure 5) be discontinued and no longer be applicable from 2005/2006 forward.
3. Customer Feedback
The NRB User Survey and submissions from other stakeholders were overwhelmingly supportive of the TRS and of the difference it is making to the lives of those who use it. Survey participants reported an overall 85% satisfaction rate with services offered under the TRS. Being very satisfied is a function of user frequency since this figure is over 70% among frequent TRS users (refer Figure 1. below from the NRB user survey). Additionally, 72% of participants rated the TRS a good way to communicate with other people for most things.
Figure 1: Very Satisfied
When asked to comment on problems that users may have with the TRS and ways to improve the service, some service performance issues were identified.
Relay Assistants
The TRS was rated by users for service performance on seven aspects of the interaction between the Relay Assistant and the caller. High ratings were generally achieved except for how long they take to answer a call. Some 12% of users rated how long Relay Assistants take to answer calls as "often not good". Feedback from submissions also commented that there were occasional long delays in connecting to a Relay Assistant and that this was quite frustrating if an urgent call had to be made.
Speed of answering calls is one of the Service Quality Measures specified in the TRS TSO Deed. The required performance standard is that at least 85% of calls will be answered within 15 seconds. The Commission's draft determination for the 2004/2005 year has concluded that Sprint have achieved material compliance with this measure. At the time of writing this report, Sprint was actively taking steps to bring performance consistently above 85% by adding more staff and making technical changes.
Many positive comments about the Relay Assistants were received. Feedback from submissions indicated that they were generally helpful and competent in the art of relaying conversations, and were found to be personable and considerate. Some 85% of survey participants similarly rated the manner of Relay Assistants as always or mostly good. However, some submissions indicated that there were varying levels of consistency among Relay Assistants in respect to typing ability and allowing users to manage their own conversations.
Recommendation
The speed of answering relay calls continues to be improved so there is minimal variability in performance between months.
Recommendation
The consistency of performance of Relay Assistants in some areas be improved.
Support and Training for TRS Users
The adoption of the TRS is being held back in that 4 out of 10 users1 who attempt to use the service are finding aspects of it difficult to understand and use. Individual user comments included:
- Have problems understanding how to use the TTY
- Instructions are not user friendly
- Not sure how to apply for a TTY and whether it is free or not
- Have language and spelling difficulties when typing (for many NZSL users English is their second language)
- Key pad difficult to use
- Need more time and practice to be comfortable with TTY and TRS
Submissions commented that much of this support including teaching users how to access and use TTY machines is currently being undertaken by relevant NGOs at their own cost. One organisation would like to see Sprint employ someone with knowledge of communication difficulties, to either phone or visit newly registered users. There are good indicators to conclude that personal one to one assistance is required for some users.
It is noted that Sprint is already taking steps to provide additional one to one assistance to new and existing users.
Recommendation
Additional one to one assistance is provided to textphone users to increase their confidence in making relay calls.
Back to Top