The Novopay inquiry

June 4th, 2013 at 1:46 pm by David Farrar

Steve Joyce has released the report of the Ministerial Inquiry into Novopay. Major findings:

  • The problems with Novopay have affected public trust and confidence in the Ministry of Education, and also the wider public sector
  • Weaknesses in project governance and project leadership allowed Novopay to go live with a number of significant risks which the Ministry of Education and its vendors, including Talent2, were over-confident of managing
  • These risks resulted in service issues and the Ministry and Talent2 were unprepared and overwhelmed by their nature and scale
  • The School payroll is overly complex due to an accumulation of historical changes
  • There was extensive customisation of the Novopay software
  • There was a failure to involve the users of the Novopay system in the schools and appreciate their requirements
  • There was no overall accountabilityfor Independent Quality Assurance
  • The project has cost $23.9 million more than estimated for a total cost to date of $56.8 million
  • Ministers were not well served by the information they were given on the project. Reporting to Ministers was inconsistent, unduly optimistic and sometimes misrepresented the situation.

Joyce comments:

“This report makes for sober reading and, while it confirms the view that there is a lot of blame to go around for the problems with Novopay, it provides a greater understanding of the level of fault between the organisations involved,” Mr Joyce says.

“There are substantial lessons to be learned by the Ministry of Education in a number of areas which the Acting Secretary of Education is taking steps to address.

“There are also lessons to be learned by the public service and the wider State Sector on the design, delivery and oversight of major ICT projects.

“As the report notes, these problems are not unique with issues identified in the Ministerial Inquiry into the police computer system INCIS 13 years ago also evident here.

“The Government will be carefully considering the findings of the Ministerial Inquiry into Novopay. It intends to act on all the recommendations. It is critical these problems are not repeated again.

In the report itself:

Work commenced on the requirements for the schools payroll project in October 2008. This  process was lengthy, and was never actually completed. Even after Go Live, new requirements  were being discovered. There was little direct customer (boards of trustees) or user (principals  and school administrators) involvement in the definition of the requirements, and Datacom’s involvement was minimal.

And that’s where it would all have started to go wrong. You must finalise the requirement and have customer input into them.

Also:

A 5 June 2012 paper, which invited Ministers English, Parata and Foss to approve the continuation of the project following Confidence Point Two, misrepresented its state.

In more detail:

We were particularly concerned about the 5 June 2012 document, which invited Ministers to approve the Ministry’s Go Live decision. This report misrepresented the situation in two significant ways. It suggested that the Go Live decision was supported by three members of the ICT Council, which was not the case; and it stated that Confidence Point Two criteria had been met, when in fact a number either had not been met or were “deemed” to have been met (a lower criterion). These factors overrepresented project readiness and its chances of success.

The report is very interesting reading. There are even questions over the original selection decision.

Tags:

Novopay – is the worst over?

May 8th, 2013 at 6:31 am by David Farrar

Stuff reports:

Tom Parsons, president of the Secondary Principals’ Association, said Novopay had a serious image problem that would continue to dog it, but the improvement had been remarkable.

“If you’d been repeatedly underpaid by it … whenever the word Novopay comes up you’ll roll your eyes.

“Throughout the sector that’s happening already, which is a shame because the system of today bears no resemblance to what we had pre-Christmas.”

Waikanae School principal Bevan Campbell said Novopay was still not user-friendly, “[but] I think they have been working really hard to fix it up. Now that the powers that be have realised the massive stuff-up, they are actually trying to fix it.”

The key thing, in my mind, is to get it to a stage where the level of manual data entry from the helpdesk is minimal. If schools can enter and validate all the data themselves, that will make a huge difference.

Tags:

Novopay to stay – for now

May 7th, 2013 at 1:35 pm by David Farrar

Steven Joyce has announced:

Minister Responsible for Novopay Steven Joyce today announced that Talent2 will continue to administer the school payroll – but that the long-term future of the system will stay under review.

“The decision to stay with Novopay at this point was made very carefully after a great deal of consideration and weighing up of all the risks,” Mr Joyce says.

“The improvements we have seen in delivering school pay from pay period to pay period and the progress to date in clearing bugs, means it wouldn’t be sensible to make a change at this point.  Making a change now would increase the work for payroll administrators in the short-term during the cut-over from where we stand today, not decrease it.

“Three out of the last four pay periods have had a reported error rate of less than 0.5 per cent – which we have been advised by our independent technical reviewers is a reasonable error rate for a stable system. However, that is not to say that we are out of the woods yet or endorsing the current state of the pay system.

The error rate for the last pay round was 0.26%, which is well below what the previous system even had.

“The only other realistic option at this point was to switch back to the old Datacom system and then upgrade again later to a new Datacom system.  That would mean substantial additional work and two further changes for school administrators.  We would do that if required, but it is not a step to be taken lightly.

“You can’t just switch a complex $4.4 billion a year payroll that pays around 90,000 people every fortnight without creating more issues – no matter which system you are using.

“A go or no-go decision had to be made at this point because the current Datacom backup proposal is time-sensitive, and has now reached the point where it will have to be re-worked for them to be able to assist.

Datacom have made it clear they remain ready to assist if they are required at any further stage. I appreciate that, and may call on them again at any time. I thank them for all their work and assistance to date.”

Mr Joyce says the focus for the next two months will continue to be on the Novopay system remediation in which progress will continue to be monitored on a daily, weekly and monthly basis.

“Staff from the Ministry, Talent 2, and schools are working hard on the remediation.  I appreciate all their efforts as we work towards a business-as-usual state,” Mr Joyce says.

The next key milestones are the Novopay Ministerial Inquiry, which is due to be presented to the Government at the end of this month, and the review of progress in the Remediation Plan at the end of June.

The report of the inquiry was be very interesting I suspect.

Tags: ,

The future of Novopay

April 27th, 2013 at 12:00 pm by David Farrar

Stuff reports:

D-Day looms for the trouble-plagued Novopay school payroll system.

A decision on whether to ditch Australian providers Talent2 and possibly give the contract back to Datacom will be announced soon.

A spokesman for Steven Joyce, the minister charged with sorting out the pay debacle, insisted yesterday a decision had not been made, but the Government has previously signalled a decision by the end of the month.

Talent2 chief executive John Rawlinson could not be contacted for comment.

The company has been forced to hire extra staff and work round the clock to try to fix the bugs, which have numbered in the hundreds and left some school staff without pay for months.

The latest pay round showed there was still a long way to go.

There were 39 staff not paid, 210 underpaid, 142 overpaid, 264 schools affected and 123 bugs resolved.

Far from a long way to go, that is an error rate of around 0.4% – less than under the previous system. That context would be useful.

The system is still damn buggy and I don’t think it will be deemed acceptable until school staff are able to enter in all staff details themselves, and confirm the details are correct. Once they have achieved that, they won’t need the expensive call centres doing so much manual work.

Tags:

Deja vu

March 26th, 2013 at 10:00 am by David Farrar

The ODT in February 2008:

Principals around the country are criticising the Ministry of Education after major payroll errors, which have created a bureaucratic nightmare for many schools.

Datacom is contracted by the Ministry of Education to provide payroll services to more than 80,000 New Zealand teachers. The latest payroll round – the first of the school term – was filled with errors, New Zealand Principals Federation president and Balclutha Primary School principal Paddy Ford said.

‘‘A number of schools have not received any pay, and many support staff have been paid the incorrect amounts.

Staff who are on leave have incorrectly received full pay, changes that were sent to Datacom in November have not been actioned, and staff are being forced to fill forms out twice with exactly the same information.” …

Mr Ford said: ‘‘Schools deserve better. It’s a nightmare. We are dealing with unnecessary bureaucracy and huge numbers of errors.

Payroll systems have changed for the worse and we need to have immediate action from the ministry, directing their payroll service providers to shape up or ship out.”

Ministry of Education Schooling Group resourcing senior manager Kevin Wilson said 99% of school employees had been paid correctly at the start of the 2008 school year.

if you changed the dates and the names you would swear that is a story from February 2013, not February 2008!

Tags: ,

The Novopay Technical Review

March 24th, 2013 at 11:00 am by David Farrar

Ben Gracewood does a dummies guide to the Novopay Technical Review:

1. In some areas system functionality does not adequately support the business processes.

The system, as built, does not do what it was meant to do. This could be because the requirements were poorly documented from the outset (probably), but also because the system is an “off the shelf” system that has been “customised” to meet the requirements.

At some point, a decision would have been made that the “off the shelf” software was an 80% (or 70% or 90%) fit for requirements, and the rest could be built as customisations. Hold that thought and read on.

The moment I read that, I understood where all the subsequent problems probably came from.

A custom built system would, in my opinion, be far more flexible and what is needed. It will be very interesting to see in the full review on what basis the decision was made to go with an off the shelf solution.

2. Usability issues and lack of data input validations contribute to processing errors.

To me, this is the most egregious finding. As I understand it, the entire purpose (or at least a key selling point) of Novopay was to reduce costs by moving from Datacom’s human-intensive workflow to a system whereby schools do all the data entry, and the “system” just needs to calculate pay.

For a system like this to work, a massive amount of time and focus has to go into usability of the front-end system. Later on in the review there are comments about tab keys not working properly, and links between pages of the same input form not working. This is fundamental stuff, and points to Talent2 and ALESCO not giving a flying shit about end-users.

This, while fairly typical of an “enterprise” system build on Oracle Forms, is completely unacceptable if it’s so utterly important that data is entered correctly and quickly. It is an absolute core failure of a system intended to be user-driven.

Yep. It all started to go wrong when schools would not enter in casual staff details and had to fax them off for manual processing.

6. A high degree of customisation in high-impact areas has made on-going development more difficult.

This is another one that really sets my alarm bells ringing. Talent 2 sold Novopay to the government as an ALESCO system with some customisation (see point 1). This is the dirty not-so-secret of any “out of box” enterprise software installation: the base system is next to useless without massive amount of customisation.

More often than not, when you add up the cost of an “out of box” solution PLUS the required “customisation”, a greenfield development fine-tuned to your own requirements becomes a much more comparable solution.

I agree.

Like I said on Twitter: Novopay appears to be the wrong solution sold by people with a poor understanding of the problem, implemented with the typically lax approach of enterprise software development.

In other words, pretty much business as usual for Enterprise IT.

If you think this thing has run its course, you’re kidding yourself. I’ve seen Oracle systems quoted as costing $6m run up $30m of cost before being completely scrapped – as in yes, uninstalled and reverted to the system they were meant to be replacing, completely wiped away.

It will be interesting to see if they deem Novopay fixable.

Tags: ,

Novopay assistance

March 19th, 2013 at 1:09 pm by David Farrar

Steven Joyce has announced assistance for schools:

The $6 million package will be allocated as a one-off payment across the sector, calculated on a formula of $105 per Full-time Teaching Equivalents (FTTEs) plus $500 per school. For example:

  • A small school of 5 FTTEs will receive $1,025
  • A medium school of 20 FTTEs will receive $2,600
  • A large school of 120 FTTEs will receive $13,100.

If there is eventually a court battle between the Government and Talent 2, then this will be added to the disputed bill!

Joyce also updated on the system:

“In the last three fortnightly pay periods, the percentage of complaints and notifications received dropped from 2.2 per cent to 1.9 per cent to 1 per cent while at the same time the total number of people being paid increased from 74,373 to 84,822.

“Following Pay Period 26 this week, the next big challenge will be Pay Period 1, in which significant changes occur for the start of the new financial year, including adjustments to KiwiSaver and student loan repayment rates. These changes may introduce new issues. The Ministry is working closely with Talent2 to minimise any increases in error rate.

So it has got better, but may get worse with the new financial year.

Out of interest does anyone know what the error rate with Datacom was?

“In regards to bug fixes, 87 defects have been resolved as part of the remediation plan to date, with 143 scheduled for resolution in a release on 23 March and a further 46 currently scheduled for 20 April. 228 defects are being managed through business as usual processes and we are looking at a fourth release to deal with remaining defects and usability issues.

The total number of current defects is 526. These include 49 Category Two defects (very serious), 320 Category Three defects (serious), 115 Category Four defects (moderate), and 42 Category Five defects (cosmetic). It is planned that the number of Category Two defects will be reduced to 12 on completion of the second release this coming weekend.

I guess a category one defect is the system is offline!

Tags: ,

Attacking public servants

February 6th, 2013 at 7:00 am by David Farrar

Chris Hipkins attacked the appointment of Sir Maarten Wevers to the Novopay Inquiry as not being independent as he is a former head of the Department of PM and Cabinet.

Chris, of all people, knows that DPMC is scrupulously neutral and serves all Governments with total professionalism. They are totally different to the PM’s Office which is political.

Inventory at Keeping Stock blogs:

We thought more of Mr Hipkins than this small-minded affront to a respected public servant’s integrity. Sir Maarten  Weevers was appointed as head of the department of Prime Minister and Cabinet in 2004 when Helen Clark was Prime Minister. Prior to that he had been a career diplomat who had served New Zealand with distinction both domestically and overseas, and during periods in which both National and Labour governed the country. He was also Private Secretary to PM David Lange at one point in his distinguished public service career. Sir Maarten retired from his DPMC role last year.

However Chris Hipkins slights all public servants  with this indirect attack on Sir Maarten’s integrity. It is a disgraceful slur by Hipkins, which we roundly condemn. Almost all public servants manage to achieve political neutrality and separate their personal and political beliefs from their work. Sir Maarten Weevers has proved that by serving New Zealand’s two most recent Prime Ministers in an apolitical manner.

I can’t think of anyone more independent than a former DPMC head. Probably a former Cabinet Secretary only (who reports to the DPMC head).

The irony here is that Chris Hipkins has history with Novopay that he is anxious to re-write. We blogged back in November that the first Novopay contract was signed when Chris Carter was still Minister of Education. And when it was put to Hipkins on the telly last week (we can’t find the video, unfortunately) that he had been involved when he was a ministerial advisor prior to entering Parliament in 2008, he was very quick to change the subject.

Heh I’m sure he was.

Tags: , , ,

Novopay – what went wrong

February 5th, 2013 at 9:00 am by David Farrar

Danyl McL has a useful summary of what went wrong with Novopay. His summary is:

  • Novopay is designed so that schools do everything online (presumably through a browser or thin client). When the platform was launched there were ‘significant issues’ with the online user interface (UI); for example, it wasn’t possible to submit time-sheets for part-time teachers.
  • The work-around for problems submitting payments via the UI was for the schools to fill in a form and submit it to Talent2′s Novapay service desk via e-mail, and they’d manually enter the payment data into the system.
  • But the service desk wasn’t staffed or trained for this – they were supposed to be supporting anonline platform in which the schools did almost everything themselves. So this created a huge backlog of manual payments for them to enter, many of which missed the payrolls.
  • The service center also generated a vast number of errors in payments because it doesn’t have ‘robust quality assurance’ (I take this to mean there’s no verification when service center staff manually submit data: so if someone is being paid $20/hour and they work for ten hours, the manual system won’t prevent a service center staffer from accidentally paying them $2.00)

The failure of the user interface seems key. If that was working properly then schools would be entering all the data directly and more importantly being able to see a draft payroll batch and confirm it is correct.

The moment you can’t do that, and you have manual entries, help desks and the like and it turns into a logistical nightmare considering there are 100,000 pays per fortnight.

The inquiry will no doubt look into why the UI was not working before launch. Stuff reports:

Sir Maarten Wevers, the former boss of the Department of the Prime Minister is to head a Ministerial Inquiry into the botched handling of Novopay education payroll system.

Steven Joyce, who now has ministerial responsibility for Novopay announced that Sir Martin would head the $500,000 inquiry along with Murray Jack, chairman of Deloitte in New Zealand.

Can’t do much better than those two. I doubt they will sugar-coat anything.

Tags:

The Novopay papers

February 2nd, 2013 at 8:41 am by David Farrar

Stuff reports:

Novopay provider Talent2 were unwilling to staff a call centre for stricken teachers in the week before Christmas, documents show.

Then-Associate Education Minister Craig Foss had to call chief executive John Rawlinson to intervene.

Education Ministry acting chief executive Rowena Phair wrote to Talent 2 board chair Andrew Banks last month to say she was “appalled” and it was “unacceptable.”

“The impact of this decision would have been that a large number [of] schools’ staff would not receive their holiday pay prior to Christmas.”

It is obvious there were great tensions between the Ministry and Novopay for a considerable period of time. The client and the supplier were blaming each other. The ministerial inquiry announced by Joyce will no doubt shed some light on blame – once the current situation is resolved.

Finance Minister Bill English, Education Minister Hekia Parata and Associate Education Minister Craig Foss signed off the project in June last year despite advice there were 147 “software defects”.

The number of defects is not actually significant, especially in a $100m contract. What is more important is the nature of those defects.

In June last year, a report to English, Parata and Foss, outlined 147 bugs in the system. There were no problems at the most serious level, but 10 at the next level and 105 at “level 3″.

So no show-stoppers had been identified?

So why did the Government proceed?

Four independent advisers – from Pricewaterhouse Coopers, the Social Development Ministry, the Primary Industries Ministry and the New Zealand Transport Agency – gave the system the green light.

“Talent2 now has a proven way of rectifying defects and releasing the fixes,” the Education Ministry report said.

The ministers allowed the project to go ahead in August.

I’ll be very interested to see the PWC advice especially. There could be some culpability around that advice, if it was flawed.

With the benefit of hindsight, a regional pilot should have been insisted upon before rolling out nationwide. The inquiry will have to look into that also of course.

And one question for those with memories. Who was the ministerial advisor of the Education Minister who developed the original contract, and what role does he have today?

Tags:

Noverpaid

December 13th, 2012 at 7:47 am by David Farrar

Jody O’Callaghan at Stuff reports:

Schools are reporting large and unexplained overpayments in holiday pay in Novopay’s end-of-year cycle.

Well I guess that is preferable to being underpaid!

Tags:

A smart move

December 1st, 2012 at 10:00 am by David Farrar

Richard Meadows at Stuff reports:

New Zealand banks are wading into the Novopay disaster to offer interest-free overdrafts to teachers and school staff running short on cash.

Problems with the Education Ministry’s new payroll system have led to thousands of teachers paid incorrectly or not at all.

Some have reported being unable to meet mortgage payments or falling behind on hire purchase or credit card debt as a result of the cashflow squeeze.

The banks participating in the interest-free overdraft offer are ANZ New Zealand, ASB Bank, Bank of New Zealand, The Co-operative Bank, Kiwibank, SBS Bank, TSB Bank, and Westpac New Zealand.

They are offering six weeks of interest-free respite –until January 15 – up to the equivalent of any missed payroll payments.

Affected teachers and school staff have to provide evidence to their bank, for example a letter from their school, detailing how much they are owed.

Their bank will then arrange an overdraft facility on the account to which their salary is paid, or to another account by arrangement.

New Zealand Bankers’ Association chief executive Kirk Hope said the banks had offered their support to help tide people over the holiday period.

“I encourage affected school staff to get in touch with their bank so they know they’re covered and can have some peace of mind over the holidays,” he said.

A very smart PR move from the banks.

As I understand it any teacher underpaid can also get reimbursed within 24 hours by their school directly – and the school then gets reimbursement from the Ministry of Education.

However relieving teachers who have no main school, are less able to do this easily, so the move by the banks should be of assistance to them especially.

Tags: , ,