The great PBRF scam

March 26th, 2012 at 9:00 am by David Farrar

I’ve had forwarded to me a number of documents and reports from within over its attempts to what can only be called a rort of the , or Performance Based Research Fund. The TEC says that:

The primary purpose of the PBRF is to ensure that excellent research in the tertiary education sector is encouraged and rewarded. This entails assessing the research performance of TEOs and then funding them on the basis of their performance.

As I will detail below though, in fact it seems the universities gain funding on the basis of their ability to manipulate employment contracts.

The annual value of funding from the PBRF is around $500 million, so universities are highly incentivised to maximise their share of it. Sadly it seems that rather than encouraging academics to do more and better research, VUW (and probably others) have unleashed the HR Department to try and manipulate their results. The ironic thing is that if all universities do what VUW does, then no university actually gains a comparative advantage.

A complaint was made to the VUW University Council by an Associate Professor in the Accounting Economics & Finance Department, who said what VUW was doing was effectively accounting fraud.  He gave eight examples:

  1. Individual A, who is not active in research and who has a fixed-term contract maturing after June 2012, was asked to terminate this contract in favour of another one terminating in early 2012 coupled with a further contract starting in late 2012.  The effect of this would be to exclude A from the calculation of the University’s PBRF score in June 2012.  Individual A rejected this proposal on ethical grounds.      
  1. Individual B, who is not active in research and had a series of fixed-term contracts with the last maturing in 2011, was offered at that time a replacement contract with a significantly shorter duration than before and that would mature in early 2012, with the effect that they would not be included in the calculation of the University’s PBRF score in June 2012.  B accepted this contract.    
  1. Individual C, who is not active in research and has a conventional academic contract, was asked to temporarily depart from University employment in 2012.  The effect of this would be to exclude C from the calculation of the University’s PBRF score in June 2012.  C accepted this offer.  Within a week of Mr Corkill’s inquiry being set up, these arrangements with C were cancelled.
  1. Individual D, who is not active in research and had a conventional academic contract, was asked to retire early and was offered a fixed-term contract for teaching work in 2012 with the contract period excluding June 2012.  The effect of this would be to exclude D from the calculation of the University’s PBRF score in June 2012.  D accepted this contract.
  1. Individual E, who is not active in research and had a conventional academic contract, was asked to retire early and was offered fixed-term teaching contracts for early 2012 and a further period in late 2012, with the effect of excluding them from the calculation of the University’s PBRF score in June 2012.  E accepted this contract. 
  1. Individual F, who is not active in research and had a conventional academic contract, was asked to retire early and was offered a fixed-term teaching contract for late 2012, with the effect of excluding F from the calculation of the University’s PBRF score in June 2012.  F accepted this contract.
  1. Individual G was offered teaching contracts in 2012 with the terms of the contracts such that G would be excluded from the calculation of the University’s PBRF score in June 2012.  In addition, G was advised by a University manager that this PBRF consideration was the rationale for the terms offered and that the terms were chosen sufficiently well away from the June 2012 date to avoid suspicion.
  1. A Head of School circulated an email to staff in their School advising them that academic staff members of a particular type counted for PBRF and instructed staff to therefore ensure that none of this type were employed on the June 2012 census date. 

There is also other stuff happening to massage the numbers, such as putting some staff onto teaching only contracts, so they get excluded. That is arguably more legitimate, but I struggle to see how having employment contracts that exclude the month of June, purely to avoid the PBRF census can be a correct or ethical thing to do. It also undermines the whole point of the PBRF.

The VUW Chancellor appointed a QC to investigate. His full report they refuse to release, but a summary was released. It is worth noting that his “advisor” was the Chair of the University’s Audit & Risk Committee. I regard that as a conflict. Members of the University Council naturally wish to maximise the funding for VUW. They would be concerned about behaviour that costs VUW money, but be motivated to take a tolerant view of unethical behaviour which gains VUW money.

The report summary is here - Corkill Executive Summary. A response to it from the Accounting Associate Professor is here - PBRF Allegations and Investigations-to dist list. Note that Martin Lally is not the person who supplied these to me. I have had no interactions with him at all. A response from the Chancellor is here, which basically tells him to go away and even declares he will not entertain further correspondence on the issue - Chancellor Letter 28 Feb.

If you bother to read the response from Lally, he points out that the summary is so lacking in details, that he can  not know whether or not his allegations are correct. VUW refuses to allow him the full report.

Further to the original allegations, there is then the e-mail below:

Now to be fair, this is not just happening at VUW. I understand there are complaints about this gaming happening at other universities also. Apart from the low ethical standard the universities are providing to students (Lally’s  concern is the university is doing something that he lectures students on, as being accounting fraud), it would be simpler if all the universities just agreed they won’t game their numbers, if the others won’t.  The amount of time and energy put into this gaming is significant. Every Department in every Faculty has to take part.

There are a lot of unhappy academic staff about this. Since I gained the original materials, I have spoken to several acquaintances who are academics and they all confirm that this is wide-spread.

I understand TEC are aware of all this. I can’t understand why they haven’t said “No you can’t sack a staff member in May and rehire them in July, just to make your PBRF ranking look better”.

This poses an issue for the Minister also. $500 million of taxpayer funds go towards universities and others through the PBRF every year. The PBRF was set up by (I think) the last Labour Government. The intention of it is good – that universities which are producing more research and better research gain more funding. I spent a couple of terms on the Otago University Council and recall that even at Otago, there were some academics who produced minimal original research. PBRF should be encouraging more original research.

But the gaming of the system must seriously bring into question whether the $500 million is going to the right institutions in the right proportions. In theory getting rid of staff for a month who are not doing much research doesn’t actually gain an institution more money directly (as I understand it), but it increases their relative ranking, which benefits them for decisions on other issues such as where new schools might be located, or attracting staff.

At a minimum, in my opinion,  the PBRF rules need to be reviewed to discourage such gaming of the system. Ideally there would be a proper external inquiry into all the universities, to discover how widespread this gaming is, and whether it it ethical and legal.

Tags: , ,

22 Responses to “The great PBRF scam”

  1. Michael Mckee (1,091 comments) says:

    I just love the way you use the phrase “gaming” when it should mean theft.
    If their intention is to get resources they don’t qualify for or resources they actually aren’t qualifying for but it looks that way on paper to get the money then it is likely fraud or more rightly Theft.

    The Chancellor and et al should resign or be charged.
    Maybe the Minister should suspend all payments until an enquiry has cleared up the issue?
    Maybe we need people like Lally oversighting the whole shebang??

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  2. thomasbeagle (78 comments) says:

    And this is just another example of how effective it is to ask people to meet certain metrics.

    Now, about that “performance based pay for teachers” where teachers will be paid based on how well they game the system rather than how well they teach…

    [DPF: I don't think performance pay should be based on metrics. I think it should be based on the opinion of the principal as to how good a teacher is]

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  3. david (2,531 comments) says:

    The fatal flaw in the concept is that no-one could claim to have no potential conflicts as an overseer of the system apart from overseas trained fraud/ethics experts who have no connection (past or present) with a NZ University and I think that these would be pretty thin on the ground.

    Anyone else remotely qualifies is either an employee, an ex employee or an alumnus of one of the plethora of institutions claiming University status and is therefore immediately potentially conflicted.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  4. flipper (3,533 comments) says:

    A case for the SFO ???????????????????

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  5. Rob F (6 comments) says:

    Every institute will be stacking things in their favour, but I’d like to believe most aren’t illegally screwing the scrum.

    I think this is the last round of PBRF and then it is getting dropped because of its many flaws, similarly to what happened in Britain. Universities are not only a place for research but are a place for learning, yet PBRF does not reward teaching. How you measure teaching and compare that across the spectrum of departments and institutes is another matter.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  6. PaulL (5,872 comments) says:

    All reward systems have flaws. Quantitative or objective systems are subject to gaming – the classic “you get what you measure” trap. Qualitative or subjective systems are subject to bias by the individual doing the rating.

    To my mind the secret is twofold. Firstly, to design the system well enough that it’s easier to just do a good job than it is to game it. This usually means a blend of measures and enough subjective/personal input to catch those who are blatantly gaming. The second is to change the system often enough to avoid systematic gaming to build up.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  7. Viking2 (11,125 comments) says:

    So we are suddenly surprised that some socialist institution rips the taxpayer.
    Polly techs, schools uni’s and all manner of outfits have been doing this for years. Stop giving out taxpayer money and give us our own tax to spend our selves.

    You never learn the basics.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  8. Brian Harmer (686 comments) says:

    My own belief, frequently and loudly expressed prior to my retirement, is that most university staff spend far more time entering, tweaking, and adjusting their data into one of the most cumbersome systems in history, and attending numerous seminars and coaching sessions on how to present their PBRF portfolio in the best light, and having practice runs, than any benefit to be gained is worth.

    And then there is the time spent by the hundreds of very senior academics who are supposedly at the top of their profession, reading and scrutinising and sitting in judgement on each and every one of the tens of thousands of portfolios submitted each round. This elephant costs way more to feed than it will ever deliver in the way of benefits.

    Of course, this is about more than the institution’s share of the $500m … the marketing value of perceived rankings come into play, though, in my opinion, the number of top class articles published by academics has a very tenuous connection to the effectiveness of the institutions.

    On the other hand, it is important to remember that universities are not just teaching institutions. It is in their nature, and they are legally required by their empowering acts to do research, and to create new knowledge. PBRF is about measuring research output volume and quality. It just happens to be the most over-engineered, badly implemented, and ineffective mechanism for so doing, that I can imagine.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  9. simonway (371 comments) says:

    This sort of thing has been going on since PBRF was introduced. In the first couple of years Otago boosted their rankings to become #1 by culling all the low-performing staff, and hence reducing their research output. When I was at Auckland, I heard PBRF described as “an accountant’s idea of how to fund research (no offense to accountants).”

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  10. Rick Rowling (800 comments) says:

    It made sense to base the calculation on a particular “census” date, rather than an average for the entire previous period because….?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  11. djf001 (3 comments) says:

    First, it takes a degree of courage to raise such issues in a workplace, and especially within a university. Associate Professor Matin Lally’s courage ought to be noted. For raising the matter; for pursing the matter.

    First, the letter from Chancellor Ian McKinnon isn’t good. The last line in the letter doesn’t assist the resolution of matters, build relationships into the future. It is amazing the degree to which sitting down in a room and going through matters can assist their resolution, providing information relaxed and openly.

    I note that a request has been made under the Official Information Act for the full investigative report. This is just second-hand, “I’ve just heard”, Victoria University’s administration have a contemptuous attitude towards to requests for information – it takes Ombudsman’s intervention to get information from the University. It takes courage for an employee to make an Official Information Act request to their workplace (workplace that is publicly funded and subjected to the Act) when ‘things are going on’ and communication is not happening. In one public sector workplace I have heard that an employee being hurdled into a disciplinary meeting for doing such. What’s better? Make up stories, engage in conspiratorial gossip around the water cooler or to seek the information? I don’t know.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  12. RJL (143 comments) says:

    @Rick Rowling

    It needs to be based on a census date, because staff move from university to university (or out of employment, in academic NZ).

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  13. Sam (497 comments) says:

    I’ve heard that whole universtity programmes, and in some cases departments, have been culled primarily in order to improve PBRF rankings too…

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  14. cosmopolite (8 comments) says:

    “Officer, I am shocked, shocked! Round up the usual suspects!” (Claude Rains in Casablanca)

    If the 8 cases described above are all mortal sins, then every middle manager under the sun is damned. A more serious problem is that some New Zealand disciplines have little or no academic credibility. These disciplines are allowed to count as “research outputs” activities that would not count in more credible disciplines. Tourism & Leisure Studies (a fashionable and new part of Commerce) is high on my list of disciplines lacking credibility.

    All bureaucracies around the world are guilty of “gaming” any and all numerical systems put in place to measure “performance.” This will continue to the end of time. PBRF in New Zealand is a minor instance of this phenomena.

    PBRF is totally unnecessary. All info relevant to research performance is now verifiable on the internet, so that a university’s research performance can be monitored in real time by computer geeks employed by TEC and the like.

    The deep problem is that New Zealand academic salaries are too low to hire and return decent academic performers. Not willing to pay for carrots, the House has resorted to sticks. Here’s how PBRF has been turned into a stick. At least one NZ VC has threatened to dock the budget of PVCs 40K for each individual with a failing grade that is part of the PVCs staff complement. You can be sure that those receiving a failing grade will not be extended the usual professional and personal courtesies. What PBRF has done is to reward the victimisation of academic staff until they retire or resign before the PBRF census date.

    PBRF was invented by an accounting academic named Jonathan Boston. He was inspired by the now-defunct British RAE scheme. PBRF will probably die once Labour is back in power.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  15. krazykiwi (9,189 comments) says:

    Accent Learning, which provides professional development for teachers was spun off Vic Uni to improve PBRF rankings.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  16. Kimble (4,375 comments) says:

    To my mind the secret is twofold. Firstly, to design the system well enough that it’s easier to just do a good job than it is to game it. This usually means a blend of measures and enough subjective/personal input to catch those who are blatantly gaming. The second is to change the system often enough to avoid systematic gaming to build up.

    But before all that, declare that any university found to be gaming the system will lose all funding until one year after the individuals involved are removed from the payroll. Hire one guy to police the system. Make sure he’s a bastard.

    $500m is a big carrot. The stick should be bigger.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  17. PaulL (5,872 comments) says:

    Why do we need to wait for Labour to be back in power before PBRF dies? Sounds like too long to wait – who has 9 years? :-)

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  18. djf001 (3 comments) says:

    Tell about this “Accent Learning” business?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  19. Julian (156 comments) says:

    Ricky Rowling has the correct idea – why not just base it on a weighted average over the academic year (or period between census days)? Fairly obvious that this system is going to encourage gaming.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  20. Tom Jackson (2,458 comments) says:

    I think you have the wrong end of the stick here. I’m in a similar position to Individual B. A friend and I do the bulk of the first year teaching and grading for a medium sized department. This came about in both cases by accident, as I had quit my academic job for family reasons and was asked to fill in at another institution (similar case for my friend). This has carried on for a few years.

    I’m not research active, although I published this year (a holdover from old work) but this means I do all sorts of donkey work instead. I also don’t get paid anything like what the research active staff get paid (which I don’t mind). Our department is clearly better off employing me to do this stuff than some inexperienced honors or masters student, and this shows in the teaching evaluations and course outcomes (it’s hella obvious to everyone involved). Both the students and the rest of the department benefit from having us around, but the PBRF doesn’t have a way of evaluating that, and so we get this monkey business instead. while in some subjects you need active researchers to teach them, in others and at some levels you just need someone with an advanced degree and experience (e.g. Languages).

    All the PBRF does is divert attention away from teaching to research. Students are now being asked to make an even bigger financial contribution while their lecturers have no incentive to pay them much attention. I myself wonder why more of them aren’t throwing a fit about it. Contact hours have been cut (they are now 3/4 or less of what I got) and ineffective teaching is the norm.

    You can’t run a university department properly if incentives are skewed, and that’s what the PBRF does. It’s a one size fits all solution forced upon subjects that are wildly diverse.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  21. SGA (800 comments) says:

    I agree with Tom and others. By necessity, the universities are PBRF obsessed, and all of them are looking for whatever edge they can get. There are “dummy runs” in-house, and every PBRF-eligible staff will have their evidence portfolios scrutinized two or three times before they are submitted to the PBRF panels (with comments and advice about how to “spin” their entries to the maximum effect). The amount of time, direct and indirect costs, and energy that goes into this exercise is staggering. Teaching commitment and ability is now all but forgotten in terms of hiring (although the universities would deny this), it’s largely about what the person might get on their PBRF score. This particular method of rating the universities productivity is a bean counter’s paradise (and at the end of day reflects your cunning at playing the game as much as anything else).

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  22. scrubone (3,044 comments) says:

    It’s an incredibly complicated system. But so are systems overseas.

    I attended an internal talk at Otago after the first round and basically Otago was the chump who hadn’t gamed the system (hence the #3 ranking for a university everyone knows is top knotch for research). In fact Otago had actually applied an internal standard that was too high, and didn’t get scores for people that actually might have improved their rankings. They certainly learned their lesson well however and pushed Auckland off the top ranking in the next round.

    I know that a lot of staff at Otago were given 11 month contracts (December off) that paid as well as the former year long contracts, so that they didn’t count.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote

Leave a Reply

You must be logged in to post a comment.