PBRF rorting not only happening at Vic

January 26th, 2014 at 12:00 pm by David Farrar

Have heard that VUW not the only university that went to extraordinary lengths to rort the PBRF. Have heard that AUT actually got rid of tenured professors just to boost their PBRF grades, even though the Tertiary Education Commisson issued a bylaw under the Education Act specifically prohibiting such actions.

It seems there have been many academics in NZ whose contracts have been terminated in an effort to boost PBRF grades.  One of them has suggested there could be grounds for a class action. I am happy to put those affected in touch with each other, if they contact by e-mail.

Tags: ,

Quality or quality rorting?

April 12th, 2013 at 3:00 pm by David Farrar

The Herald reports:

The University of Auckland has been knocked off its perch in a ranking of universities on the research performance of academic staff.

Rival Victoria University of Wellington is now number one in New Zealand, according to the latest Performance-Based Research Fund (PBRF) Quality Evaluation, released yesterday by the Tertiary Education Commission (TEC).

The University of Auckland, which under current measures ranked first in both the 2003 and 2006 evaluations, ranked second. The PBRF is a system used to assess the research performance of tertiary education organisations.

Its results are used in the allocation of about $263 million a year in research funding.

VUW has had an active programme in place to basically rort the PBRF rankings by manipulating employment contracts and the like. I’ve blogged on these in the past.

Now it may be that their first place is genuinely deserved, but their constant attempts to artificially manipulate the rankings means we can’t be sure.

UPDATE. A reader does some sums:

VUW put 641.54 staff members into the PBRF evaluation exercise, out of a claimed 779.91 eligible staff members (i.e. 82% of its purportedly eligible academic staff participated). On its face, this looks great – most of VUW’s academic staff got included in the evaluation, meaning that its score must be a good representation of the quality of its academics as a whole. And so VUW came out top on the PBRF assessments for not only its average research quality score per participating staff member (the figure everyone is fixating on as showing the “best” research institution in the country), but also its score for research quality of its staff as a whole.

However, when you go back to the last PBRF round (in 2006), things start to look a bit strange. In 2006, VUW put up 598.5 staff for PBRF assessment – 43 fewer than in 2012. However, in 2006 VUW also reported having 988.1 staff eligible for PBRF … some 209 more than in 2012. So, in 6 years, some 21% of VUW’s staff appear to have vanished from its books when it comes to PBRF purposes – despite its roll having increased by about 1000 EFTS (equivalent full time students) in that period. (I’d also note that VUW is the only University to show anything like this level of decline in staff numbers, so it cannot solely be a consequence of changing the rules on who counts as a PBRF eligible staff member).

Then you compare VUW’s reported staff numbers in 2012 to the other Universities in the process, and some other funny things start to jump out at you. VUW had 16,690.43 EFTS on its books in 2012. In comparison, AUT has 15,771.56 EFTS … about 1000 less. Yet, somehow, AUT has 952.1 eligible staff members … 170 more than VUW does. So despite having 6% fewer students on its books, AUT apparently has 22% more staff members who are eligible for PBRF purposes than VUW does. Which seems somewhat odd.

Equally, Otago has 18,715.9 EFTS – 2000 more than VUW. But for the PBRF exercise, Otago put 1168.24 staff members in for PBRF evaluation, out of its 1567.5 eligible staff. That is to say, despite having only about 12% more students than VUW, Otago put forward almost twice the number of staff for assessment, and apparently has twice as many PBRF eligible staff on its books. Which again seems odd.

Now, you’d expect this fact to show up somewhere in the PBRF results, and it sort of does. One of the figures the Tertiary Education Commission reported is a measure of “the extent degree-level and above teaching and learning is underpinned by research at each [institution]” In other words, how much are students actually getting their knowledge from the people who are doing the research the PBRF exercise is measuring. And here VUW’s result plummets – from being top of the research rankings, it falls to sixth out of the nine Universities. There is one PBRF assessed staff member for every 26 students at VUW, as compared to one for every 20 at Auckland or one for every 16 at Otago.

Why is this? Well, because VUW has only put in a relatively small number of staff members into the PBRF exercise compared to other Universities, there aren’t as many of them to stand in front of a classroom and teach the (comparatively large) number of students enrolled at VUW. 

So, if we’re going to treat the PBRF results as telling us anything useful about Universities, then here’s the message it give. VUW apparently has a comparatively small number of researchers who produce higher quality work than their peers at other institutions. But if you are a student wanting to study at a University where your teachers are also actively engaged in research activities, you should be cautious about attending VUW as there will be comparatively more of you competing for the attention of those higher-quality research staff – meaning that the actual teaching you receive is more likely to come from staff members who aren’t active in researching the topics they are telling you about. 

Alternatively, we could recognise the PBRF process for what it is, and treat any claims that it accurately measures “quality” as requiring more than a pinch of salt.

It is pretty clear from these numbers that VUW’s ascendancy to the top of the table is more to do with their disciplined program to reclassify staff, than any actual increase in quality.

The Government has already made some rule changes in response to VUW’s rorting and rigging attempts. I think they have a bit more work to do.

Tags: ,

Looks like VUW still trying to rort the PBRF

November 15th, 2012 at 11:00 am by David Farrar

I got sent a copy of the e-mail below, which speaks for itself. The e-mail was not supplied to me by the author, and I have had no contact with him on this.

Dear Council members

I have recently received what some people might think to be the unsurprising result from having repeatedly raised questions about the integrity of VUW’s PBRF submissions: the threat of disciplinary action.

The background is as follows.  TEC’s removal of ‘R’ staff from the PBRF assessment, in order to undercut the widespread attempts by universities to manipulate their PBRF scores, has closed off one avenue for manipulation but others remain.  One of these would be to grant an employment contract start date before June 14 to a staff member who arrives after June 14 so as to ensure that the individual could be submitted for PBRF (especially if they are not coming from another NZ university, in which case the entire PBRF benefit from the individual would flow to VUW rather than being shared with the NZ university from which they came).  I think that any such behaviour would contravene the requirement to observe the “spirit” of the PBRF process and, since it would involve money as well as PBRF rankings, is much more serious.

Regrettably I have heard a number of rumours of this type of activity at VUW.  In respect of one such case, involving A of the School of XX, who came from overseas, I spoke to him on October 8.  Based on that conversation and some earlier information, I understand that his contract start date is 1 June and he arrived on 27 June and he is being submitted for PBRF.  During this conversation I also suggested to him that the 1 June contract start date may have been chosen by University management to manipulate the University’s PBRF score (and also improperly increase the money it received from government under the PBRF scheme) and therefore he might be an unwitting participant in improper activities.

You might wonder whether it is normal practice at VUW to start paying academic staff for a month before they arrive, and therefore no grounds for concern in this case.  However I understand it is not normal practice here and, in the experience of most of the people I spoke to on this matter (former Heads of School or those in a similar position to be aware of VUW’s practices), it is unprecedented behaviour.

You may also wonder whether this is mere quibbling over a ‘few days’ and therefore would not concern TEC.  However, since I have worked as an auditor, I can confidently state that auditors (reasonably) take a considerable interest in potentially manipulative dating of transactions close to accounting census dates and my discussions with TEC’s auditors (at KPMG) leave me satisfied that they take the same view in the present circumstances.

Despite these grounds for suspicion over A’s situation, I continued to collect further information on the general issue believing that I needed considerably more information before raising this matter with the Chancellor.  However, my hand was forced by the receipt on 16 October of the attached letter from Professor Buckle, PVC of Commerce, threatening me with disciplinary action and demanding certain information.  Evidently the letter was prompted by A discussing our conversation with Prof Buckle.  This letter and subsequent events have magnified my suspicions about A’s situation, as follows.

Firstly, in his letter, Prof Buckle did not deny any impropriety on the part of University management despite the suspicion of it.  If there is no impropriety, why would Prof Buckle fail to deny it and instead focus exclusively on the question of how I learned of A’s contract start date being 1 June and his arrival date being 27 June?  This looks like an attempt to ‘shoot the messenger’.

Secondly, in his letter, Prof Buckle demanded to know how I learned of A’s arrival date and he characterises this information as “private and confidential”.  The arrival of most new employees to a workplace is an event observable by many people and managers usually go to some trouble both to advertise their arrival and to introduce them to their colleagues.  Universities are no exception to this general pattern of behaviour.  Furthermore an examination of the staff list for School XX on the University’s webpage in late June (with a 20 June update date for the page) gives A a phone number that is actually the number for a School administrator, and the only other academic staff so treated are adjunct professors who are usually absent, whilst later versions record a phone number for A corresponding to A’s office.  These webpages alone suggest that A did not arrive until late June and they are publicly available information.  If there is no impropriety, and therefore no impulsive reaction at its discovery, why would Prof Buckle make the extraordinary claim that A’s arrival date was “private and confidential information”?

Thirdly, in his letter, Prof Buckle demanded to know how I learned of A’s contract start date and he characterises this information as “private and confidential”.  Implicit in this demand is the belief that some third party who was entitled to know the information, but not to disclose it to me or others, did so.  However there are a number of other possibilities, including A having disclosed it to me (which is the case) and A having disclosed it to someone else who then gave it to me, neither of which would presumably constitute an offence.   Furthermore one might have suspected that A’s contract start date was around 1 June purely on the basis that a University webpage was created for A on 30 May (the webpages have footers that note the date they were last changed).  If there is no impropriety, and therefore no impulsive reaction at its discovery, why would Prof Buckle have apparently forgotten that there are many legitimate channels through which information can be conveyed, leap to the conclusion that the channel was illegitimate, and threaten me with disciplinary action without first simply checking with me whether the channel was legitimate?

Fourthly, in his letter, Prof Buckle claimed that A arrived in his office in a “very distressed state”.  This is very strong language.  Perhaps it is true.  If so, it might be because A is an expert in XX (as is apparent from A’s University webpage) and quickly appreciated the potential damage to his professional reputation from being even an unwitting participant in improper activities.

Fifthly, in his letter, Prof Buckle threatened to subject me to some sort of disciplinary process.  Whatever the merits of that might be (probably minimal since A himself provided me with the allegedly private and confidential information about his contract start date), I think that Prof Buckle is seriously conflicted in threatening disciplinary action against me because the allegedly confidential information raises the suspicion of impropriety by himself.  If there is no impropriety, and therefore no impulsive reaction at its discovery, why would Prof Buckle have overlooked the obvious conflict of interest?

Upon receiving this letter from Prof Buckle, I passed it over to my lawyers and after discussion with them I sent the below email message to Mr McKinnon and Prof Buckle on October 19.  As you will see I offered to leave aside past events so long as a satisfactory process to deal with future concerns was implemented.  My lawyer also sent an email to both Mr McKinnon and Prof Buckle on October 23 referring to my October 19 email.  By Nov 5, neither I nor my lawyer received even an acknowledgement of receipt, let alone a substantive response to either email from either recipient.  I therefore emailed both Mr McKinnon and Prof Buckle again on November 5, pointing out the failure to respond, submitting a draft version of an email to Council, and inviting both of them to point out any errors of fact or interpretation by 10am on November 7.  I also copied this to A and invited him to point out any errors of fact or interpretation.

This email to Mr McKinnon and Prof Buckle finally produced a response.  On the evening of Nov 6 I received a reply from Mr McKinnon stating that my Oct 19 email had not been forwarded to him (with a subsequent email to my lawyer offering the same explanation for not having seen my lawyer’s email).  In his email to me Mr McKinnon stated that the issues here were matters to be “addressed by University Management”.  I am very surprised about Mr McKinnon’s explanation for not having received either of the two email messages, sent by different people and several days apart, and also copied to a another person (Prof Buckle) who might reasonably be expected to have discussed this with the VC and the VC in turn with Mr McKinnon.  Furthermore, I do not think that suspicions of impropriety by University management are matters to be “addressed by University Management”; the conflict of interest is stark.

In addition, on the evening of Nov 6 and shortly after Mr McKinnon’s email, I received an email from Prof Buckle, acknowledging receipt of my Nov 5 email (which has the Oct 19 email appended to it), denying any impropriety on the part of the University, and stating that he now proposed to deal “informally” with my conduct towards A.  Presumably this email constitutes the response from University management that Mr McKinnon referred to.  Remarkably Prof Buckle did not offer any explanation or apology for the delay in responding to my Oct 19 email.  I am left with the impression that he had elected not to respond to my October 19 email until my email of Nov 5 forced a response.  Furthermore, whilst Prof Buckle denies any impropriety by the University, he does not dispute any of the specific points raised in my Oct 19 email.  In particular, he does not dispute that A arrived on 27 June and that his contract start date was 1 June and that he has been submitted for PBRF (in any case, the first two points are implicitly conceded in the first two sentences of the third para of Prof Buckle’s letter).  In respect of Prof Buckle’s change of heart on the merits of a disciplinary process, this suggests that he now realises that his actions of October 16 were unwarranted but there is no apology, he still thinks that he has some quasi-judicial role to play, and he therefore still does not accept that he is conflicted.

In addition I now understand that University management has lodged an OIA request for my communications with TEC and their auditors.  Given University management’s concern with my possession of “private and confidential information” relating to A, it is rather ironic that they are seeking to obtain information about communications between parties that were clearly intended to be “private and confidential”, and the clear effect of any such OIA request would be to discourage University staff from providing information to TEC or their auditors (thereby undermining TEC’s legitimate efforts to actively seek out information from university staff). 

After my October 8 discussion with A I did not think that there was sufficient cause to contact the Chancellor or you.  As a result of events since then, I now think that there is.  These additional events include unwarranted threats that have now been retracted, conflicts of interest, systematic non-response to emails, the failure at any point to dispute any of the specific indicators of impropriety, actions undermining TEC’s legitimate efforts to obtain information, and the extraordinary claim that the arrival date of an academic at this University is “private and confidential information”.  I now believe that the circumstances relating to A should be investigated.  Even if you limit yourself to information from University management, the first two sentences of the third paragraph of Prof Buckle’s letter concede that the contract start date is 1 June and the arrival date is later in June.  Furthermore, anyone with a contract start date of 1 June is prima facie eligible for inclusion in PBRF; coupled with the fact that A’s CV is impressive, and that the University has a strong interest in its PBRF performance, one would further strongly suspect on this basis alone that A had been submitted for PBRF.

As you will see from my email, I refer to an additional case to that of A, and I have heard rumours of other such cases.  Quite apart from my actions, any activities of this kind that are occurring are very likely to be detected by TEC’s auditors anyway.  As a former auditor, I can confidently state that auditors (reasonably) take a considerable interest in potentially manipulative dating of transactions close to accounting census dates.  It would therefore be basic audit practice in the present situation to draw up a list of staff being submitted for PBRF whose contract start dates were on or within the few weeks  preceding June 14 (as per a review of the contracts) and who arrived after June 14 (as per an itinerary or air ticket).  Furthermore, TEC’s earlier public rebuke of universities for attempting to manipulate PBRF scores by hiding ‘R’ staff would have made their auditors even more likely to suspect that universities would attempt to manipulate the contract start dates of staff arriving just after 14 June.  So, if such actions are occurring, it would be naïve for University management to imagine that TEC’s auditors will not detect them (just as it was naïve for management to believe that all of the earlier activities concerning ‘R’ researchers that were detected by TEC would not be detected).  It is also highly damaging to the reputations of every staff member who becomes an unwitting participant in improper activities because there will be at least the suspicion that they perfectly understood what was happening (particularly if they are an expert in XX).

Despite all of this, my offer of October 19 to leave aside past events so long as a satisfactory process to deal with future concerns is implemented still stands.  I think this would be a much better outcome than an ongoing flow of accusations followed by a further public rebuke of universities by the TEC.

I would like to finish with some quotations from University management:

An employee’s obligations include “..a positive duty to draw to your employer’s attention the improper actions of another employee.” (Prof Buckle, Oct 16 letter).

We require ethical behaviour in all our activities and interactions” (from the University’s Mission Statement at http://www.victoria.ac.nz/home/about/working/missionandvalues).

 

Martin Lally

The issue is a pretty simple one. Does VUW arrange its employment contracts to state someone has started employment earlier than they actually have, in order to gain more funding via the PBRF?

Hopefully someone who can get answers to those questions, is asking them.

Tags: ,

TEU on PBRF

May 8th, 2012 at 3:00 pm by David Farrar

A guest post by the Tertiary Education Union on the Performance Based Research Fund:

A recent report on the behaviour of New Zealand’s universities around research performance funding confirms a truism – you get what you reward. And when rewards for universities are based on their ability to generate research outputs, we should not be surprised at the lengths they will go to in order to be on top.

The Tertiary Education Commission confirmed last week that universities were gaming its performance research funding scheme (PBRF) to make sure they could get a higher ranking in the exercise than other universities. Universities have been changing people’s employment agreements, restructuring departments and people’s jobs and in some cases making teaching-focused academics redundant simply so that they can appear higher on a rankings ladder than other universities.

While institutions get no more tax-payer dollars from improving their rankings in the PBRF exercise, they are fighting for prestige – to be “New Zealand’s Leading University” or our “top ranked university for research quality”. This race for rankings was not the intent of the research performance exercise and the commission now believes the system warrants change.

But the commission faces a Herculean task in reforming PBRF while trying to remain true to the core principle of performance based research funding – that you can drive up research performance with the right competitive incentives.

One thing we have learned in recent years from universities is will take the most direct route to the cash and the rankings no matter what stands in their way (think Pamplona’s running of the bulls for the right mental image of universities pursuing performance funding).

A decade ago, when the incentive was ‘bums on seats’ tertiary institutions restructured courses and sucked students out of schools and other tertiary education providers onto their campuses. Some of the outcomes of this market model were positive – New Zealand has one of the highest tertiary education participation rates in the world. However, there were, as has been widely documented in the past, many perverse outcomes – from massive marketing budgets to the now infamous ‘twilight golf’.

So we identified the perverse outcomes of funding based on student numbers and declared the way forward was not about mass participation but about high-end research.

Student numbers were capped and a new competitive incentive introduced centred on published research ‘outputs’ (primarily peer reviewed articles in international journals written for the academic community). 

The aim was to calculate an aggregate research performance score for each institution in order to allocate them funding commensurate with their overall research performance.

TEU has no problem with rewarding excellence in our institutions, but research outputs are only one part of the work of tertiary education staff – academics also are responsible for teaching, administrative tasks, engaging in public meetings to share their knowledge, and so on.

Yet the lure of being ranked number one has meant universities have punished award-winning teachers; asked academics to shift their focus from community research to peer reviewed journals; and questioned the future of academics who perform vital administrative roles, because this takes time away from writing journal articles.

So how do we stop universities from narrowing the role of an academic down to ‘research’, when that is what is what the government rewards?

TEC has proposed changing the way the ‘rankings’ are put together, by only calculating the average quality score of tertiary institutions based on those staff who actually receive one of the quality scores (As, Bs, and Cs) and excluding those who (for a whole host of good reasons) don’t even reach the bar in this narrow performance measure.

This move will not help students who have lost their favourite university teacher, departments who have lost great administrative academics, or communities who couldn’t find an academic expert to attend a public meeting because they are all too busy writing journal articles.

It is a warranted action and will reinforce what the PBRF exercise was about and show institutions they should ‘play by the rules of the game’ with integrity. After all the rules are explicit: PBRF scores are not to be used for hiring and firing academics; not to be used for promotions or performance management. What PBRF aggregate scores are for is to provide an auditing and accountability tool that can help the government dish out the dollars ($1.6 billion over six years) to institutions.

However, changing the rules for calculating the average scores of universities is only a stopgap measure. What the latest report from the commission highlights are fundamental problems with designing performance measures in the tertiary education system. This means a much bigger debate is needed: how do we get the best out of our tertiary education staff?

TEU often argues that the evidence shows performance incentives do not work – you need only look at the mostly incentive-based packages of today’s CEOs to see that any chief executive not worth his or her salt is still clever enough to rort the system and generate an exponentially increasing take home pay. And the creative industries – such as major companies in Silicon Valley – are again realising that giving workers greater autonomy (rather than strict line-management) is the way to get great work from staff. Our worry is that the government sees the perverse outcomes of auditing measures like PBRF but thinks this can be solved by just adding in more performance incentives.

So instead of rethinking whether performance measures work in the tertiary sector, the government has set up a performance exercise looking at student retention and completion. For tertiary institutions the quickest route to achieving in this exercise is making sure students pass their courses. The simplest way to ensure students pass is to put pressure on academics to elevate grades (and in a few isolated cases this is already beginning to happen in a range of institutions across New Zealand).

Should we just sit back and hope that gaming is not so accepted by our tertiary providers, and that they will not pervert good credentialing (the awarding of qualifications) or research performance exercises just to be ranked number 1? Or should we review whether performance based funds really help us get the best out of tertiary education sector?

Only the commission and the government can decide where we head next, but to us it seems clear – the report into PBRF gaming shows it is time to sit down and talk about how we protect the reputation of our tertiary education sector and ensure that we are getting a well-rounded set of outcomes from universities. 

Dr Sandra Grey
TEU National President

Tertiary Education Minister Steven Joyce also said:

But he has now revealed further reforms are planned for universities, including a review of the councils and a separate review of a the controversial Performance Based Research Fund (PBRF). …

Joyce, meanwhile, said funding for engineering and the physical sciences would get a boost in the Budget.

Research would also get a boost, though a modest increase in funding to the controversial Performance Based Research Fund (PBRF).

The PBRF, set up under the previous Labour administration, has been the subject of allegations of rorting by universities hiding away their least productive researchers to maximise their share of the funding. There has also been growing concern about the amount of time wasted by academics tinkering with their PBRF portfolios.

While the PBRF had been ”broadly successful,” Joyce said there were problems which would be addressed in a review this year.

“Frustrating gamesmanship” needed to be addressed and the Government was also keen to look at introducing incentives for the commercial success of research.

”You make it [PBRF] all about [publishing] papers, so if somebody steps out of the system for a year to go and work on a commercial project somewhere, does that damage their PBRF score so that they can’t contribute to what the university is being asked to contribute to?” Joyce said.

There may be a way to introduce incentives to the PBRF process for commercialising research, he said.

”You’ve got to be careful because obviously, it’s fine in engineering and economics and science and things but it’s a little bit harder in humanities.”

So further change looks likely, but whether there will be agreement on what it should be is another matter.

Tags: , , ,

TEC moves on PBRF rorting

April 12th, 2012 at 4:31 pm by David Farrar

The Tertiary Education Commission appears to be taking action over the attempts by universities to improve their relative PBRF ranking, by hiding staff with low research scores from the census done in June. I blogged extensively on this a couple of weeks ago. It does not gain the university any extra funding from the PBRF, but it does lead to them having a better average rating, and hence a higher rank.

In a consultation paper just released, the TEC states:

36 The incentives for TEOs to maximise the number of PBRF-eligible staff whose EPs meet the standard for a funded Quality Category (ie. “A”, “B”, “C”, “C(NE)”) relate to the funding that those EPs attract, and the positive contribution that funded Quality Categories make to the calculation of the AQS. It is important that the calculation of the AQS for TEOs be equitable and reliable because of the reputational impact of the relative ranking of TEOs based on the AQS.

37 As a high AQS will tend to enhance the reputation of a TEO, there are strong incentives for TEOs to minimise the number of PBRF-eligible staff whose EPs would not meet the requirements for a funded Quality Category (ie. those assigned an “R” or “R(NE)”), either by supporting these staff to develop sufficient evidence of research activity, or by seeking to have them deemed ineligible for the PBRF.

38 There are strong incentives for TEOs to take steps to minimise the number of individuals reported as PBRF-eligible, where their evidence portfolios are not likely to be assigned a funded Quality Category (A, B or C), because unfunded Quality Categories reduce the reported AQS, and by extension the ‘ranking’ of TEOs, subject areas, and nominated academic units.

39 The TEC places considerable reliance on TEOs to apply the staff eligibility criteria in a manner that is consistent with both the wording, and importantly, the intent, of the staff eligibility criteria.

40 As part of the preparations for the 2012 Quality Evaluation, the TEC has contracted KPMG to undertake an audit of TEOs. The first stage of the audit process has indicated that there appears to be differences in Human Resource practices across TEOs. In particular, differences in the way in which TEOs are applying the staff eligibility criteria in relation to staff whose EPs are not likely to meet the standard for a funded Quality Category.

A nice summation of the problem, and nice to see the have sent in auditors who have discovered how some universities are trying to rort the system.

41 The extent of these differences indicates that there is potential for these practices to affect the credibility of the published results of the 2012 Quality Evaluation, including the way in which TEOs are ranked (at both the overall organisational level, and by subject areas), and for the reporting of the relative quality of research by panel, subject area, and nominated academic unit generally.

The translation is no one would believe the results of the rankings, with the current practices employed by universities.

The TEC recommends:

55 This option involves the TEC making changes to the reporting framework for the Quality Evaluation so that any staff whose EPs were assigned the “R” or “R(NE)” Quality Categories would be excluded from the calculation of the AQS at all levels.

56 The main advantage of this approach is that it serves to remove one of the most significant incentives for TEOs to manage the employment relationships of staff for the purposes of maximising their ‘ranking’ through the Quality Evaluation measure. This approach does this by ranking TEOs solely on the basis of staff whose EPs were assigned a funded Quality Category. There would therefore be no rationale other than standard employment practices to make changes to the employment agreements of staff.

57 This approach would also ensure the results of all participating TEOs would be treated equitably and no individual TEOs would be unduly advantaged or disadvantaged by their application of the staff eligibility criteria or HR practices.

This seems a sensible way forward. It is a pity that it is necessary. If the universities were not trying to rort or skew their rankings by manipulating employment contracts, this would not be necessary. But relaying on the universities to act ethically and within the spirit of the policy seem too much to hope for, so the TEC has recommended this change.

It is good to see the Government heeding the concerns over what was happening, as it was going to reduce the rankings to a bit of a joke – the university with the most manipulative HR department would probably end up being the one which got the highest rating.

It will be interesting to see what the Tertiary Education Union says on this proposed change, as they had also expressed concern over the manipulation of employment contracts to improve ratings.

Tags: ,

More on PBRF

March 26th, 2012 at 7:22 pm by David Farrar

An update to my earlier post.

  1. The total mount of PBRF funding is $250m a year, not $500m. I did take my figure from the Treasury Appropriations, but obviously misinterpreted it.
  2. PBRF funding is itself based on actual research done, not the level of research per average staff member, so the tactics used by VUW will not in itself lead to any increase in funding.
  3. It does however allow them to be placed higher up the comparative table with other universities. This is designed to increase their reputation, attract more staff, and possibly win contests with other universities for new schools, if any eventuate. Ironically I think their reputation gets lowered, not increased, by their attempt to artificially boost their average.
  4.  Considering that they don’t actually gain any extra government funding from it (which I wasn’t clear about in the earlier post), I think it is ridiculous they spend so much time and energy trying to skew their averages, so people are fooled into thinking they are better ranked than they are.
  5. I understand that the tertiary education union has been concerned about this for sometime. Sadly their solution seems to be common to education unions – wanting the research data kept confidential! But they do support an investigation, which is good.

There’s been a number of releases in response to my blog post. They are:

  • VUW, saying their independent investigation found no grounds for the allegations
  • The TEU, backing the call for an independent investigation over the sector
  • Labour saying there should be a review of the system
  • Greens saying the system is at fault, not Vic Uni

It is worth noting that the PBRF was introduced by the former Labour Government, as far as I can tell.

Tags: , , ,

The great PBRF scam

March 26th, 2012 at 9:00 am by David Farrar

I’ve had forwarded to me a number of documents and reports from within Victoria University over its attempts to what can only be called a rort of the PBRF, or Performance Based Research Fund. The TEC says that:

The primary purpose of the PBRF is to ensure that excellent research in the tertiary education sector is encouraged and rewarded. This entails assessing the research performance of TEOs and then funding them on the basis of their performance.

As I will detail below though, in fact it seems the universities gain funding on the basis of their ability to manipulate employment contracts.

The annual value of funding from the PBRF is around $500 million, so universities are highly incentivised to maximise their share of it. Sadly it seems that rather than encouraging academics to do more and better research, VUW (and probably others) have unleashed the HR Department to try and manipulate their results. The ironic thing is that if all universities do what VUW does, then no university actually gains a comparative advantage.

A complaint was made to the VUW University Council by an Associate Professor in the Accounting Economics & Finance Department, who said what VUW was doing was effectively accounting fraud.  He gave eight examples:

  1. Individual A, who is not active in research and who has a fixed-term contract maturing after June 2012, was asked to terminate this contract in favour of another one terminating in early 2012 coupled with a further contract starting in late 2012.  The effect of this would be to exclude A from the calculation of the University’s PBRF score in June 2012.  Individual A rejected this proposal on ethical grounds.      
  1. Individual B, who is not active in research and had a series of fixed-term contracts with the last maturing in 2011, was offered at that time a replacement contract with a significantly shorter duration than before and that would mature in early 2012, with the effect that they would not be included in the calculation of the University’s PBRF score in June 2012.  B accepted this contract.    
  1. Individual C, who is not active in research and has a conventional academic contract, was asked to temporarily depart from University employment in 2012.  The effect of this would be to exclude C from the calculation of the University’s PBRF score in June 2012.  C accepted this offer.  Within a week of Mr Corkill’s inquiry being set up, these arrangements with C were cancelled.
  1. Individual D, who is not active in research and had a conventional academic contract, was asked to retire early and was offered a fixed-term contract for teaching work in 2012 with the contract period excluding June 2012.  The effect of this would be to exclude D from the calculation of the University’s PBRF score in June 2012.  D accepted this contract.
  1. Individual E, who is not active in research and had a conventional academic contract, was asked to retire early and was offered fixed-term teaching contracts for early 2012 and a further period in late 2012, with the effect of excluding them from the calculation of the University’s PBRF score in June 2012.  E accepted this contract. 
  1. Individual F, who is not active in research and had a conventional academic contract, was asked to retire early and was offered a fixed-term teaching contract for late 2012, with the effect of excluding F from the calculation of the University’s PBRF score in June 2012.  F accepted this contract.
  1. Individual G was offered teaching contracts in 2012 with the terms of the contracts such that G would be excluded from the calculation of the University’s PBRF score in June 2012.  In addition, G was advised by a University manager that this PBRF consideration was the rationale for the terms offered and that the terms were chosen sufficiently well away from the June 2012 date to avoid suspicion.
  1. A Head of School circulated an email to staff in their School advising them that academic staff members of a particular type counted for PBRF and instructed staff to therefore ensure that none of this type were employed on the June 2012 census date. 

There is also other stuff happening to massage the numbers, such as putting some staff onto teaching only contracts, so they get excluded. That is arguably more legitimate, but I struggle to see how having employment contracts that exclude the month of June, purely to avoid the PBRF census can be a correct or ethical thing to do. It also undermines the whole point of the PBRF.

The VUW Chancellor appointed a QC to investigate. His full report they refuse to release, but a summary was released. It is worth noting that his “advisor” was the Chair of the University’s Audit & Risk Committee. I regard that as a conflict. Members of the University Council naturally wish to maximise the funding for VUW. They would be concerned about behaviour that costs VUW money, but be motivated to take a tolerant view of unethical behaviour which gains VUW money.

The report summary is here – Corkill Executive Summary. A response to it from the Accounting Associate Professor Martin Lally is here – PBRF Allegations and Investigations-to dist list. Note that Martin Lally is not the person who supplied these to me. I have had no interactions with him at all. A response from the Chancellor is here, which basically tells him to go away and even declares he will not entertain further correspondence on the issue – Chancellor Letter 28 Feb.

If you bother to read the response from Lally, he points out that the summary is so lacking in details, that he can  not know whether or not his allegations are correct. VUW refuses to allow him the full report.

Further to the original allegations, there is then the e-mail below:

Now to be fair, this is not just happening at VUW. I understand there are complaints about this gaming happening at other universities also. Apart from the low ethical standard the universities are providing to students (Lally’s  concern is the university is doing something that he lectures students on, as being accounting fraud), it would be simpler if all the universities just agreed they won’t game their numbers, if the others won’t.  The amount of time and energy put into this gaming is significant. Every Department in every Faculty has to take part.

There are a lot of unhappy academic staff about this. Since I gained the original materials, I have spoken to several acquaintances who are academics and they all confirm that this is wide-spread.

I understand TEC are aware of all this. I can’t understand why they haven’t said “No you can’t sack a staff member in May and rehire them in July, just to make your PBRF ranking look better”.

This poses an issue for the Minister also. $500 million of taxpayer funds go towards universities and others through the PBRF every year. The PBRF was set up by (I think) the last Labour Government. The intention of it is good – that universities which are producing more research and better research gain more funding. I spent a couple of terms on the Otago University Council and recall that even at Otago, there were some academics who produced minimal original research. PBRF should be encouraging more original research.

But the gaming of the system must seriously bring into question whether the $500 million is going to the right institutions in the right proportions. In theory getting rid of staff for a month who are not doing much research doesn’t actually gain an institution more money directly (as I understand it), but it increases their relative ranking, which benefits them for decisions on other issues such as where new schools might be located, or attracting staff.

At a minimum, in my opinion,  the PBRF rules need to be reviewed to discourage such gaming of the system. Ideally there would be a proper external inquiry into all the universities, to discover how widespread this gaming is, and whether it it ethical and legal.

Tags: , ,