Quality or quality rorting?

April 12th, 2013 at 3:00 pm by David Farrar

The Herald reports:

The University of Auckland has been knocked off its perch in a ranking of universities on the research performance of academic staff.

Rival Victoria University of Wellington is now number one in New Zealand, according to the latest Performance-Based Research Fund (PBRF) Quality Evaluation, released yesterday by the Tertiary Education Commission (TEC).

The University of Auckland, which under current measures ranked first in both the 2003 and 2006 evaluations, ranked second. The PBRF is a system used to assess the research performance of tertiary education organisations.

Its results are used in the allocation of about $263 million a year in research funding.

VUW has had an active programme in place to basically rort the PBRF rankings by manipulating employment contracts and the like. I’ve blogged on these in the past.

Now it may be that their first place is genuinely deserved, but their constant attempts to artificially manipulate the rankings means we can’t be sure.

UPDATE. A reader does some sums:

VUW put 641.54 staff members into the PBRF evaluation exercise, out of a claimed 779.91 eligible staff members (i.e. 82% of its purportedly eligible academic staff participated). On its face, this looks great – most of VUW’s academic staff got included in the evaluation, meaning that its score must be a good representation of the quality of its academics as a whole. And so VUW came out top on the PBRF assessments for not only its average research quality score per participating staff member (the figure everyone is fixating on as showing the “best” research institution in the country), but also its score for research quality of its staff as a whole.

However, when you go back to the last PBRF round (in 2006), things start to look a bit strange. In 2006, VUW put up 598.5 staff for PBRF assessment – 43 fewer than in 2012. However, in 2006 VUW also reported having 988.1 staff eligible for PBRF … some 209 more than in 2012. So, in 6 years, some 21% of VUW’s staff appear to have vanished from its books when it comes to PBRF purposes – despite its roll having increased by about 1000 EFTS (equivalent full time students) in that period. (I’d also note that VUW is the only University to show anything like this level of decline in staff numbers, so it cannot solely be a consequence of changing the rules on who counts as a PBRF eligible staff member).

Then you compare VUW’s reported staff numbers in 2012 to the other Universities in the process, and some other funny things start to jump out at you. VUW had 16,690.43 EFTS on its books in 2012. In comparison, AUT has 15,771.56 EFTS … about 1000 less. Yet, somehow, AUT has 952.1 eligible staff members … 170 more than VUW does. So despite having 6% fewer students on its books, AUT apparently has 22% more staff members who are eligible for PBRF purposes than VUW does. Which seems somewhat odd.

Equally, Otago has 18,715.9 EFTS – 2000 more than VUW. But for the PBRF exercise, Otago put 1168.24 staff members in for PBRF evaluation, out of its 1567.5 eligible staff. That is to say, despite having only about 12% more students than VUW, Otago put forward almost twice the number of staff for assessment, and apparently has twice as many PBRF eligible staff on its books. Which again seems odd.

Now, you’d expect this fact to show up somewhere in the PBRF results, and it sort of does. One of the figures the Tertiary Education Commission reported is a measure of “the extent degree-level and above teaching and learning is underpinned by research at each [institution]” In other words, how much are students actually getting their knowledge from the people who are doing the research the PBRF exercise is measuring. And here VUW’s result plummets – from being top of the research rankings, it falls to sixth out of the nine Universities. There is one PBRF assessed staff member for every 26 students at VUW, as compared to one for every 20 at Auckland or one for every 16 at Otago.

Why is this? Well, because VUW has only put in a relatively small number of staff members into the PBRF exercise compared to other Universities, there aren’t as many of them to stand in front of a classroom and teach the (comparatively large) number of students enrolled at VUW. 

So, if we’re going to treat the PBRF results as telling us anything useful about Universities, then here’s the message it give. VUW apparently has a comparatively small number of researchers who produce higher quality work than their peers at other institutions. But if you are a student wanting to study at a University where your teachers are also actively engaged in research activities, you should be cautious about attending VUW as there will be comparatively more of you competing for the attention of those higher-quality research staff – meaning that the actual teaching you receive is more likely to come from staff members who aren’t active in researching the topics they are telling you about. 

Alternatively, we could recognise the PBRF process for what it is, and treat any claims that it accurately measures “quality” as requiring more than a pinch of salt.

It is pretty clear from these numbers that VUW’s ascendancy to the top of the table is more to do with their disciplined program to reclassify staff, than any actual increase in quality.

The Government has already made some rule changes in response to VUW’s rorting and rigging attempts. I think they have a bit more work to do.

Tags: ,

Looks like VUW still trying to rort the PBRF

November 15th, 2012 at 11:00 am by David Farrar

I got sent a copy of the e-mail below, which speaks for itself. The e-mail was not supplied to me by the author, and I have had no contact with him on this.

Dear Council members

I have recently received what some people might think to be the unsurprising result from having repeatedly raised questions about the integrity of VUW’s PBRF submissions: the threat of disciplinary action.

The background is as follows.  TEC’s removal of ‘R’ staff from the PBRF assessment, in order to undercut the widespread attempts by universities to manipulate their PBRF scores, has closed off one avenue for manipulation but others remain.  One of these would be to grant an employment contract start date before June 14 to a staff member who arrives after June 14 so as to ensure that the individual could be submitted for PBRF (especially if they are not coming from another NZ university, in which case the entire PBRF benefit from the individual would flow to VUW rather than being shared with the NZ university from which they came).  I think that any such behaviour would contravene the requirement to observe the “spirit” of the PBRF process and, since it would involve money as well as PBRF rankings, is much more serious.

Regrettably I have heard a number of rumours of this type of activity at VUW.  In respect of one such case, involving A of the School of XX, who came from overseas, I spoke to him on October 8.  Based on that conversation and some earlier information, I understand that his contract start date is 1 June and he arrived on 27 June and he is being submitted for PBRF.  During this conversation I also suggested to him that the 1 June contract start date may have been chosen by University management to manipulate the University’s PBRF score (and also improperly increase the money it received from government under the PBRF scheme) and therefore he might be an unwitting participant in improper activities.

You might wonder whether it is normal practice at VUW to start paying academic staff for a month before they arrive, and therefore no grounds for concern in this case.  However I understand it is not normal practice here and, in the experience of most of the people I spoke to on this matter (former Heads of School or those in a similar position to be aware of VUW’s practices), it is unprecedented behaviour.

You may also wonder whether this is mere quibbling over a ‘few days’ and therefore would not concern TEC.  However, since I have worked as an auditor, I can confidently state that auditors (reasonably) take a considerable interest in potentially manipulative dating of transactions close to accounting census dates and my discussions with TEC’s auditors (at KPMG) leave me satisfied that they take the same view in the present circumstances.

Despite these grounds for suspicion over A’s situation, I continued to collect further information on the general issue believing that I needed considerably more information before raising this matter with the Chancellor.  However, my hand was forced by the receipt on 16 October of the attached letter from Professor Buckle, PVC of Commerce, threatening me with disciplinary action and demanding certain information.  Evidently the letter was prompted by A discussing our conversation with Prof Buckle.  This letter and subsequent events have magnified my suspicions about A’s situation, as follows.

Firstly, in his letter, Prof Buckle did not deny any impropriety on the part of University management despite the suspicion of it.  If there is no impropriety, why would Prof Buckle fail to deny it and instead focus exclusively on the question of how I learned of A’s contract start date being 1 June and his arrival date being 27 June?  This looks like an attempt to ‘shoot the messenger’.

Secondly, in his letter, Prof Buckle demanded to know how I learned of A’s arrival date and he characterises this information as “private and confidential”.  The arrival of most new employees to a workplace is an event observable by many people and managers usually go to some trouble both to advertise their arrival and to introduce them to their colleagues.  Universities are no exception to this general pattern of behaviour.  Furthermore an examination of the staff list for School XX on the University’s webpage in late June (with a 20 June update date for the page) gives A a phone number that is actually the number for a School administrator, and the only other academic staff so treated are adjunct professors who are usually absent, whilst later versions record a phone number for A corresponding to A’s office.  These webpages alone suggest that A did not arrive until late June and they are publicly available information.  If there is no impropriety, and therefore no impulsive reaction at its discovery, why would Prof Buckle make the extraordinary claim that A’s arrival date was “private and confidential information”?

Thirdly, in his letter, Prof Buckle demanded to know how I learned of A’s contract start date and he characterises this information as “private and confidential”.  Implicit in this demand is the belief that some third party who was entitled to know the information, but not to disclose it to me or others, did so.  However there are a number of other possibilities, including A having disclosed it to me (which is the case) and A having disclosed it to someone else who then gave it to me, neither of which would presumably constitute an offence.   Furthermore one might have suspected that A’s contract start date was around 1 June purely on the basis that a University webpage was created for A on 30 May (the webpages have footers that note the date they were last changed).  If there is no impropriety, and therefore no impulsive reaction at its discovery, why would Prof Buckle have apparently forgotten that there are many legitimate channels through which information can be conveyed, leap to the conclusion that the channel was illegitimate, and threaten me with disciplinary action without first simply checking with me whether the channel was legitimate?

Fourthly, in his letter, Prof Buckle claimed that A arrived in his office in a “very distressed state”.  This is very strong language.  Perhaps it is true.  If so, it might be because A is an expert in XX (as is apparent from A’s University webpage) and quickly appreciated the potential damage to his professional reputation from being even an unwitting participant in improper activities.

Fifthly, in his letter, Prof Buckle threatened to subject me to some sort of disciplinary process.  Whatever the merits of that might be (probably minimal since A himself provided me with the allegedly private and confidential information about his contract start date), I think that Prof Buckle is seriously conflicted in threatening disciplinary action against me because the allegedly confidential information raises the suspicion of impropriety by himself.  If there is no impropriety, and therefore no impulsive reaction at its discovery, why would Prof Buckle have overlooked the obvious conflict of interest?

Upon receiving this letter from Prof Buckle, I passed it over to my lawyers and after discussion with them I sent the below email message to Mr McKinnon and Prof Buckle on October 19.  As you will see I offered to leave aside past events so long as a satisfactory process to deal with future concerns was implemented.  My lawyer also sent an email to both Mr McKinnon and Prof Buckle on October 23 referring to my October 19 email.  By Nov 5, neither I nor my lawyer received even an acknowledgement of receipt, let alone a substantive response to either email from either recipient.  I therefore emailed both Mr McKinnon and Prof Buckle again on November 5, pointing out the failure to respond, submitting a draft version of an email to Council, and inviting both of them to point out any errors of fact or interpretation by 10am on November 7.  I also copied this to A and invited him to point out any errors of fact or interpretation.

This email to Mr McKinnon and Prof Buckle finally produced a response.  On the evening of Nov 6 I received a reply from Mr McKinnon stating that my Oct 19 email had not been forwarded to him (with a subsequent email to my lawyer offering the same explanation for not having seen my lawyer’s email).  In his email to me Mr McKinnon stated that the issues here were matters to be “addressed by University Management”.  I am very surprised about Mr McKinnon’s explanation for not having received either of the two email messages, sent by different people and several days apart, and also copied to a another person (Prof Buckle) who might reasonably be expected to have discussed this with the VC and the VC in turn with Mr McKinnon.  Furthermore, I do not think that suspicions of impropriety by University management are matters to be “addressed by University Management”; the conflict of interest is stark.

In addition, on the evening of Nov 6 and shortly after Mr McKinnon’s email, I received an email from Prof Buckle, acknowledging receipt of my Nov 5 email (which has the Oct 19 email appended to it), denying any impropriety on the part of the University, and stating that he now proposed to deal “informally” with my conduct towards A.  Presumably this email constitutes the response from University management that Mr McKinnon referred to.  Remarkably Prof Buckle did not offer any explanation or apology for the delay in responding to my Oct 19 email.  I am left with the impression that he had elected not to respond to my October 19 email until my email of Nov 5 forced a response.  Furthermore, whilst Prof Buckle denies any impropriety by the University, he does not dispute any of the specific points raised in my Oct 19 email.  In particular, he does not dispute that A arrived on 27 June and that his contract start date was 1 June and that he has been submitted for PBRF (in any case, the first two points are implicitly conceded in the first two sentences of the third para of Prof Buckle’s letter).  In respect of Prof Buckle’s change of heart on the merits of a disciplinary process, this suggests that he now realises that his actions of October 16 were unwarranted but there is no apology, he still thinks that he has some quasi-judicial role to play, and he therefore still does not accept that he is conflicted.

In addition I now understand that University management has lodged an OIA request for my communications with TEC and their auditors.  Given University management’s concern with my possession of “private and confidential information” relating to A, it is rather ironic that they are seeking to obtain information about communications between parties that were clearly intended to be “private and confidential”, and the clear effect of any such OIA request would be to discourage University staff from providing information to TEC or their auditors (thereby undermining TEC’s legitimate efforts to actively seek out information from university staff). 

After my October 8 discussion with A I did not think that there was sufficient cause to contact the Chancellor or you.  As a result of events since then, I now think that there is.  These additional events include unwarranted threats that have now been retracted, conflicts of interest, systematic non-response to emails, the failure at any point to dispute any of the specific indicators of impropriety, actions undermining TEC’s legitimate efforts to obtain information, and the extraordinary claim that the arrival date of an academic at this University is “private and confidential information”.  I now believe that the circumstances relating to A should be investigated.  Even if you limit yourself to information from University management, the first two sentences of the third paragraph of Prof Buckle’s letter concede that the contract start date is 1 June and the arrival date is later in June.  Furthermore, anyone with a contract start date of 1 June is prima facie eligible for inclusion in PBRF; coupled with the fact that A’s CV is impressive, and that the University has a strong interest in its PBRF performance, one would further strongly suspect on this basis alone that A had been submitted for PBRF.

As you will see from my email, I refer to an additional case to that of A, and I have heard rumours of other such cases.  Quite apart from my actions, any activities of this kind that are occurring are very likely to be detected by TEC’s auditors anyway.  As a former auditor, I can confidently state that auditors (reasonably) take a considerable interest in potentially manipulative dating of transactions close to accounting census dates.  It would therefore be basic audit practice in the present situation to draw up a list of staff being submitted for PBRF whose contract start dates were on or within the few weeks  preceding June 14 (as per a review of the contracts) and who arrived after June 14 (as per an itinerary or air ticket).  Furthermore, TEC’s earlier public rebuke of universities for attempting to manipulate PBRF scores by hiding ‘R’ staff would have made their auditors even more likely to suspect that universities would attempt to manipulate the contract start dates of staff arriving just after 14 June.  So, if such actions are occurring, it would be naïve for University management to imagine that TEC’s auditors will not detect them (just as it was naïve for management to believe that all of the earlier activities concerning ‘R’ researchers that were detected by TEC would not be detected).  It is also highly damaging to the reputations of every staff member who becomes an unwitting participant in improper activities because there will be at least the suspicion that they perfectly understood what was happening (particularly if they are an expert in XX).

Despite all of this, my offer of October 19 to leave aside past events so long as a satisfactory process to deal with future concerns is implemented still stands.  I think this would be a much better outcome than an ongoing flow of accusations followed by a further public rebuke of universities by the TEC.

I would like to finish with some quotations from University management:

An employee’s obligations include “..a positive duty to draw to your employer’s attention the improper actions of another employee.” (Prof Buckle, Oct 16 letter).

We require ethical behaviour in all our activities and interactions” (from the University’s Mission Statement at http://www.victoria.ac.nz/home/about/working/missionandvalues).


Martin Lally

The issue is a pretty simple one. Does VUW arrange its employment contracts to state someone has started employment earlier than they actually have, in order to gain more funding via the PBRF?

Hopefully someone who can get answers to those questions, is asking them.

Tags: ,

More on PBRF

March 26th, 2012 at 7:22 pm by David Farrar

An update to my earlier post.

  1. The total mount of PBRF funding is $250m a year, not $500m. I did take my figure from the Treasury Appropriations, but obviously misinterpreted it.
  2. PBRF funding is itself based on actual research done, not the level of research per average staff member, so the tactics used by VUW will not in itself lead to any increase in funding.
  3. It does however allow them to be placed higher up the comparative table with other universities. This is designed to increase their reputation, attract more staff, and possibly win contests with other universities for new schools, if any eventuate. Ironically I think their reputation gets lowered, not increased, by their attempt to artificially boost their average.
  4.  Considering that they don’t actually gain any extra government funding from it (which I wasn’t clear about in the earlier post), I think it is ridiculous they spend so much time and energy trying to skew their averages, so people are fooled into thinking they are better ranked than they are.
  5. I understand that the tertiary education union has been concerned about this for sometime. Sadly their solution seems to be common to education unions – wanting the research data kept confidential! But they do support an investigation, which is good.

There’s been a number of releases in response to my blog post. They are:

  • VUW, saying their independent investigation found no grounds for the allegations
  • The TEU, backing the call for an independent investigation over the sector
  • Labour saying there should be a review of the system
  • Greens saying the system is at fault, not Vic Uni

It is worth noting that the PBRF was introduced by the former Labour Government, as far as I can tell.

Tags: , , ,