Lumley on League Tables

July 3rd, 2012 at 1:00 pm by David Farrar

Thomas Lumley at Stats Chat blogs:

work well in sports.  The way the competition is defined means that ‘games won’ really is the dominant factor in ordering teams,  it matters who is at the top, and people don’t try to use the table for inappropriate purposes such as deciding which team to support.  For schools and hospitals, not so much.

The main problems with league tables for schools (as proposed in NZ) or hospitals (as implemented in the UK) are, first, that a ranking requires you to choose a way of collapsing multidimensional information into a rank, and second, that there is usually massive uncertainty in the ranking, which is hard to convey.   There doesn’t have to be one school in NZ that is better than all the others, but there does have to be one school at the top of the table.  …

This isn’t to say that school performance data shouldn’t be used.  Reporting back to schools how they are doing, and how it compares to other similar schools, is valuable.  …

While it’s easy to see why teachers might be suspicious of the government’s intentions, the rationale given by John Key for exploring some form of official league table is sensible.  It’s definitely better not to have a simple ranking, and it might arguably be better not to have a set of official comparative reports, but the data are available under the Official Information Act.  The media may currently be shocked and appalled at the idea of league tables, but does anyone really believe this would stop a plague of incomplete, badly-analyzed, sensationally-reported exposés of “New Zealand’s Worst Schools!!”?

The data that I think would be most valuable is the “value add” over time for each school, moderatd by decile.

Tags:

8 Responses to “Lumley on League Tables”

  1. rouppe (962 comments) says:

    Not to mention that several of my public examinations came back not with a mark out of 100, but a grade score.

    League tables could be flattened into A, B, C, D, E and F.

    C and D would be flirting with barely average, E would be “needs to apply themselves more” and F “when is the administrator arriving?”

    When I recently saw the pass rates for National Standards I saw quite a few schools around where I live with as low as 45% pass rate. This is quite appalling and clearly those schools need attention from MinEd

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  2. BeaB (2,106 comments) says:

    The ERO report shows primary school principals and staff are out of their depth when it comes to valid assessments.
    We have a choice of excellent NZ standardised tests (eg ASTLE, PAT) that schools have used for years. They should have to send in their results year by year and the statistics per level, per school (and nationally) then collated in a publishable form by professionals who know what they are doing. And who can show year by year comparisons and progress – or lack of it.
    The ERO report shows a lack of leadership and knowledge so let experts deal with the data and let us, the paying public, see how primary schools are doing. Good or bad, it’s better than trying to keep it all secret in the hope that parents are all fools.
    The sky hasn’t fallen in on high schools who have had years of being ranked publicly on their results.
    I think NZEI are opposing league tables because they know primary schools aren’t up to the scrutiny.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  3. mikenmild (11,247 comments) says:

    ‘We have a choice of excellent NZ standardised tests (eg ASTLE, PAT) that schools have used for years.’
    So why does the government want performance information based on the nonsensical national standards?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  4. BeaB (2,106 comments) says:

    The reason is results are kept secret so parents and taxpayers have no data to compare.
    Schools were asked to use their chosen test (ASTLE, PAT etc) to measure kids against the national standards – a simple thing to do. Not necessarily accurate to the nth degree but clear enough to give a useful indication of how kids are progressing.
    It is a benign process, not at all like the more draconian standardised testing in other countries though the unions would like us to believe that is the case.
    I think they were idiots to complain as now we think they are hiding something, ERO has found many are incompetent in assessment and the Government thinks there should be league tables which will mean much more rigour will be needed.
    Schools brought it on themselves by being so intransigent and secretive but I think the outcome will be much better for everyone – in the end.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  5. Maaik (33 comments) says:

    He has half a point with the collapsing of multidimensional information into a single rank. But then he demonstrates a lack of imagination when he says that “there has to be a single school at the top of the table”.

    I can’t see parents sending their kids to a school on the other side of the city if that school is 0.5% ahead of the local school. Some would though. What I would like to see is the schools simply ranked by decile (computed from the combined data by a “secret” formula) – so the local school will still be in the same decile as the one on the other side of the city (unless they straddle the divide), but parents can see that they should not send their kids to the low-decile local school, but they can get to the 3-deciles-above school in the next suburb.

    Some parents will still calculate the scores themselves, using the official formula or even one of their own (e.g. “we weighted science more than the official formula, and cultural activities less”), and come up with their own preferences, but most will simply see what they have always suspected – that the school known for its pot-smoking bullies and hoodie-wearing teenage dropouts has a lower score than the school that does not have these obvious signs. They might be surprised by some of the scores as well – a poor neighbourhood is not always a sign of a poor school – but at least they will have an objective basis to make their decisions.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  6. Sofia (852 comments) says:

    It seemed all along that the media were those enthusiastic to have league tables, even create them from OIA questions, to the point the PM suggests ‘official’ tables would be better, but Thomas Lumley says “The media may currently be shocked and appalled at the idea of league tables …”
    What?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  7. Paulus (2,602 comments) says:

    A large number of schools are deliberately stuffing the reporting of National Standards to create confusion.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  8. mpledger (425 comments) says:

    Thomas Lumley said
    “and second, that there is usually massive uncertainty in the ranking, which is hard to convey. ”

    which is something I mentionated ages ago.

    The raw score needs to be published as well as the ranking. The difference between the school ranked 1 and the school ranked 50th may only be 1% (on a scale of 0-100%) but a differenence of 49 places implies a huge difference.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote

Leave a Reply

You must be logged in to post a comment.