Salmond on education data

October 26th, 2011 at 10:00 am by David Farrar

First John Pagani writes a post on I agree with, and now does a post on assessment data which I also largely agree with.  Rob blogs:

The Los Angeles Times has produced a detailed set of estimates about how much value each teacher in Los Angeles adds to their classroom. That is hugely valuable information. New Zealand’s establishment should be doing something similar.

I blogged last year on the remarkable data published by the LA Times. It basically measures the effectiveness of individual teachers.

Why should we follow the Times’ lead? Because it helps us to reward great teachers and provide remedial support for teachers in difficulty. And because it allows us to diagnose, early, easily, and with reasonable precision, what is going wrong when a school is performing badly. Is it one or two bad teachers? A bad english department? Poor school-wide leadership? Or is the issue in the community itself, a problem at home rather than in the classroom? The data can answer that crucial question better than a big round of finger-pointing in front of an inspector from ERO.

We can do all kinds of helpful things with this information. If one school has a dysfunctional maths department and there is a great maths teacher at another school, the government can fund the Board of Trustees to pay generous incentives to convince the great teacher to take on the troubled department as HoD. Same thing for giving great teachers powerful incentives to teach at generally underperforming schools.

Absolutely agree.

It is true that there are already multiple ways to assess teachers in New Zealand. There is teacher registration. There are periodic assessments against professional standards. In some situations, there are Teacher’s Council investigations. There is ERO. Those are all good things to have, and this data-driven assessment should be used to extend those assessment regimes, not to replace them. The data based assessment does add real value, however, both as a nationwide diagnostic tool for educators and administrators and as an individual assessment tool for rewarding great teachers and helping others improve.

True. But with teacher unions so against even allowing data on schools to be collated and analysed, I can only imagine how far they would go to stop what Rob proposes.

Who should find out the results? Well, the teachers for a start. They need to know how they are doing. And their local Board of Trustees. And the government folk should know, too. They are collectively charged with improving the educational outcomes for New Zealand’s tragically long “education tail.” Once they know how their teaching resources are distributed, they can better shuffle them around to make the system more effective.

Which is of course what the Government is trying to do with national standards, as well as give parents better information.

Parents should probably get some information about how their kid’s school does compared to other schools with similar student demographics. That is a valuable accountability mechanism for Principals, who get paid good money to be accountable to their local communities. But unfiltered league tables of area schools do more harm than good, presenting an apples to oranges comparison as if it were apples to apples.

The answer to bad league tables is good league tables. Not banning league tables.

Parents should also not get access to individual teacher rankings. Here I disagree with the Times. Why? Because it is little more than a recipe for school administrators to be drowned in a tide of the pushiest, over-caffeinated parents demanding that Little Johnny should move over to that excellent Mrs Paki’s home room. Now! We don’t get to see the latest performance review of the cop that pulled us over, or the nurse in the hospital ward, or the customs agent at the border. And rightly so. Teachers are no different.

I’m okay with parents not seeing results of individual teachers, so long as School Boards and the Government does.

Rob also says in his comments:

Secondary teachers with a BA and a teaching diploma start at $47k and can earn up to $71k at current scales, even without any of the additional salary Units under the control of Boards of Trustees. The top of their base salary scale is more pay than 90% of New Zealand adults recieve, according to IRD data. I think **great teachers** should receive substantially more compensation than this, but I do not think **all teachers** should get a big raise.

Again I agree. I’d love School Boards and Principals to have the ability to have .

Tags: , , ,

19 Responses to “Salmond on education data”

  1. KH (687 comments) says:

    Two and half hours and no comments. That’s interesting but genuinely I am unsure of what it means.
    As for seeing the results of individual nurses. Why not. That can be the difference to your whole future.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  2. expat (4,048 comments) says:

    If only we had data to measure student achievement and teacher performance. NZEI especially not keen on letting this happen.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  3. Lucia Maria (2,207 comments) says:

    KH,

    It just gets tedious. Now, back to Classical Writing for the afternoon …

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  4. BeaB (2,057 comments) says:

    Performance pay would be a nightmare and good teachers already get rewards from astute principals. You look after your best staff.
    The criteria the Ministry of Education would dream up for performance are too ghastly to contemplate and would eat up time much better spent on the kids.

    How about attracting the best graduates in with high entry pay? Or a hefty bonus after five years to help stop them leaving in those first difficult years? Or allowing schools to sign over a cohort of kids to a group of teachers on a performance contract within the school – eg literacy levels, NCEA results etc etc. Teachers are great individualists and many would relish the challenge.
    Naming and shaming only strengthens the teaching unions.

    A bit of generosity, some real carrots and some clear, high standards might work much much better than the tired old right-wing mantras of vouchers, performance pay, league tables etc etc.

    [DPF: I would never dream of having the Ministry of Education decide performance pay criteria.

    I would just allow each Board and Principal to pay the teachers at their school, however much they think they are worth, within their overall budget.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  5. mpledger (429 comments) says:

    DPF said
    I’m okay with parents not seeing results of individual teachers, so long as School Boards and the Government does.

    ~~
    As soon as the government gets the data then it’s public information and anyone can request it.

    ~~
    DPF quoted
    The Los Angeles Times has produced a detailed set of estimates about how much value each teacher in Los Angeles adds to their classroom.
    ~~~

    It’s how much the teacher added to a multi-choice test in maths and English. That’s very different to what a teacher adds overall to their classroom.

    And teacher’s can add value by
    a) “drill and kill” – emphasising memorisation rather than understanding
    b) teach to the test – there are only certain types of question you can ask on a multichoise test and they tend to get repeated
    c) spend more and more time on the subjects that get tested rather than other areas especially PE, music and exploratory science.

    None of these things make for good teaching or for keeping kids engaged in learning.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  6. Rob Salmond (260 comments) says:

    @BeaB The performance criteria is very simple, and does not distract at all from time spend with students. The performance criteria is: do your students learn better with you than they would with the average teacher? And assessment of that criteria is entirely unobtrusive.

    Nobody is proposing any form of naming or shaming. The idea is to use the data we already have to help assess teacher performance. The problem with having performance done entirely informally and within a particular school’s structure is that it hurts teacher mobility. A great teacher is only known to be a great teacher by one Principal / BoT. This system would allow a teacher’s quality to be credibly known to other schools as well, including underperforming schools armed with extra money to recruit great teachers.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  7. Rob Salmond (260 comments) says:

    @ mpledger

    The government does not have to release all information is possesses. It can withhold information on the grounds of individual privacy, for example, which is why nurse and doctor performance evaluations are not in the newspaper. The same exception would hold for teachers.

    It is true that a system which assessed teachers **only** against these criteria would be open to abuse in some of the ways you suggest. But the suggestion is to add this metric to the other metrics already in use, many of which are well placed to guard against the types of abuses you list.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  8. mpledger (429 comments) says:

    Rob Salmond said
    But the suggestion is to add this metric to the other metrics already in use, many of which are well placed to guard against the types of abuses you list.

    ~~~~~~~~
    What are these metrics already in use?
    And how are they well placed to guard against these abuses?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  9. mpledger (429 comments) says:

    Which reminds me …
    the LA teacher data was supposed to be anonymous but the LA Times were able to work out who was who and then they published it.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  10. Rob Salmond (260 comments) says:

    @mpledger

    1. You asked about the other mechanisms for assuring quality in teaching that are already in place. I listed some in my original post. They include initial teacher registration, regular assessment of teachers by their school’s leadership, assessment of the school and its teachers by ERO, among others. These inquiries are well placed to find out which teachers are gaming the system because they are qualitative, in depth, and concerned with ensuring fairness for all teachers across the school or country. Every teacher who games this kind of system succeeds only in making all other teachers look, artificially, a little bit worse. Realizing that dynamic, I am confident that the rest of the education assessment apparatus can ensure the integrity of results-based assessment.

    2. You are wrong about the LA data. The LA times generated the data themselves because the LA Unified School District did not want to do it for themselves. There was never any breach of previously private data by the LA Times. See http://projects.latimes.com/value-added/faq/#private_information Remember, their California Public Records Act is not the same as our Official Information Act, which would protect individual teacher performance reviews from broad public scrutiny just as it does all other public workers in New Zealand.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  11. mpledger (429 comments) says:

    The LA Times hired a statistician to analyse the data. It was his understanding that the teachers would not be named. That’s standard practise amongst statisticians (of which I am one). It was after the analyses were done that the newspaper connected his results with teacher names and published them. (I suspect the statistician and the RAND corporation learnt some unwanted lessons out of that fiasco.)

    There are lots of ways teachers have cheated the system in the States…
    - changing the answers on test papers (“rubbing out parties”)
    - unsealing the tests packets in the days prior to the exam to make study notes for the students and then resealing them
    - giving away the answers as the test is in progress

    A lot of these cheats went undetected because management were sometimes complicit and state education officials wanted to believe the results because it suited their agenda even when the results were obviously uncredible. And the reason some teachers were found out was because, when pressed, the teacher admitted it.

    Unless the ERO, school leadership and BOT are present when the cheating occurrs then how are they to stop it? The ERO only comes around once every three or fours years so how do they tell if they are seeing “normal practice” or the teacher has adjusted his/her performance for their inspection.

    Rob Salmond said
    Realizing that dynamic, I am confident that the rest of the education assessment apparatus can ensure the integrity of results-based assessment.
    ~~~

    Yea, that’s an easy thing to say because there is nothing holding you to account if you are wrong, if what you suggest has negative consequences that hinder academic achievement.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  12. mpledger (429 comments) says:

    As it said in the FAQ you directed me to

    What are the limitations of value-added?(Back to top)
    It is based on standardized tests, which many teachers and others consider a flawed and narrow gauge of learning. In California, the tests are not given to students in kindergarten or first grade, so scores are not available for those teachers. Scholars continue to debate the reliability of various statistical models used for estimating value-added, and estimates may be influenced by non-random assignment of students, students’ mobility during the school year and other factors.

    ~~
    The value added method is stupid because it rewards a teacher who keeps her kids functioning at the 1th percentile the same at the teacher keeping them functioning at the 50th percentile. Whereas it takes good teaching to keep kids performing at a high level.

    It assumes that all the added value comes from teacher input and things like taking up an instrument or starting a new hobby, parents losing their job or the kid getting cancer, have no impact on the learning capacity of children during the year.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  13. orewa1 (428 comments) says:

    Yes, principals should be empowered to award performance pay. No, school boards should not – this would undermine the most fundamental principle of accountability. Give principals the power and, if they consistently screw up over time, the Board should fire the principal.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  14. BeaB (2,057 comments) says:

    Rob Salmond
    What tosh. Performance reviews at present are carried out to Ministry requirements and involve a great deal of work.
    As for good teachers being a secret – everyone knows who and where the good teachers are and good principals devote considerable effort to poaching them.
    Good performance is presently rewarded through units that carry extra money – and higher standing.
    Most of you will be thinking about chickenfeed amounts while these units currently are worth gaining. They are directly within the control of the principal.
    My experience is that good teachers do good work whatever their pay and the very few poor ones will never improve. Shift them out and attract more good people in.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  15. Clint Heine (1,563 comments) says:

    Put it this way – if my department at work underperforms due to a couple of bad eggs, we will all be punished or put into the same basket rather than the bad eggs retrained or cut loose.

    Worse still, the unions don’t even want performances rated fullstop – this article is a slight compromise that will not hold any weight at the next PPTA meeting.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  16. Rob Salmond (260 comments) says:

    @ mpledger

    First, I do not feel the need to defend everything the LA Times did. In my post I was very clear that I think the individual teacher’s performance evaluations should not be public. Second, if you have some evidence of the RAND economist being upset at the LA Times releasing the names, I would be interested to see it. Is there a link?

    It is true that unless carefully monitored there is potential for cheating to occur. But that is true of all forms of assessment, including current teachers assessments. It is easy enough to talk about innovative learning strategies in an interview and do it once or twice during the year to generate examples, without actually being a consistently innovative teacher. This problem is by no means unique to quantitative assessment, and it can be managed pretty effectively.

    You are also right to point our that quantitative assessment has limitations, as do all forms of assessment (including current forms of teachers assessments). Using multiple methods do develop a complete picture is the best way to overcome the shortcomings in any one method. Also, as a statistician, you will know that events like getting cancer or taking up the violin are either stochastic or can often be accounted for in relation to a person’s background, which forms part of the statistical model. In either case, they will have little impact on a teacher’s overall rating, which is based on either around 60 (for primary teachers) or around 120 (for high school teachers) pieces of information per teacher per year, over multiple years.

    While it is true that quantitative assessment treats an “average” or “zero value-added” techer the same regardless of the student’s overall level of achievement, it has the flexibility to treat high value added teachers quite differently depending on where the students sit. If, for example, a teacher has a group of high performing students who achieve at the 95th percentile under almost all teachers, but one exceptional techers pushes them to the 98th percentile, that teacher will score a long way above average. Conversely, hweover, if a class hovers around the 50th percentile but has a wide teacher-to-teacher variance, then it will take larger rises in student achievement to mark the teacher out as a high achiever.

    @BeaB

    See my comment to mpledger above for why qualitative performance reviews are not the gold standard.

    I cannot speak for everyone else, but for my own part I am not talking about chiekcn feed when it comes to rewarding great teachers. Current Units are $4,000 each as I understand it. I think a 15-20% salary bonus on the basis of great teaching performance is a good thing to aim for. That is up to $14,000 extra for a great teacher.

    And your experience that great teachers will always be great teachers regardless of slary and poor teachers will never improve regardless seems boyth defeatist and silly. Teachers do respond to financial incentives – just ask their unions. Don’t kid yourself that teacher are immune to the same economic forces that act on everyone else.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  17. BeaB (2,057 comments) says:

    Sorry. Good teachers will still be good whatever they are paid. Poor teachers usually lack personality, knowledge, character or likability or all of these and all are very hard to change in an adult.
    Of course unions fight for more money but I have never heard even the PPTA claim that good teachers will teach badly if they don’t get an increase.
    I’d pay teachers double what they get now but only take the best graduates with good personal traits (like Finland) and then leave them to it.
    NZers are obsessed with trying to control the uncontrollable yet miss the glaring failure which is the failure of primary schools to teach every kid to read and write in EIGHT years. We crack that one and just about everything else will take care of itself.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  18. mpledger (429 comments) says:

    Rob Salmond Says:
    Second, if you have some evidence of the RAND economist being upset at the LA Times releasing the names, I would be interested to see it. Is there a link?

    ~~~~
    I can’t find the original source because that type of information hasn’t persisted in the way that the LA Times info has but
    http://scholasticadministrator.typepad.com/thisweekineducation/2010/08/chat-about-grading-the-teachers-readers-representative-journal-los-angeles-times.html
    “The RAND researcher has already distanced himself from the use of individual names, which he says was done after he completed his work.”

    Rob Salmod says:
    Also, as a statistician, you will know that events like getting cancer or taking up the violin are either stochastic or can often be accounted for in relation to a person’s background, which forms part of the statistical model. In either case, they will have little impact on a teacher’s overall rating, which is based on either around 60 (for primary teachers) or around 120 (for high school teachers) pieces of information per teacher per year, over multiple years.

    ~~~~~~~~~~~~~~~

    A primary school teachers has 20-30 kids of which, say, 25 kids might provide the beginning of the year and end of the year information i.e. the 2 pieces of information, neccessary to calculate value added i.e. post score – prior score. (At some schools this will be way less at they can have more than 50% attrition. )

    In such small samples, random chance (the star pupil falls over in the playground 5 mins before the big test and then does badly – i.e. the stoachastic part) can make big changes to the value added estimate.

    And how do you adjust for the father losing his job as part of the kid’s background, it’s so entertwined with socio-economic status i.e. the turnover in unskilled workers is higher than for skilled workers. But socio-economic status is also entertwined with other education factors – what schools are available to you to go to.

    Most of the time researchers won’t know information about out of school tutoring, out of school educational activities and hobbies that started during the year. The information is not collected. And even if you did collect it, the amount of information is to much to be supported when you only have 25 obs per teacher.

    ~~~~~~~~~~~~~~~~~~~~
    Rob Salmond said:
    If, for example, a teacher has a group of high performing students who achieve at the 95th percentile under almost all teachers, but one exceptional techers pushes them to the 98th percentile, that teacher will score a long way above average. Conversely, hweover, if a class hovers around the 50th percentile but has a wide teacher-to-teacher variance, then it will take larger rises in student achievement to mark the teacher out as a high achiever.

    ~~~~
    If one teacher moves the kids from the 95th to the 98th percentile the value added is 98-95 = 3%.
    If one teacher moves the kids from the 50th to the 53th percentile the value added is 53-50 = 3%

    The variance doesn’t factor into the calculation.

    But it really should. We should have confidence intervals on the VAMS because then I think people would realise how meaningless the statistic is at the teacher level.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  19. Rob Salmond (260 comments) says:

    @ BeaB You had one point, which was a repeat of your earlier point. I disagree as before, noting that your proposal to only takie great teachers with wonderful qualifications sounds super in theory but in the real world will result in a teacher shortage.

    @ mpledger You had three points:
    1. On the LA Times disclosure, I am still unsure why this is a big issue for this discussion about NZ. I oppose releasing the individual teacher ratings, as I said in my post. So does David Farrar. It is very easy to ensure that outcome through legislation. The LA Times issue does not arise.

    2. For primary school teachers, your 25 data points are actually x2 (literacy; numeracy), but grouped into 25 clusters. You are right that with those N’s stochastic error becomes an issue, which is why you run multi-year rolling averages, depressing the impact of any particular random shock. As with any statistical measure, you will not get a perfect estimate every time. But that is not the relevant yardstick. The relevant yardstick is: do we know more with this information that we would know without it? I think the answer to that is yes. For high school teachers, where the N exceeds 100 each year, that conclusion is clearer still.

    3. Value added measures do take account of stochastic variability. The correct comparison is not in terms of percentiles, but in terms of standard deviations. Using your examples:
    If one teacher moves the kids from the 95th to the 98th percentile, where the average variability is +- 1 percentile point, the value added is (98-95)/1= +3sd
    If one teacher moves the kids from the 50th to the 53th percentile, where the average variability is +-6 percentile point, the value added is (53-50)/6 = +0.5sd

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote

Leave a Reply

You must be logged in to post a comment.