Good data analysis

September 25th, 2012 at 12:00 pm by David Farrar

Teacher unions fear assessment data being released because they worry about league tables published by media. And look I agree a league table which takes account of no control factors is not very helpful.

But what has excited me about the data being released, is what some real data experts can do with it (talking of data experts the Herald editorial moaned about ” A high priesthood of data analysis bemoans news media interest” which has caused me to label Keith Ng as Cardinal Keith!). An example is Luis Apiolaza at Quantum Forest. First he did the standard average proportion of students meeting the reading standard at each decile.

So you look at that, and think wow it is all about decile. But he then looks at the variance in each decile, not just the average.

The box shows the middle 50% for each decile, and the line in 1.5 times that interquartle range. What this shows is that the lower decile schools may have a lower average, they have much more variability. This is good, because it dispels some of the myth that all low decile schools have few students meet the national standard.

What I think this data may allow, is to then look at that huge variance in decile 1 to 3 schools, and identify the factors that have some schools higher than others. Eric Crampton has tweeted he has started some analysis and ethnicity is a big factor.

Of course this is all just a snapshot of data, and the lack of moderation means you take care with it (but does not make the data useless by any means). Th real value will be over time, as we get trend information.

I suspect there is going to be lot of sites doing their own analysis of the data.

Tags:

31 Responses to “Good data analysis”

  1. Hamnida (905 comments) says:

    Wow, poor kids are poor and rich kids are rich – whatever next.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  2. Graeme Edgeler (3,261 comments) says:

    Part of the reason for greater variation in lower deciles is that there are many fewer students attending lower decile schools.

    A decile is 10% of schools, not 10% of students, and higher decile schools have, on average, more students.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  3. tristanb (1,133 comments) says:

    I know it has to fit easily on a page, but having the y-axes starting at >0 is a bit misleading.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  4. Hamnida (905 comments) says:

    Even Hekia has worked it out:

    http://www.stuff.co.nz/dominion-post/comment/tom-scott-cartoons

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  5. Hamnida (905 comments) says:

    I doubt a Tory would ever manipulate the y axis of graph for political gain.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  6. Ross12 (1,131 comments) says:

    DPF what would worth finding out is what were the test methods or assessment methods used. Were they all the same ? If not are there moves to make them all the same ? ( NB Unlike Hamnida . I am not against the standards )

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  7. Bevan (3,965 comments) says:

    Wow, poor kids are poor and rich kids are rich – whatever next.

    Help the poor kids leave school with the best chance possible to succeed in the future. Who knows, maybe if we get it right – their kids wont be classed as poor when they go to school.

    Fat chance the way you and your lot carry on though, I mean why would you want a poor person to get ahead – probably lose their vote.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  8. Hamnida (905 comments) says:

    Bevan – Not that I am a Labour Party man, but it is my understanding their new policy is too feed poorer children so they can learn at school.

    That is a better use of taxpayer’s money than simply testing them and saying they’re dumb.

    You may have noticed Hekia and John have finally worked this out too.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  9. Bill Courtney (131 comments) says:

    Here’s my favourite so far:

    http://www.ben.geek.nz/2012/09/schools-names-starting-with-i-work/

    But be warned, it will upset the anti Island Bay right wingers on this site.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  10. RRM (9,394 comments) says:

    I notice that the top students in that box & whisker plot seem to be achieving at about the same sort of reading level, regardless of what decile school they go to.

    [I wonder how many of those kids have read "My struggle against my tory capitalist pig oppressors" by Hamnida yet? :-)]

    Given what we saw on Campbell Live last week about poor kids not getting fed by their dead-beat parents, it seems like a bit of a triumph (IMHO) that even the decile 1 schools are reporting more than 50% of kids meeting the reading standard.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  11. Pete George (22,519 comments) says:

    Not that I am a Labour Party man, but it is my understanding their new policy is too feed poorer children so they can learn at school.

    Soon to be announced Labour policy:
    - to feed poor unemployed so they will be successful in job interviews
    - to feed poor workers so they are more productive at work and will earn higher wages
    - to feed poor pregnant women so they will have healthier babies
    - to feed poor MPs so they will perform better in policy development

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  12. Adolf Fiinkensein (2,775 comments) says:

    Eric Crampton will find what people close to reality have known for decades.

    Race is a dominant contributor.

    You won’t get any budding Einsteins or Mozarts coming out of households which are immersed in Maori gangland or are generational welfare beneficiaries, no matter how good the school. Sadly, too many Paori and PI people are represented here.

    It has to do with the attitude of the parents and there’s not a damned thing any gummint official can do about it but wait for the next generation and hope it will be more responsible.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  13. Hamnida (905 comments) says:

    Good to see some half decent policy ideas coming from Colin Craig’s Conservatives, although I’m sure on $140,000 plus a year MPs don’t need assistance.

    Could be some wasted votes when the Conservatives get 4.2% with no electorate MP.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  14. Lloyd (125 comments) says:

    Just a reminder, from the Ministry of Education’s site:
    “A decile is a 10% grouping, there are ten deciles and around 10% of schools are in each decile. A school’s decile rating indicates the extent to which it draws its students from low socio-economic communities. Decile 1 schools are the 10% of schools with the highest proportion of students from low socio-economic communities, whereas decile 10 schools are the 10% of schools with the lowest proportion of these students.
    The lower a school’s decile rating, the more funding it gets. The increased funding given to lower decile schools is to provide additional resources to support their students’ learning needs. A decile does not indicate the overall socio-economic mix of the students attending a school or measure the standard of education delivered at a school.”

    “The lower a school’s decile rating, the more funding it gets.” i.e. students at a low decile school are funded at a far higher level than students at a decile 10. Suggesting that the government leaves inequality unchecked is a myth.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  15. Kevin (1,122 comments) says:

    AF what a knee jerk load of bollocks. Sometimes I’m embarrassed to be right leaning. He will find that people who IDENTIFY as Maori and PI are over-represented. The whole thing is an Iwi scam. My maori children are doing just fine, thanks.

    But because of the Iwi/media/charitble trust axis saturation campaign on how negative it is to be Maori or PI, those that succeed often self identify as European or NZer while those that fail, go to jail or are otherwise no hoping will self identify as Maori.

    Iwi sycophantic Epidemiologist and social scientist have got it sussed though because even if Maori are postively over-represented there must be something you can find.

    Maori are over-represented in the All blacks but they are still underpriveledged because they die younger…

    What a load of bellding crap like that putrifying line up of trough feederts on Campbell live last night.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  16. Dave Stringer (182 comments) says:

    Catherine Delahunty suggests on frogblog that only people with cometition as a “mantra” believe schools’ achievements should be reported and compared.

    Competition is not a mantra, it is a natural state of humanity. Adults compete for jobs, when children are conditioned to a total absence of competition leaving school is a great culture shock. I know because my brother-in-law went through that shock.

    Having got a good “COMPREHENSIVE” education in England, he applied for a job and got a rejection letter saying that a number of new school-leavers had better qualifications for the position than he did. After some encouragement, he aplied for four other jobs, all on the same day; the result was four rejection letters all saying essentially the same thing. He didn’t apply for another job for 16 years, the shock of “not being good enough” (his words) dropped him into depression and he lived off the state.

    This is NOT the type of citizens and residents we need to be developing. Let everyone learn, at school, their strengths and weaknesses compared to others so that their expectations on leaving school are reasonable based on their inherent ability. We are not all the same, nor are we all equal as human beings; if we were there would be no Mother Theresas, no All Blacks, no outstanding scientists, or entrepeneurs, or doctors or judges or members of parliament. It is mankinds differences which make him top of the food chain, and we should celebrate them rather than pretend they do not exist.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  17. Kevin (1,122 comments) says:

    Two things. First that definition of decile gives unlimited room to scam the system so the sociologists and epidemiologist can get the correlations they want – which is always (1) I have a theory – poor and maori are “deprived”, (2) the government should put more resources into this problem, (3) the government should pay me to do more research on this problem.

    Secondly the state sponsored ineqity that errr you work hard, earn more, encourage your kids at school so they are not entitled to as much of the countries resources.

    Great way to run a modern democracy and try to keep the people here who we need to pay for all this crap.

    But the politicians and Campbell etc dont give a rats because as long as they can draw a fat salary who cares about increasing the pie.

    Oh wait, its called the 3rd world.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  18. Bevan (3,965 comments) says:

    That is a better use of taxpayer’s money than simply testing them and saying they’re dumb.

    Wow, you must have failed dismally in comprehension if you think that is the policy.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  19. PaulL (5,828 comments) says:

    @Ham, do deal with some of your stupidity:
    – No, it doesn’t tell us that rich kids always do well. If it did, that would tell us that the only hope of equality is to redistribute income. I know you’d like that to be the case, but it isn’t what the graph says. You might as well infer that it tells us that stupid people are poor. Which is also not (entirely) true.
    – Cutting the Y axis at greater than zero is to DPF’s political disadvantage. You’re just being rude there, it was clearly so that it fit on the damn page
    – Key pushed for feeding poor school kids over the objection of Labour. Refer http://www.kiwiblog.co.nz/2012/09/who_started_food_in_schools.html. You’re wrong again.

    @RRM: this is telling us the best _schools_ in low decile areas achieve about the same, not the best students. Which is much more interesting than just observing that every school has a few bright students. It’s telling us that even when you have a poor catchment area it’s possible to have a school that’s doing well.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  20. DylanReeve (179 comments) says:

    Frankly can’t be bothered wading into the statistics debate but Adolf Fiinkensein said something above that’s a pretty commonly held belief – and one that’s really hard to get past. In fact there probably an element of truth to it – that being raised in a less than ideal family environment, especially in poverty, is basically a death sentence in terms of getting a good education.

    It’s obvious to us all that a bad attitude, no matter how good the school programs or teachers, will end in a poor result. So how do you deal with that?

    There was a fantastic This American Life episode last week that dealt with exactly this issue and I highly recommend it to anyone who has been thinking about this stuff.
    http://www.thisamericanlife.org/radio-archives/episode/474/back-to-school

    It boils down to the idea of non-cognitive skills – basically attitude, temperament, social skills etc… They tend to be underdeveloped among kids from difficult family environments and especially those who deal with a lot of stress in the home . But they are also very pliable – they can be taught very easily and they have a MASSIVE affect on the ability to perform in school (and society).

    It’s about an hour long – listen to it if you can find the time, it’s fascinating and very different to most of the stuff you hear about education.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  21. Hamnida (905 comments) says:

    Sorry, silly me, I forgot – Torries just want to help poor kids.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  22. tom hunter (4,300 comments) says:

    but it is my understanding their new policy is too feed poorer children so they can learn at school.

    Key parts here:
    – understanding
    – school
    – learn
    -too feed

    Today’s lesson has been bought to you by Hamnida, @TheStupidItBurns

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  23. Mobile Michael (405 comments) says:

    The top performers are pretty stable across the deciles – is it the student or the teacher (or both)?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  24. PaulL (5,828 comments) says:

    @mobile michael: the measures are of schools. The top performers are top performing schools, not top performing students.

    I’d hazard that the top performing schools are examples of schools that are doing all the things you’d expect in a good school, and they get good results pretty much independently of the students.

    The long tail suggests to me that schools that aren’t doing things well are much more susceptible to the quality of their students.

    And, since I’m a good right-wing type, I believe that this is a classic case where markets and choice help. If that is really the profile, then if school vouchers existed and parents sent their kids to schools that performed well, then those schools at the bottom of each decile would basically close down. They’d be replaced by new schools, and those schools presumably would strive to do better than the ones they replaced.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  25. Mobile Michael (405 comments) says:

    @PaulL – thanks for that.

    If that is the case, why aren’t we praising the top performing low decile schools and finding what it is that makes them good. If we can bottle it, we’d solve our problems for good.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  26. KH (687 comments) says:

    This is just the debate we should be having. Teachers want us not to have data. They, who should be setting a better example, are proving what is the matter with them.
    A thing about data, is that as you look at it, it creates another question. Data can answer a narrow question, but as well it’s more useful than that. I love data, because as you look at it, and ask the next obvious question, you become much more informed about the subject.
    And to explain some of the understanding you gain,iyou need to specify some of the ifs, but and maybe.
    You need a little IQ, which teachers do not want to let us use. They prefer to dumb down, which is a worry as they are supposed to be educating our kids.
    A recent example: I kid you not and this is astonishing. Because ‘the public’ might misunderstand, the ministry is not publishing what decile each school is. Well some idiot will misunderstand. But thats not a reason to keep who is who a secret. I would call such a move anti -educational

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  27. Bob R (1,328 comments) says:

    Harvard economist Greg Mankiw explains the correlation between academic results and socio-economist status.

    1. People vary in their innate talents, as measured by, say, IQ tests.

    2. More talented people tend to earn higher incomes.

    3. More talented people tend to have more talented biological children–that is, talent is partially heritable.

    4. As a logical implication of the above three points, the raw correlation of kids’ SAT scores and family income conflates the true effects of family income with the biological transmission of talent.

    http://gregmankiw.blogspot.co.nz/2009/08/and-i-thought-i-was-being-boring.html

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  28. Bob R (1,328 comments) says:

    edit: socio-economic, not socio-economist :)

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  29. slijmbal (1,204 comments) says:

    tristanb (907) Says:
    September 25th, 2012 at 12:10 pm

    I know it has to fit easily on a page, but having the y-axes starting at >0 is a bit misleading.

    If there were lots at the zero range yes – but there arent ditto such graphs are only misleading if the range of values is a small factor of the overall measurement – neither case counts here. Also where one is measuring the variance from a common target it is common practise to graph from a non -zero base.

    The significant correlation implies a common cause. Could be high decile schools fudging figures to be fair but common sense suggests not – could be smarter parents avoiding lower decile schools – it could be that being with the right peers, having parents who put the effort in etc. It does show that the wisdom of the crowds in avoiding low deciile schools despite what the teachers say that it is a poor predictor – it looks like a great predictor.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  30. Kevin (1,122 comments) says:

    Studies in the states have shown a much greater correlation between “success”, academic, income, financial etc and parents capital not parents income. If the lefties really want to address the so called problem they need to look at parental capital, not parental income.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  31. PaulL (5,828 comments) says:

    Kevin – very true. I recall a great podcast I listened to in Australia from a guy who used to run a thinktank over here (CIS?) and was leaving. He was talking about his time. One of his biggest achievements was to get the Aus Bureau of Statistics to stop reporting income in the bands they were by demonstrating that a large proportion of people in the bottom band of income had quite high spending habits. In short, they were people with a taxable income near zero, but quite large assets (actually, some of them were drug dealers too, who also had a lot of spending but little taxable income).

    Bottom line, you couldn’t generalise from their income to their lifestyle or prospects. Whilst we progressively tax income, all the philosophical arguments that lead to progressive taxation are based on wealth, not income. And (taxable) income is a poor proxy for wealth. Which is one of the reasons I’m a strong supporter of a land tax – since property seems to be something that a lot of wealthy people have, it’s kind of hard to dodge paying it through tax fiddles, and you can’t take it overseas. Other forms of wealth are harder to tax (more mobile), but it’s a start.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote

Leave a Reply

You must be logged in to post a comment.