PISA overestimates achievement

I have been reading the 2006 PISA  data to see if Chuck is right and there is some useful information to be found in the European data on African immigrants.  I now have less trust in PISA.

No Jamaicans:  The PISA does not collect race so you have to infer ancestry from a question asking their parents' place of birth.  It turns out that in England their are under 40 people out of 11000 with parents from Africa and only one with a Caribbean parent.  This could be because the students the students categorized themselves as from "other countries" or the sample could have just missed them.  I suspect the former but to investigate if PISA could have missed the Jamaicans I perused the fascinating PISA technical manual which reminds all of the sad state of sociology data

Non-response:  The PISA is a well designed two stage sampling study where schools are randomly sampled within the country and students within the schools.  The survey was run by  the beltway bandit seasoned government contractor Westat.  But no matter how well designed or executed  any study can be compromised if students do not take the test and there is a lot of that in the PISA.

No Special Ed Kids: All schools are allowed to ignore 2.5% of students who either do not speak the local language or have disabilities and a school only needs to have 85% of the sampled 15 year old students take the test to be included.  If they provide less than 85% of the sampled students the school gets replaced by another school although the under sampled school's data is included if the rate is over 50%.  The US school response for both the primary and replacement schools was less than 80.  To deal with poor response rates the Westat up -weights schools in the sample with similar characteristics (e.g., demographics in the US, school level GCSE scores in the UK) as non-responsive schools. This might work but they can't account for the high probability that the dumb kids of all races tend to skip the test.  I predict that in countries like the US, UK, and Scotland that the stated overall PISA's scores are an over estimate of students ability.  In Countries where a large number of 15 year old students are out of school (looking at you Mexico) are probably worse.

Shenanigans:  Schools could also game the system by making sure the dimmer of the sampled students stay home. There is little incentive in the US since the PISA is not a big deal. In England, however, drops in the PISA "league tables" can actually lead to policy changes so the local education establishment has an incentive to keep the dim kids at home.