Wednesday, December 4, 2013

Our PISA Results. Good? Bad? Does it really matter?

The latest PISA results are out and the ratings on Canada's performance range from “This is on the scale of a national emergency,”  bad to "We didn't improve at all, but we didn't slide much" bad.  In others the reaction seemed confused, for instance  as Jeffrey Simpson of the Globe and Mail says here "Before everyone goes off half-cocked, as a few commentators already have, the OECD report contains both good news and bad." Yet later in the same commentary he claims that, "Given this need (to think about what is going on and make improvements) everyone in British Columbia should grab pitchforks and stick them in the educational “reform” proposals now on the table."  If throwing out years of work toward improving an already strong system due to results of one international test doesn't constitute "going off half cocked" I'm not sure what would.

Of course rating Canada as a single jurisdiction is pretty much meaningless anyway since education is strictly a provincial responsibility.  But whether it's hand wringing because of mediocre to poor results or jubilation due to having made 'great strides' the question still remains "So, what does it all mean?" The answer, as an increasingly large number of commentators are saying, is a resounding "Well, not much really," as Diane Ravitch outlines here.  Ms. Ravitch points out that the US has been doing quite poorly on these types of tests at the senior grades ever since they were started in the 60s.  Claims to be "slipping" are disingenuous in that American kids have never really done very well on international standardized tests, particularly at math.  Nobody would argue however that the US has been anything short of an economic juggernaut over those same 50 years.  It's also been a world leader in creativity, innovation and productivity over that same period.  According to Keith Baker in this 2007 PDK article, the predictive value (according to most 'quality of life' type indices) of any country's results on these tests is essentially zero.

That's not to say that these tests have no uses.  They do provide a very general indication of how our education systems are performing, at least in so much as how well we're preparing kids for tests.  But more importantly (and this is something that 'mean' measures don't address) they shine a light on differences in academic achievement across socio economic boundaries.  In other words, it's not the norm that's important it's the deviations from the norm that are a better indication of how we're doing.  In this respect Canada does very well, with fewer exceptions than most countries, rich and poor alike have access to quality education in this country.

To the kid in grade 9, tomorrow's math test can mean everything…as adults we know it doesn't.  To a kid in grade 12, those final marks can mean everything…as adults we know they don't.  To the Fraser Institute, FSA and other "ranking tools" can mean everything…as adults (I hope) we know they don't.  To countries (or provinces in the case of Canada) why does PISA seem to mean so much?…As adults why can't we believe that it doesn't?