Reader Beware

Reader Beware

A superintendent I know once heard somebody at a party discussing the latest study on reaching at-risk children. Although the findings didn’t exactly jibe with his first-hand knowledge, he was so eager to improve this part of his district that he immediately began revamping some policies.

Unfortunately, the storyteller was missing a few vital pieces of information and the superintendent’s changes exacerbated an already problematic area.

Even on such a grand scale, with such respected people involved, mistakes can happen.

OK, so this might seem like a stretch to many of you, and it is. But I made up this example to prove a point. It’s rare that people in the media ever cast doubt on any of the stories that they report, but the reality is that those of us who put together this magazine each month are somewhat at the mercy of our sources. If someone or some agency reports new findings, the best we can do is be sure those statistics are accurate, question how a certain study was done, and ask other people in the field for comments.

I bring all of this up because of the recent story in The New York Times about Harvard Professor Paul Peterson. In the middle of the 2000 presidential race, Peterson made a splash by coming out with a study that, he said, showed that vouchers significantly improved the test scores of black children.

Three weeks later, Peterson’s partner in the study, New Jersey-based Mathematica Policy Research, came out with its own interpretation of the research. The group did find some gains by a specific group of blacks, but stopped well short of a sweeping generalization about the effectiveness of vouchers.

After the discrepancy, Mathematica opened the study’s entire database for outside review. A new analysis of the stats showed that the tremendous gains made by African-American students in the sixth grade were attributed, in part, to an error in the way race was calculated in the original report. When that mistake was corrected, the gains disappeared.

The lesson here is clear. Even on such a grand scale, with such respected people involved, mistakes can happen. DISTRICT ADMINISTRATION has been increasing the number of reports cited and we’ve even created two new pages, Research Corner and By the Numbers. To help administrators delve deeper into the topics covered on these pages, the Web versions will include more detailed information on our sources.

Wayne D’Orio, Editorial Director


Advertisement