Saturday, March 17, 2007

The First "Bunkum Award" for Logic-Train Derailment in Education Blogging

Given that we just started this blog, now might not be the best time to rag on blogs as a type of media. But a recent post on Edwize - NY's United Federation of Teachers blog – reminded me that sometimes the “everyone gets a voice” aspect of blogging creates legitimate venues for illegitimate ideas and analysis.

In this instance, Edwize’s Jackie Bennett takes on William Ouchi, UCLA prof and Grand Poobah of the Weighted Student Funding approach to education, and his description of the instances in which WSF has helped improve educational equity and performance.

Here’s Bennett writing about Ouchi’s use of Boston as WSF success story:
Ouchi points to Boston schools as an example of WSF success. Ouchi says, “The results have been good. In Boston…scores on state tests…are 30 to 50 percent higher than they are at the regular public schools with similar student bodies.”

But here’s what the most recent on-line data I could find says about Boston’s WSF schools:
1. They serve substantially fewer poverty (free lunch) students (61% K-12, as opposed to 78% in the overall school population).
2. They serve substantially fewer ELL kids (3.8% as opposed to 15.8)
3. They serve substantially fewer regular special education kids (1.7% as opposed to 9.8%)…So much for Boston.

Actually, so much for basic reasoning.

Bennett compares WSF schools to rough district-wide averages and says that Ouchi’s analysis is flawed because he is comparing fundamentally different students. But Bennett misses the point because Ouchi actually says in the passage she quotes that he controls for “schools with similar student bodies.”

This basic logic-misstep discredits the entire post. And not to be too much of an elitist but isn’t it silly to think that a fully tenured research professor with Ouchi’s credentials would make as significant a mistake as to make claims about comparative student performance without controlling for essential student characteristics? Maybe we need some “Bunkum Awards” for education blogging.

2 comments:

Unknown said...

After I posted on EdWize, Mr. Ouchi and I actually exchanged several emails. In them, Mr. Ouchi never disputed my conclusions about Boston. He did point out that he was not a paid consultant to Mr. Klein, and I put a correction in the comments related to that.

If you look again at Mr. Ouchi’s article, you will see that he never makes the claim that in the comparison between the Boston Pilot (WSF) schools and the public schools, he himself controlled for schools with similar student bodies. He does assert that they are similar, however, and that assertion also appears in the source for the data I used. A close look at the data from that source however, reveals otherwise.

This is not such an unusual occurrence when it comes to data an research. If we look at the website of NYC’s schools, for example, we often find claims about our schools that a good look at the data on the same website simply does not substantiate.

I would be happy to put all of my correspondence with Mr. Ouchi on this website, but it does not seem at all appropriate for me to publish what was professional, but also essentially private, correspondence.

Unknown said...

I apologize for typos in the previous comment. I thought I was previewing, not publishing.
--Jackie Bennett