Rowdiness and wrongly done research

The other day, I called for journalists to better understand basics of research methods.

Politicians and political organizations sometimes make empirical claims which, when you look closely, turn out to be badly flawed.

They compare the cost of an existing single-family home in the state to building an apartment building in Portland, thus mixing apples and oranges. Or they evaluate the current system for tracking people registered to vote by using data from before the current system was put into place.

These sorts of errors should be identified by journalists in news stories.

But it’s not just politicians making basic errors, nor political reporters overlooking research errors.

The case of the Rockport, Maine third-grade boys-only math classroom is yet another example. Here’s what the Bangor Daily News reported:

The major difference between the all-boys class and the two other third-grade classrooms isn’t the material taught, but the environment. The class can get very noisy. Boys are allowed to get up, move, do their math while lying on the floor if they need to. . . [The room is] full of stuff — computers, a turtle tank, a plant tank, bookshelves, sand trays, sinks and boxes of bones. . .

On this particular morning a group of four boys went into an open room outside the class to look at their “dig site.” The boys already had dug up artifacts from the school sandbox, re-created the dig site inside the school and mapped the artifacts on paper charts. On Friday the boys had to categorize artifacts and count them. The rubble was supposedly from an early Native American site.

This approach is predicated on the view that boys are rowdier than girls. The classroom will be tracked and learning results will be compared to other classroom arrangments to see if outcomes are any better.

But how could anyone know that any improvement is due to the class being boys-only?

Let’s think about this for a minute. There are two differences between this classroom and others:

1. It’s boys-only.

2. It uses a very different teaching approach, one which is very engaging and which requires students to apply their math to many subject areas.

Thus the answer to my question is — they can’t.  It is impossible to know if any improvements are due to the class being boys-only.

Well, why not?

One, you don’t know if different outcomes are due to having a boys-only classroom or having a different model for learning. It could be that the new model for learning would produce great outcomes in a classroom with both boys and girls or for girls only. Better outcomes might have nothing to do with putting boys together.

Two, it’s well established that teachers trying out new curricula tend to be very enthusiastic about trying out something new. That, by itself, typically produces better outcomes — until the new teaching approach becomes the standard.

Third, people who are involved in a new approach that is being studied by others typically perform better, at least for awhile. This very well-known effect is called the Hawthorne effect.

All of these add up to a classic problem when it comes to just trying new curricula or new programs. You have to try them awhile to make sure there aren’t enthusiasm and Hawthorne effects. And if you don’t have the proper research design, you can’t know the impact of the curricula or program.

What’s the right research design for determing the impact of new educational approaches?

One must use an experimental design. This means setting up proper control groups in which each different variable is held constant.

Thus you could have two classrooms with the new, highly engaged curricula; one is boys-only.

Or you could have the old curricula with one class that’s boys-only and the other is not.

But comparing a boys-only classroom with a new teaching/learning approach with a non-boys only classroom using the older teacher/learning style won’t tell you if boys learn better if they are sex-segregated.

To assess effectiveness, the two variables must be untangled or else you know — well, nothing.

Amy Fried

About Amy Fried

Amy Fried loves Maine's sense of community and the wonderful mix of culture and outdoor recreation. She loves politics in three ways: as an analytical political scientist, a devoted political junkie and a citizen who believes politics matters for people's lives. Fried is Professor of Political Science at the University of Maine. Her views do not reflect those of her employer or any group to which she belongs.