Educational Insanity

[NOTE: this is the second in a series of posts about a report recently issued based on a study done by Marzano Research Laboratory.  Part I is here.]

PART II: RESEARCH DESIGN ISSUES

From a research design perspective, this study (or collection of studies?)  is best described as unusual.  In fact, what Marzano’s research team tells us is that they conducted 85 separate small studies and then “synthesized” the results of those studies through a meta-analysis (a set of complex statistical analyses used to examine “effects” across multiple studies).  Meta-analyses are not unusual, and are quite helpful as a way of combining the results across lots of studies.  However, it IS very unusual to use meta-analysis as an a priori technique built in to an evaluation.  Frankly, I’ve never seen it before.  Meta-analysis is more typically used when there is a mature body of research within a topic area comprised of studies by multiple researchers across a number of years.  Furthermore, they are typically done using lots of studies, each of which is fully reported and each of which was selected for the meta-analysis because the full report of the study allowed the analyst to determine its trustworthiness.  That, to me, is one of the biggest problems with this study/analysis; we don’t know enough about each of the 85 individual studies.

Is Marzano’s meta-analytic approach “wrong?”  Not necessarily, but to me, it’s indicative of a certain laziness.  High-quality evaluation research in education is complicated and costly.  It requires a ton of coordination and planning, especially to make sure that key data are high-quality and comparable.

Ultimately, Marzano got 79 teachers to agree to “participate” and got data on over 2,700 students.  That’s commendable.  But those teachers varied by grade level taught and subject taught (other than the elementary teachers; more on them later).  They also taught in different states.  So, he had a huge “dependent variable” problem.  In other words, there was no single, comparable measure of “student achievement” (his stated outcome of interest).  He needed a way to account for that and chose to deal with it by way of analytic techniques (i.e. meta-analysis), rather than by focusing the study (perhaps within a single state within one or two grade levels). He also chose a lazy way to get data on student achievement (more on that in Part III).

COMING NEXT:

WEDNESDAY – Part III: Construct validity and reliability issues

THURSDAY – Part IV: Internal validity issues

FRIDAY – Part V: Summary and recommendations

1. Educational Insanity » Blog Archive » “Peer-review” of Marzano’s IWB Study Report, Part III says:

   June 3, 2009 at 4:08 pm

   [...] [NOTE: this is the third in a series of posts about a report recently issued based on a study done by Marzano Research Laboratory.  Part I is here and Part II is here.] [...]
2. Mike Scott says:

   June 4, 2009 at 6:17 pm

   In Marzano’s case, meta-analysis is just a case of dancing the with the girl that he brought to the dance. It’s the way he initially analyzed work in his in the study: A Theory-Based Meta-Analysis or Research on Instruction. His basic idea was to use meta-analysis to determine which instructional strategies could be correlated with increased rates of learning. Supposedly, he scoured the literature for research on effective practices and used meta-analytic techniques to discern which kinds of instruction got the most bang for the buck. He came up with nine of’m. Like the Promethean study, there is absolutely no peer review of the research. Didn’t keep it from having legs though. If you teach in Virginia, you pretty much have to have been under a rock for a long time not to know about Marzano.

   I don’t have any problem with strategies he talks about in “Classroom Instruction that Works”, but I do have a problem with when such ideas are presented as “research based”, when they really are not. Good research and sound analysis is difficult, expensive and time consuming. We can better.
3. Educational Insanity » Blog Archive » “Peer-review” of Marzano’s IWB Study, Part IV says:

   June 5, 2009 at 12:51 am

   [...] a report recently issued based on a study done by Marzano Research Laboratory.  Part I is here, Part II is here, and Part III is [...]