Educational Insanity

[NOTE: this is the final post in a series of posts about a report recently issued based on a study done by Marzano Research Laboratory.  Part I is here, Part II is here, Part III is here, and Part IV is here.]

PART V: SUMMARY AND RECOMMENDATIONS

[NOTE #2:  I know, I know...I'm a couple of days late on this one. Sorry.]

Before I sum up and conclude, I should point out one other major flaw in this study.  Marzano and his team use percentile ranks incorrectly.  On page 18 of the report, they write: “Of particular interest is the column entitled ‘% Gain.’ Again, this column contains the percentile gain (or loss) in achievement associated with the treatment (i.e., use of Promethean technology).”  Two problems here.  First, percentiles are not the same as percentages (or % as it is written in the report). Second, they then go on to write: “This value [the percentile gain] was determined by consulting a normal curve table for the area for each reported effect size.”  This would be fine if the scores on the dependent variables are normally distributed, which they most definitely are not.  For Marzano to go around saying that incorporating Promethean IWBs into instruction will improve student achievement by 17 percentiles is wrong on lots of levels.

It should be clear by now that if I were reviewing the Marzano IWB study report as a manuscript submitted for publication in a peer-reviewed journal, I would reject it.  I would not even mark it as “revise and resubmit.”  The problems with the work are too critical and, in most cases, impossible to fix.

In summary, those problems are:

• Misuse and misapplication of meta analysis.
• Incorrect usage of key terms.
• Serious problems with measurement validity and reliability.
• Major threats to internal validity.

Those last two points are with respect to each of the 85 classroom-based studies that serve as the basis for the meta-analysis.  The ultimate problem, then, is that the hallmark of good meta-analysis is the use of strong criteria as decision points for including individual studies.

As a point of comparison, I’m linking to two reviews of research.  Each is described as having used “best-evidence synthesis” which very closely resembles meta-analysis.  The methods used in the studies reported in the articles below are also consistent with those used by the What Works Clearinghouse.

Effective Programs in Elementary Mathematics: A Best-Evidence Synthesis

Effective Reading Programs for Middle and High Schools: A Best-Evidence Synthesis

In the first article, you’ll notice on the seventh page of the document (p. 432 of the article) a list of criteria for inclusion.  The authors of those articles also provide a list of studies that were considered for inclusion but that were ultimately excluded along with the reasons for exclusion.  This combined approach is critical; it gives the consumer of the research confidence that the data used in the meta-analysis come from many solid studies.

The impact of sample size for any given study included in a meta-analysis is another important point raised in the articles above.  According to the authors of the second article, “[p]revious research (e.g., Rothstein et al., 2005; Slavin, 2008; Sterne, Gavaghan, & Egger, 2000; Taylor & Tweedie, 1998) has shown that studies with small sample sizes report larger effect sizes than studies with large samples.”  As a result, in their meta analysis, the authors weight the individual findings by sample size.  In each of the separate sites/studies used by Marzano and his team in their meta analysis, sample sizes were tiny.  Consider for example site #34, teacher #57 where there were 9 students in the control group and 5 in the treatment group.  There is no way that study gets included in any decent meta analysis.

There is a bit of irony in my choice of articles to post as exemplars.  The lead author in each of those studies is Dr. Robert Slavin, the developer/founder of Success for All.  Slavin has been frequently critiqued for being the lead researcher/analyst/author on many evaluation studies of Success for All, the program that he created.  In other words, he has been accused of producing biased research.  I don’t know enough to say if his research is biased or not; it’s certainly legitimate though to raise the question of bias where he is involved in the research.  What I do know, though, is that each of the articles appears in one of the most well-respected, highly selective peer-reviewed journals.  The math study appears in the Review of Educational Research which is dedicated to only publishing exquisite and top-notch reviews, syntheses, and meta-analyses in education.  Thus, there is good reason to believe that those two articles present exemplars of how meta-analysis type research should be done.

I wrote earlier that doing good, comprehensive program evaluation in education is difficult and resource-intensive.  That said, I believe it would actually be reasonably easy to evaluate the impact of IWBs on student achievement.  In this era of standards and accountability, in any given state, we have year-to-year state test scores (at least in math and reading/language arts) from grades 3 to 8. So, Marzano’s team could have focused on one or two grade levels in one or two subject areas in one state.

Let’s say they focused on 8th grade student achievement.  All they needed to do was to find about 20 middle schools that were willing to participate.  In those 20 schools, there would be one subject-area teacher teaching in a classroom with the IWB and one teacher teaching a comparable class (NOTE: comparable here refers to students who are demographically similar and who are no different with respect to student achievement at baseline) without the IWB.  Surely there are at least 20 middle schools in any state where there are two 8th grade teachers teaching comparable classes.

A common way to get schools and teachers to participate in such a study would be to offer an incentive.  For Promethean, the promise of a free IWB to the teacher/classroom in the control condition the year after the study would be a wonderful incentive.  Given this sampling framework, Marzano’s research team could work with the schools, districts or state departments to get student achievement data on the students in those 40 classrooms (20 treatment + 20 control).  This could easily be done without violating any privacy laws.  The students’ scores on the 7th grade state exams could serve as the pretest or the covariate.  Their scores at the end of the 8th grade year would be the dependent variables.  Over 40 classrooms, we’d be talking about a sample size of well over 800, with well over 400 students in each condition.  Such a study would have lots of power.  Analytic decisions would have to be made with respect to the unit of analysis.  Marzano and his team could use the classroom as the level of analysis and conduct matched-pairs statistical test.  Or, they could use the student as the unit of analysis and account for the nesting or lack of independence by using multilevel modeling techniques.  Either way, this design would be much more appropriate and powerful for estimating the effects of IWBs on student achievement.

In the last couple of days, I spoke about this series of posts to two professors who I respect greatly.  Interestingly, each one was very surprised to hear my opinion that Marzano was affiliated with sloppy work.  One said, “he’s always been so careful.”  That may very well be.  I don’t intend for this series to be an indictment of Marzano (or even of IWBs).  My hope is that I’ve provided a sensible critique of research that is being widely disseminated.

I often lament that decisions in education are too often made in the absence of empirical evidence.  I wish policymakers in education would consult research more often.  However, if educational decision makers decide to make an investment in interactive white boards, I would strongly urge them to do so for reasons other than the evidence offered by the Marzano Research Labs.

1. Mike Scott says:

   June 8, 2009 at 8:50 am

   Well, that’s about what I would have concluded as well and as I read through the paper I wondered what I would have done to conduct such research and I think I’m going to stick with my assertion that much of the research has already been done… just not in the context of k12 education. If it’s just new media (the IWB), then the question falls into a “media comparison effects study” and I’m contending there’s a long and thoughtful history of that work with generalized answer that media doesn’t make much difference in learning. Joe’s response in my last post was skeptical about this generalization, and I really need to carefully read what I can about his work. My primary reference is text book summary authored by Ruth Colvin Clark and Richard E. Mayer, “e-Learning and the Science of Instruction.” Media comparison projects were forbidden in my masters course work at Virginia Tech and one of my professors presented a pretty detailed history of the work as well. The very first question that students of instructional technology ask is ” I wonder if kids will learn better if I use tool “X” compared to tool “Y”?”

   The research question that should be asked is: “Which instructional strategies, procedures and activities best take advantage interactive classroom technologies?”

   I use these technologies frequently. Geez, I have the Web resources, video resources, classroom response systems and software that allows the students to engage and interact with anything I project. That should be better than a copy of worksheet.

   Thanks Jonathan, for opening this can of worms.
2. Linda704 says:

   June 8, 2009 at 9:09 pm

   Jon, your series should be required reading for any research/seminar as a model of critical review of research. Our professors tell us to critique the literature, but not much more. Had I read this, I would have had a much clearer understanding of what that means. I’m sending the link to my advisor, who teaches research methods.
3. Joe says:

   June 9, 2009 at 10:05 am

   Excellent review, your point on percentages vs. percentile ranks is one that I didn’t at all catch.

   Two questions hang out there for me:
   1. Is this a problem with only this work or all of Marzano’s work? I think your suggesting it’s more than just this work.
   2. Is this study completely worthless? I think not, I think it is completely preliminary. If the purpose of the research is purely to sell whiteboards, that’s one thing. The result of the research is to suggest that there may be something happening in classrooms that use interactive whiteboards that increases learning (very preliminary).

   The little gem in the research that you didn’t mention is the flat line of achievement in 7th grade (even given what you said, there would still seem to be that flat line). Something happened there and it’s something that may be related to how IWB’s work in middle school environments vs. elem. or high school.

   As far as this issue of “media research”, in fairness to this study, the IWB classrooms are interactive classrooms (ie. the IWB itself, clickers, slate tablet, plus the software for creating the IWB materials). That is a whole lot of media to be studying at once in a study.

   Mike, When you look at media research you have to consider the media you are speaking of; is the media a kind of learning object (which you are inserting into instruction) or is it a tool you are integrating. Tool integration is much more difficult to measure b/c of the varieties of variation of practice.

   I have to disagree with Jon’s statement that “I believe it would actually be reasonably easy to evaluate the impact of IWBs on student achievement.” The only way to do this is with both qualitative and quantitative measures. If they raise student achievement, what does that proof? You won’t know how they did it which means people couldn’t replicate this possible “best practice”, b/c you won’t know what exactly the best practice is. IWB aren’t a media object they are a “practice” that requires deep study.
4. Ira Socol says:

   June 10, 2009 at 3:13 pm

   As Linda above suggests, your whole analysis needs to be required reading both in grad schools and for political leaders. You did a masterful job of review here, one I have learned a great deal from.

   That said, I will agree re Slavin. Though I have torn apart his work for questions of combined effects and self-promotion (intentionally or not), you can read his papers and make the arguments against them coherently because he presents an actual research argument.

   What Marzano has done here is assemble a random group of studies – many of which might be quite interesting as Action Research – and simply claimed that they collectively form a reasonsable data set.

   I see this as the fault of a number of trends – the national push for numbers in things difficult to quantify, the need to promote one’s own work, insufficient “neutral” research funding, and the ease with which people are fooled by numbers.

   As I say, “statistics are to education reform what the white coat is to the actor in hair growth commercials” or something like that.
5. Jon Becker says:

   June 10, 2009 at 5:17 pm

   Thanks all for the comments.

   @Mike, I’m not familar with the Clark and Mayer book, but I’ll be sure to check it out. I don’t love that they consider instruction a science (at least in the title they do), but I won’t judge a book by its cover

   @Joe, I’m not ready to write off Marzano’s entire body of work. That said, in response to your second question, yeah, I think this study is completely worthless. Also, while I agree that mixed methods inquiry is ideal, the order of the day is “student achievement.” Especially in tight budget times, if folks can’t tie their investments to student achievement, those investments will go away. Hence, the need to do research that way. Sad, but true.

   @Ira – apropos of nothing, my favorite quote is “Educational research is like sausage. If you like to consume it, you don’t want to watch it being made.”
6. Mark Robinson says:

   June 24, 2009 at 7:24 am

   I work for Promethean and can provide some additional information.

   Questions about the meta-analytical research methodology and how the research should be interpreted were put to Marzano Research recently.

   http://www.prometheanplanet.com/server.php?show=nav.19109

   I must also clarify a couple of points made in this critique.

   The report is freely available from the Marzano Research Website. This was not made clear in your comments.

   http://www.marzanoresearch.com/documents/Preliminary%20Report%20on%20ActivClassroom.pdf

   Also, it is characterised throughout as an “IWB study”. In fact every classroom involved was required to be using a Promethean learner response systems that was integrated with the board via Promethean software which made for a more sophisticated media mix.

   Clearly much more still remains to be learned about effective use of technology in classrooms than is known. As the first in a series of studies – this research should beget further research and all comment and review is welcome and indeed essential.

   Thanks for letting a vendor representative post.

   Mark Robinson
   Group Head of Education Product Strategy
   Promethean
7. Josh Vick says:

   October 29, 2009 at 9:39 pm

   Very nice review of the research. It seems that Promethean over the years has been involved in many studies, along with IWBs in general, and Promethean has touted many of these studies as conclusive (for marketing purposes). Thanks for unveiling the truth that IWB research (even when coupled with “learner response systems” AKA student clickers or student response systems here in the states) seems to have a hint of dishonesty if not more.

   I think we can read into Mark’s own comments that Promethean will continue to fund further self-aggrandizing research.
8. Myrtle&Eustacia says:

   November 5, 2009 at 12:12 am

   Jon, mike, linda, etcetera,

   Most welcome discussion for we two,too/also/additionally.

   thanks, flowers, etcetera,

   Any one read/reading Hattie.J.A.C. 2008. Visible Learning: a synthesis of over 800 meta-analyses relating to achievement. Oxford. Routledge, and can refer us to reviews/comments of same?

   Most refreshing discussion…many thanks to all.

   eustacia and myrtle – 23yrs. Primary teacher trainees…New Zealand.