Two of the more well-known brand names in education recently combined forces. Robert J. Marzano (wildly popular consultant/author/speaker) produced a report of a study he conducted of Promethean ActivClassroom (wildly popular interactive white board (IWB) technology).
The report has received lots of publicity; I have seen multiple references to it on Twitter and elsewhere. You can get a copy from Promethean, but only by first providing them with lots of contact information here. I wasn’t willing to make that exchange, but the day after discussing the report on Twitter, Sonny Magana, the Director of Education Strategy at Promethean, Inc. was kind enough to e-mail me a copy of the report.
Marzano’s work has not yet been formally reviewed by any “peers” (at least as far as I can tell). While I am very critical of the way peer-review is typically conceived and carried out in academia, there is real value in the process. Therefore, I’m using this space to do just that. This first post is a bit of an introduction. In subsequent posts, I’ll address methodological and analytical issues.
In this first post, I’ll try to do two things simultaneously: address a key criticism and establish some semblance of credibility as a reviewer. The report states that it was prepared by the Marzano Research Laboratory for Promethean, Ltd. That undoubtedly means that Promethean funded a study of their own product(s). Such an arrangement, which certainly gives the appearance of a lack of objectivity, is not unpredecented and not even unusual. I should know; I’ve done plenty of evaluation research as a “third-party, independent” evaluator funded by vendors. For the better part of ten years, I was part of a research team that conducted evaluation research funded by private vendors such as Lightspan [since purchased by Plato Learning], Scholastic, eChalk, Jostens [sic.], etc to study their own products/programs. Based on my experiences, I can state confidently that those sorts of arrangements should be viewed with skepticism and examined critically. I stand by much of the work I did and would defend the work against any critique. However, there were certainly instances where the vendor/funding source “influenced” the contents of the final report. More often, the final report was written in a way that would be most palatable to the client.
[NOTE: of the privately funded evaluation research I was a part of, I've only been able to find one report that is publicly available. This report of a large-scale evaluation of Scholastic's READ 180 (funded by Scholastic) happens to be one by which I swear. There are a number of reasons why this study is credible, but the most important factor is that the main stakeholder was the Council of Great City Schools and not Scholastic].
Ultimately, without being present at the initial negotations between the parties and without being privy to conversations between the researcher(s) and the client, it is hard to know how “objective” or “honest” a research report is when the study is of a product/program and the study is funded by the vendor of said product/program. The best we can do is to (peer-)review these sorts of reports against the standards of educational research. Onward then…
PART II: Research Design
Thanks for this, well stated … looking forward to the review.
Cheers–looking forward to the review as well
Thanks, Jon. I’m very much looking forward to reading, and potentially discussing, your review.
As always, excellent work.
[...] [NOTE: this is the second in a series of posts about a report recently issued based on a study done by Marzano Research Laboratory. Part I is here.] [...]
[...] of posts about a report recently issued based on a study done by Marzano Research Laboratory. Part I is here and Part II is [...]
Dr Becker is being too kind to Promethean and Dr Marzano. While he can’t say what arrangements were made between researcher and the vendor, we do know the outcome. Promethean slapped Marzano’s face on their faster than Wheaties could put Michael Jordon on their box. The entire media blitz is designed to promote their products with an endorsement by a prominent instructional lecturer. From this peer review, I sincerely hope that others look critically at the original paper that got Marzano’s ball rolling. The same critcisms that have been addressed in this are evident in his earlier “research”.
[...] of posts about a report recently issued based on a study done by Marzano Research Laboratory. Part I is here, Part II is here, and Part III is [...]
1. I reviewed this research on my blog over a month ago.
http://edtechleadership.com/wordpress3/?p=246
2. The PDF was available freely on the web whether it still is, I don’t know.
Here is the link to the Marzano PDF
http://www.scribd.com/doc/14646326/Marzano-Preliminary-Report-on-ActivClassroom
Jon,
In your fourth paragraph, you wrote that as an independent researcher, you allowed your clients (Lightspan, Scholastic, eChalk, Jostens) to unduly influence the contents of your research reports. This is a corruption of the spirit of independent research. It’s called bribery. Your logical assertion is: since you did it, everyone does it.
Shame on you, sir.
Since your opinion can be bought and sold so easily, why should we believe you now? You, sir, by your own admission, you have no credibility whatsoever.
In addition, you eschew the “peer review” process, ostensibly due to the highly political influence peddling that this practice continuously elicits in academia. Yet you still offer a peer review after admitting your incapacity to even conduct an unbiased peer review of action research, stating, “I am not an expert in action research.”
So, in your own words: 1. You have accepted bribes in the past; 2. Your influence is readily up for sale, and 3. You don’t know what you are talking about.
Your whole thesis is underpinned by a foundation of corruption and ineptitude, and is therefore worthless in terms of any scholarly pursuit.
Again, shame on you, sir.
[...] of posts about a report recently issued based on a study done by Marzano Research Laboratory. Part I is here, Part II is here, Part III is here, and Part IV is [...]
Nina,
Thanks for stopping by and adding your thoughts.
First, you wrote, “You, sir, by your own admission, you have no credibility whatsoever.” I didn’t admit to being not credible. I was being honest about what I did. Do past indiscretions automatically discredit us? Do former drug addicts who become drug counselors have no credibility?
Second, you wrote, “[i]n addition, you eschew the “peer review” process…” I didn’t completely eschew the process. In fact, I wrote: “While I am very critical of the way peer-review is typically conceived and carried out in academia, there is real value in the process.” What about “there is real value in the process” is unclear? Furthermore, you are helping me make my exact point. I have offered a crtique that itself is not peer-reviewed. Yet, here you are offering a review. This open process is exactly how I hope peer-review is carried out in the future in academia.
Finally, as for my statements about action research…I’m not an expert, but I know enough to feel perfectly confident in stating that no part of what Marzano and his team did resembles action research in any way.
Maybe you’d prefer I be less than honest and pretend to be an absolute expert on every aspect of educational research?
I believe you’ve offended a time traveler. That’s the only explanation for the use of ‘sir’ in such a pejorative manner and the oddly stilted writing style.
I would expect a follow up comment (or handwritten note) from her calling you a ‘scallywag’ or a ‘ne’er-do-well.’
Take advantage of this unique opportunity to study this rare creature.
Dang Jon,
You got a finger wagging, shame on you from Nina. At least you put your conflicts on the table, which I view as an ethically positive disclosure and a realization of the inherent conflicts in this kind of work. Haven’t seen such from Marzano yet.
Clearly Nina is affiliated with either Promethean or Marzano and is trying to discredit you because of your damaging comments on their work, shame on them.
This debate is more heated than yesterday’s episode of the “Jerry Springer Show.” Entertaining and enlightening, nonetheless.
As in many fields, practitioners often unquestioningly accept garbage fed to them from experts. Teachers and administrators should and must conduct their own research. Though flawed, research at that level is not corrupted – at least not usually.
My district now has 90 Promethean ActivBoards and ActivVotes and ActivExpressions. Due to this technology integration, our students are more engaged and our teachers have an excellent formative assessment tool. However, like an teaching and learning tool, it is only has powerful as the skills and knowledge of the teacher using the tool.
That is my two cents.
Enjoy.
[...] Jon Becker’s five part “Peer-review of Marzano’s IWB Study” from this summer: Part 1, Part 2, Part 3, Part 4, and Part [...]
I know I’m a little late to this, but I appreciate your critique of the Marzano’s research. I remember being skeptical when it came out and was announced that Promethean funded it. Because of his reputation, Marzano’s work is being cited wildly and I’m looking forward to reading the rest of the series. Thanks for addressing this.