PART IV: INTERNAL VALIDITY
“Internal Validity is the approximate truth about inferences regarding cause-effect or causal relationships” (Trochim, 2006).
I consider research questions as one of three types: descriptive, relational and causal. From there, certain research designs lend themselves to best answering the research questions. For example, naturalistic inquiry methods such as those used in ethnographies or case studies are suited only to answer questions of description.
Marzano and his team set out mainly to answer a question of causality: does use of the IWBs cause improvement in student achievement? How they chose to do that is less than ideal, but we know that was their intention based on the use of quasi-experimental designs in each of the 85 classroom-based studies. Experiments and quasi-experiments are designs intended to address questions of causality.
When such designs are employed, the primary consideration of trustworthiness is internal validity. “All that internal validity means is that you have evidence that what you did in the study (i.e., the program) caused what you observed (i.e., the outcome) to happen” (Trochim, 2006). In any effort to prove causation, one key is to be able to rule out alternate explanations. In other words, alternate causes are threats to internal validity.
There are any number of ways to think about possible threats to internal validity, and there are many ways to describe them. There are at least a couple of general threats to internal validity for each of the 85 classroom-based studies that Marzano and his team used for his meta-analysis. The threats result mainly from the fact that, at least in the secondary schools, the same teacher taught the treatment and the control groups. At one level, that might seem like a benefit in that it eliminates teacher-level confounds. However, consider:
Social threats to internal validity – also known as or related to intervention or exposure bias. The students in the control group are presumably in the same classroom (at a different time) than the students in the treatment group. They see the IWBs. They also probably hear about the teachers using the IWBs from their friends in the treatment class. Might there be some compensatory rivalry and/or resentful demoralization? Trochim (2006) defines the latter:
Here, students in the comparison group know what the program group is getting. But here, instead of developing a rivalry, they get discouraged or angry and they give up (sometimes referred to as the “screw you” effect!). Unlike the previous two threats, this one is likely to exaggerate posttest differences between groups, making your program look even more effective than it actually is.
Experimenter’s bias – it is impossible to tell from the report all that the teachers knew and/or were told about the study. We know that in the instructions to the teachers, it says “Thank you for agreeing to participate in an action research [sic.] study regarding the effectiveness and utility of the Promethean technology in your classroom.” So, they knew what Marzano and his team were looking at/for. This knowledge could easily cause the teacher to pay more attention to her/his teaching in the treatment class. Additionally, just the fact that the teacher had to take an existing unit and figure out how to integrate the IWB technology means that the teacher was biased toward that group (i.e. she/he was more planful about that teaching).
Earlier, I wrote that I’d never seen meta-analysis included purposefully as an a priori part of a separate research design. Frankly, I’ve never seen or heard of a quasi-experiment in education where the (non-random) selection is at the classroom level and yet the treatment and control classes still have the same teacher. I could be wrong here, but in summary, this approach raises a number of general threats to internal validity for each of the 85 individual studies upon which the meta-analysis was based.
FRIDAY – Part V: Summary and recommendations