As the result of a quick Twitter back-and-forth, I told @JeffNugent and @mcglaysia that I would write a blog post about the research linking technology integration and student achievement.  A couple of points before I get into it:

• I have been the lead investigator on many (maybe a dozen or so?) studies aimed at examining the relationship between technology integration and student achievement.  These studies ranged from small studies (one or two schools) to federally-funded, statewide investigations.  So, I have a pretty decent practical understanding of this body of research and how the work gets done.  Some day I’ll write about the politics and the nitty gritty of this sort of work.  For now, though, I’ll just say that the old joke is more true than it is funny: “educational research is like sausage.  If you like to consume either one, you don’t want to watch it being made.”
• This pool of literature is deep and getting deeper all the time.  I can’t possibly get to everything.  In fact, I’m only going to cover those with which I am most familiar.  That means, I’m not necessarily presenting the “best” research; just those that I know of and that I think are reasonably respectable.
• The Ed. Tech. Action Network (ETAN) has done a decent job of summarizing some of the research.  You can find their page with lots of links here.
• Finally, I’m not terribly proud of my work in this area.  I know that advocates of ed. tech. say that we MUST show positive student achievement effects to move the policy agenda forward.  But, for me, student achievement, especially as typically measured in these studies, is not even close to the most important outcome we need to be considering when evaluating the impact of technology in education.  I’m much more interested in outcomes such as student engagement and student learning (as distinct from student achievement).

Anyway, onward…

*The study that’s getting the most attention and that is politically loaded is one that is still being undertaken.  Mathematica, Inc., along with SRI, two of the major independent research firms in the country, have been contracted to conduct The National Study of the Effectiveness of Educational Technology Interventions.  The project’s website contains all the information you need to know about the study, including the first report which was issued last year.  So far, after one year, according to the press release issued at the time of the release of the report, “On average, after one year, products did not increase or decrease test scores by amounts that were statistically different from zero.”  So, no link between tech. and achievement.  But, that was after one year. The next report should be out soon.

*I’ve been pleased with the amount of attention the West Virginia study I co-lead in 1998-99 has received over the last decade.  The report from that study documents fairly significant positive relationships between the use of computers and student achievement.

*Harold Wenglinsky has done some significant work in this field.  His first major study, conducted in 1998 while he was at ETS, demonstrated that under the right conditions, the use of computers in schools was positively related to math achievement.  More specifically, “higher mathematics scores were related to adequate access to computer technology (hardware, software, and overall infrastructure) in conjunction with teachers trained in technology use and the use of computers to learn new, higher-order concepts.”

*The USEiT (Use, Support, and Effect of Instuctional Technology) study, stands out to me for its high quality and for the quality of the many reports that have been disseminated from that one study.  Take a look, particularly, at Reports 10 and 13.  Of the many findings coming from that study, the researchers discovered that “students who reported greater frequency of technology use at school to edit papers were likely to have higher total English/language arts test scores and higher writing scores on the Massachusetts Comprehensive Assessment System (MCAS) than students who did not.”  I still use some of the scales that the USEiT researchers developed for my own work.

*Finally, and more recently, Missouri’s eMints program has been well-documented and thoroughly studied.  There’s an entire page of research reports, including the most recent analysis of student achievement.  eMints has been consistently positively associated with student achievement.

There’s more; gobs more.  If you cross-reference the works I’ve pointed to, you’ll be well on your way to collecting a critical mass of the work that’s been done in this area.

Tags: achievement, education, research, technology

Well, other than “technology,” if I had to choose five tags to describe myself, sports, education and law would be in the top 5.  So, imagine my surprise when I was pointed to this video of a Malcolm Gladwell speech/presentation (what is it that he does exactly?) covering those three areas.  The main topic of his speech is the mismatch problem; the idea that in making hiring decisions employers regularly use metrics that are very poor predictors of success within their particular area of employment.  The substance of the presentation is certainly interesting, but here’s what I want to do with this video:

I want to use it as part of a major project for a doctoral level educational research course.  It’d be like a fact-checking exercise.  Students would have to listen to/watch the segment about hiring teachers and note each claim that Gladwell makes which is presumably research-based (i.e. that reducing class sizes from 22 to 16 will lead to increases in achievement of 5 percentile points).  Then, for each claim, they would have to find the research that either supports or refutes his claim.  The students would synthesize the research and write up their findings.  That would be fun/cool, right?

Tags: education, Gladwell, law, research, Sports

Tomorrow, I head out to West Virginia for a week of data collection. From Tuesday through Friday, I’ll be visiting 15 schools as part of a statewide evaluation of their Technology Integration Specialist (TIS) program. That program, funded through federal EETT funds, puts specialists in schools to work with teachers to facilitate better integration of technology. It’s not an unusual model, but the implementation varies widely across the state. That allows us to document what works and what does not with respect to facilitating technology integration.

I will be using this space this week to accomplish two things (two birds…one stone…?). First, I will post daily about some aspect of our (I’m part of a research team; this is our fourth set of visits this year) findings. My hope is that I can use those posts (along with modifications based on comments?) to craft an article about the findings of the study. This is my second go-round with this particular program; I was a PI on a federally-funded study that ended a year or so ago of this same program. So, some of the data from which I’ll draw conclusions will come from that earlier study (I’ve got tons of field notes already and visiting 15 schools in 4 days will just add to that mix).

I will miss my family, but I always value the “naturalistic” research endeavors I get to undertake. My reports from the road begin tomorrow. Until then, enjoy a little John Denver (RIP):

Tags: research, technology integration, West Virginia

Gary Stager rightfully challenged edubloggers to follow/care/blog about the Reading First study. I’d ignored his words… until now. After reading an article by Sol Stern about the study (txs to eduwonkette for pointing me to the article), I had to comment…

You’re probably aware of the old adage, “If you’re losing the game, change the rules.” Well, I think the corollary for the social sciences is, “If you don’t like the results of a study, bash the methods.” Thus, given the unnecessarily political nature of reading instruction, and especially research on reading instruction, it was unsurprising to read a methodological critique of the recent IES-supported study of the Reading First program. It was even more predictable that the critique would come from a “scholar” at the Manhattan Institute.

What IS incredible and even ironic about Stern’s attack, in my opinion, is that he basically makes the argument against everything that the current federal administration has argued for over the last 8 or so years. And, having read a bit about Stern, I doubt he wants to do that.

A little background…

Reid Lyon and Grover “Russ” Whitehurst are two of the most influential figures in federal education policy and research circles. Lyon was the head of an influential branch of the National Institute of Child Health and Human Development (NICHD). In that capacity, he had his hands in the policy circles as well as within the research arenas. He is often credited with being the architect of the Reading First program. Whitehurst is the Director of the Institute for Education Sciences (IES), the education arm of the U.S. Department of Education. Thus, he is the chief federal educational researcher; as such, he certainly has influence on educational policy as well. Lyon and Whitehurst are both trained as psychologists and worked in the realm of special education and especially children with learning disabilities. Their research and development work has probably helped advance reading outcomes for lots of children with learning disabilities.

However, they arrived in Washington and tried to push the idea that all children must learn to read the same way that children with learning disabilities learn to read. Furthermore, and more importantly for this blog post, they came to Washington with narrow conceptions of educational research. Like just about everyone I know in the field of educational research who is trained within the disciplinary traditions of psychology, they are paradigmatically disposed to experimental research designs. Thus, Lyon, Whitehurst and their ilk are chiefly responsible for the hegemony of experimental research that emanates from inside the beltway. As a result of their influence, virtually all educational research conducted and funded by the federal government must be done according to the dictates of one particular research paradigm; i.e. the “gold standard” experimental design.

So, it is incredibly ironic when Whitehurst’s agency (IES) funds/supports/directs a study of Reid Lyon’s baby, Reading First, and the initial result is that “on average across the 18 study sites, Reading First did not have statistically significant impacts on student reading comprehension test scores in grades 1-3″ [that language comes directly from the IES website where the study report is located]. That irony and that result notwithstanding, the real irony may be Stern’s criticism of the study which amounts to a condemnation of experimental research in education.

Research using experimental designs rarely employ large samples. Experimental designs, to function properly, must be “controlled” so as to avoid contamination (i.e. internal validity). And, to carry out a large-scale experimental study is terribly expensive and difficult to do. In fact, experimental research is not so concerned with sample size as it is with assignment and validity. Stern does not seem to recognize this and can’t seem to make up his mind about the sample in the Reading First study. Early in the article, he writes, “The study found that students in a small sample of Reading First schools…” So, he characterizes the final sample as “small.” Then, later, complaining about how the study methods were compromised, he writes, “Instead of 30 Reading First schools in six districts, the study would compare 128 Reading First schools in 13 states to a control group of schools that applied for Reading First but didn’t qualify for the federal grants.” So, if a sample involving 128 schools in 13 states is “small,” what would he have said if they stuck with the original sample of 30 schools? Would that have been a REALLY small sample?

Stern also writes, “…instead of using the ‘gold standard’ —random-assignment design—the study would instead compare the schools using a statistical technique known as a ‘regression discontinuity model,’ a less rigorous and comprehensive approach.” If RDM is less rigorous and comprehensive, why is it then one of the favored approaches on virtually all of the calls for proposals coming from IES these days? For example, consider page 9 of this current RFP, where, under the section called “Implementing Rigorous Designs,” in subsection A, it is written: “One approach to rigorously evaluating an intervention is to employ a regression discontinuity design (http://www.socialresearchmethods.net/kb/quasird.php).” Mr. Stern, would you like to tell Dr. Whitehurst that grant proposals put out by his agency are favoring less rigorous approaches to evaluation?

Finally, and most importantly, Stern is concerned that “the study was compromised because ‘the control groups were often doing the same thing that the Reading First groups were doing.’” In methodological terms, we would refer to that problem as a threat to internal validity (eduwonkette specifically refers to this threat as SUTVA). In common sense terms, we would refer to that problem as, well…unavoidable. Do you know why? BECAUSE THESE ARE REAL SCHOOLS AND CLASSROOMS UNDER STUDY. THESE ARE NOT PRISTINE LABORATORIES WHERE EVERYTHING IS CONTROLLED BY THE RESEARCHER!!!

You see, researchers like Russ Whitehurst believe that the best research starts with randomly assigned treatment and control groups and accounts for every possible threat to internal validity. That may be fine in a small-scale study with laboratory-like conditions. However, in education, threats to internal validity abound. Teachers and students come and go. Teacher and students talk to each other about what they do. They are not independent observations. And, Mr. Stern, I know it’s hard to believe, but some teachers do very similar things to each other. One teacher may actually be teaching or using a particular curriculum very much like that of a teacher randomly assigned to a treatment group for a study. Thus, “treatments” in education may not be so distinguishable.

Eduwonkette wrote that “[e]xperiments in social science are fundamentally different than experiments in medicine, and it turns out gold standard is often more silver or bronze than we would have hoped.” Exactly. And, I would add, experiments in SCHOOLS are fundamentally different than just about any other “industry.” Why, because in experimental research, “noise” is a huge problem. In education, I would argue, “noise” is very often a very good thing.

So, Mr. Stern. Thanks for pointing out why IES has been naive and wrong all these years.

Tags: education, experiments, research, validity

For me, stories, articles and/or discussions that combine real estate and public schooling are almost always pregnant with racism and/or classism.  In some cases, that racism and/or classism is more obvious than others.

Yesterday’s Wall Street Journal points to a “study” (a very loose term in this case) done by the folks at Trulia.com.  This “study” combined real estate data with data about public schools and yielded a list of 27 neighborhoods in the suburbs of 9 major cities that offer “the best education bang for your buck.”  Like Newsweek’s and now the U.S. News & World Report’s ranking of high schools, this list is largely a proxy for white affluent school systems.  I’d bet the ranch that if you looked at the demographic data from these 27 school districts, you’d find that they serve largely white, affluent families.  As one example, Ridge HS is the high school that serves Basking Ridge, NJ (the “best” neighborhood in the NYC suburbs).  That high school serves students that are 83% white and 13% Asian.  Less than 1% of the students in that school are eligible for free or reduced-price lunch.

When my family moved to Richmond, VA, I knew that I wanted a house in Henrico County because that county embarked on one of the earliest and largest-scale laptop programs in the country.  For the last 6 or 7 years, every student in grades 6-12 in the county gets a laptop.  Furthermore, we bought a house in an area served by a high school that is considered the lighthouse school for technology in a technologically-progressive county.  My son is not yet 3, so rezoning could change things, but I couldn’t worry too much about 10-12 years from now.

So, I’m calling for a “study” that locates neighborhoods that offer “the best educational technology for your buck.”  This would allow real estate agents to “steer” buyers to districts and schools that are forward-thinking and relevant.  Now, how do we get the right data???

Tags: classism, education, racism, real estate, research, technology