As the result of a quick Twitter back-and-forth, I told @JeffNugent and @mcglaysia that I would write a blog post about the research linking technology integration and student achievement.  A couple of points before I get into it:

• I have been the lead investigator on many (maybe a dozen or so?) studies aimed at examining the relationship between technology integration and student achievement.  These studies ranged from small studies (one or two schools) to federally-funded, statewide investigations.  So, I have a pretty decent practical understanding of this body of research and how the work gets done.  Some day I’ll write about the politics and the nitty gritty of this sort of work.  For now, though, I’ll just say that the old joke is more true than it is funny: “educational research is like sausage.  If you like to consume either one, you don’t want to watch it being made.”
• This pool of literature is deep and getting deeper all the time.  I can’t possibly get to everything.  In fact, I’m only going to cover those with which I am most familiar.  That means, I’m not necessarily presenting the “best” research; just those that I know of and that I think are reasonably respectable.
• The Ed. Tech. Action Network (ETAN) has done a decent job of summarizing some of the research.  You can find their page with lots of links here.
• Finally, I’m not terribly proud of my work in this area.  I know that advocates of ed. tech. say that we MUST show positive student achievement effects to move the policy agenda forward.  But, for me, student achievement, especially as typically measured in these studies, is not even close to the most important outcome we need to be considering when evaluating the impact of technology in education.  I’m much more interested in outcomes such as student engagement and student learning (as distinct from student achievement).

Anyway, onward…

*The study that’s getting the most attention and that is politically loaded is one that is still being undertaken.  Mathematica, Inc., along with SRI, two of the major independent research firms in the country, have been contracted to conduct The National Study of the Effectiveness of Educational Technology Interventions.  The project’s website contains all the information you need to know about the study, including the first report which was issued last year.  So far, after one year, according to the press release issued at the time of the release of the report, “On average, after one year, products did not increase or decrease test scores by amounts that were statistically different from zero.”  So, no link between tech. and achievement.  But, that was after one year. The next report should be out soon.

*I’ve been pleased with the amount of attention the West Virginia study I co-lead in 1998-99 has received over the last decade.  The report from that study documents fairly significant positive relationships between the use of computers and student achievement.

*Harold Wenglinsky has done some significant work in this field.  His first major study, conducted in 1998 while he was at ETS, demonstrated that under the right conditions, the use of computers in schools was positively related to math achievement.  More specifically, “higher mathematics scores were related to adequate access to computer technology (hardware, software, and overall infrastructure) in conjunction with teachers trained in technology use and the use of computers to learn new, higher-order concepts.”

*The USEiT (Use, Support, and Effect of Instuctional Technology) study, stands out to me for its high quality and for the quality of the many reports that have been disseminated from that one study.  Take a look, particularly, at Reports 10 and 13.  Of the many findings coming from that study, the researchers discovered that “students who reported greater frequency of technology use at school to edit papers were likely to have higher total English/language arts test scores and higher writing scores on the Massachusetts Comprehensive Assessment System (MCAS) than students who did not.”  I still use some of the scales that the USEiT researchers developed for my own work.

*Finally, and more recently, Missouri’s eMints program has been well-documented and thoroughly studied.  There’s an entire page of research reports, including the most recent analysis of student achievement.  eMints has been consistently positively associated with student achievement.

There’s more; gobs more.  If you cross-reference the works I’ve pointed to, you’ll be well on your way to collecting a critical mass of the work that’s been done in this area.

Tags: achievement, education, research, technology

One of my posts from almost four months ago has been resurrected by comments from Tina K. and Amir.  In that post, I suggested I’d do some more digging.  So, I dug.

Some background…these are NAEP data with tables and statistics generated by the NAEP Data Explorer.  The NDE is an awesome (free!) tool for analyzing NAEP data.  It’s kinda amazing to me that more folks, including the media, haven’t picked up on this tool to do some really quick and easy data analysis.  Anyway, to satisfy the inquiries of Amir and particularly Tina, I analyzed 8th grade data from the 2007 NAEP administration.  The 8th grade assessment includes the best “type of computer” use data; i.e. we can break down computer use by some specific applications.  That’s what I did.  The math achievement results disaggregated by response category follow:

[NOTE: click on images to enlarge them]

So, quite clearly, the same results appear as with the 4th grade data in my earlier post.  The group of students who never or hardly ever use computers score significantly higher than the other groups, across all applications.  Again, I don’t know anything about those students demographically.  But, still…

Going one step further, I ran a regression analysis with four of the independent variables (i.e. the “types” of uses).  The NDE would only allow me to use four; it’s a statistical/psychometric thing…don’t ask.  So, I took out word processing and drawing as those seemed likely the most remotely associated with math achievement.  The results are as follows (again, click on the image to enlarge):

Make sense? Yeah, I know, unlikely. Unless you are well versed and regularly practiced in regression analysis, there’s no reason that would make any sense to you. So, let me try to summarize some key results:

• Of all the variance in math achievement, differences in these four types of computer use for math account for 16%. That’s not that high; not terrible, but it’s safe to say that, overall, computer use for math does not explain much of why kids differ on their math scores.

The independent variables are “contrast coded” which is the right way to do this analysis. But, it limits what we can say. That being said,…

• The average score for a student who never or hardly ever uses computers in any of those ways is 291.
• Students who use the Internet for math once every few weeks score a bit higher than the previously mentioned student (i.e. never or ever uses in any of the ways).  That is, by simply adding Internet use for math once every few weeks adds a little bit to the average score of the non-computer using student.
• Same story for using graphing programs for charts.
• Adding Internet use once every few weeks AND graphing programs once ever few weeks has a cumulative positive effect on the non-computer using student (again, though, VERY small positive effect).
• The more frequently kids use math programs to drill on math facts, the lower they score.

So, there you have it.  I’ll likely play around a bit more with the NDE to see what else I find with respect to other subjects and other uses of computers.  Fun times!

Tags: achievement, math, NAEP, technology

I think I’m on to something here (of course I do; why else would I be writing this?)…

A. There have been smatterings of dissatisfaction with the recently concluded NECC, particularly among the more experienced and “followed” edubloggers (see e.g. Will and Karl)

B. Great discussions about systemic educational change have been occurring in the edublogosphere, especially over at Chris Lehmann’s blog and Will Richardson’s blog.

I think part of the reason for A is a desire for what’s being discussed in B.

I say that because NECC (the subject of A) is ALL about the micro.; it’s about pedagogy, classroom tools, projects, etc.    The subject of B is macro; it’s about changing the “system.”

So, here’s my proposal.  Let NECC (and ISTE) be what it is; a place to learn about technology integration; a VERY important place for LOTS of educators.  CoSN’s annual conference, on the other hand, is the place where learning and conversations about policy, leadership, change can happen.  Apparently, CoSN doesn’t think professors (or higher ed. more generally) are worthy of attending their shindig, but I’m likely to crash their party next year (besides, I’ve been dying to go to Austin, TX anyway!).

Tags: COSN, education, NECC, policy, technology

There’s an interesting discussion happening in a comment thread over at Will Richardson’s blog.  If I could summarize, I’d say the conversation is about how large-scale, future-oriented school change might happen.  Will and others are searching for ways to make change happen consistent with what he and we know about what’s possible.  I particularly like the idea of folks like Will delivering keynotes at conferences such as AASA and ASCD.  I think the ed. tech. echo chamber needs to “infiltrate” the general ed. world.

It occurs to me, though, that someone (me?) also needs to infiltrate higher education and schools of education in particular.  I can’t speak for all schools of education, and what evidence I do have is anecdotal.  But, here’s what I believe to be happening in the places that serve pre-service teachers and school leaders:

• Courses on school change or school reform, if they are offered at all, are typically taught in departments of educational leadership.  That’s understandable, but shouldn’t pre-service teachers understand theories of change?
• The school reform courses typically focus on reform per se; i.e. how does change happen (theories of change).  However, there’s not much emphasis on the “why” or the “towards what”.  There might be some discussion of changing school climate or school culture, but I don’t know of many education faculty members who are leading courses on future-oriented change.
• The technology courses are just that, courses.  Typically, a pre-service teacher ed. program will include one (maybe two) standalone tools-oriented courses (here’s how you can use PowerPoint with your kids!).  The use of technology is not integrated across the teaching methods courses.  You’ll typically find one faculty member in each department of teaching and learning who is the “tech. prof.”  She/he teaches those standalone courses.
• I know that in the school of education in which I currently work, and the one I left last year, there are no cross-department conversations about the future of education.  Everyone is so focused on the here and now, meeting accreditation demands and other practical matters of making sure our students are prepared to work with their students.

So, my number one priority for this year is to make change happen locally.  I’m going to insist that my colleagues join me in learning about the future of education and the future of schooling.  As that happens, we’ll discuss the implications for us as professors of education.  I will argue that we are preparing educators for an outmoded system; others will push back.  That would be wonderful.

photo credit: MarmotChaser

Tags: education, future, reform, school, technology

I’m sure I’m not the first to write about this, but Scott McLeod delivered his latest list of the (potentially) top 50 P-12 education blogs and he got me thinking. The list is what it is…a ranking based on one metric: how many other blogs (and blogs only) have linked to a given blog. As long as Scott’s clear about that, which he very well is, then readers can make what they want of the list.

I commented, noting first that the list overwhelmingly consists of ed. tech. focused blogs. There are many reasons for this, most notably that blogging requires some technological awareness and a little bit of tech. savvy that ed. tech. folks are most likely to have. I also noted my surprise not to see certain educational policy bloggers on the list, such as Sherman Dorn, Matthew Tabor, Alexander Russo and Gary Stager (who I see as more of a bridge blogger). Dorn’s been blogging practically since I was born, and I learn more from him than just about any of the other bloggers I read.

I’m still searching for the right metaphor, because I don’t know that “echo chambers” is quite right; maybe it is, though. According to Wikipedia, “[m]etaphorically, the term echo chamber can refer to any situation in which information, ideas or beliefs are amplified or reinforced by transmission inside an ‘enclosed’ space.” I’ve gotten myself in trouble in the past for suggesting that the edtechblogosphere is a closed space, so I have to be careful here. But, I’ve noticed that the ed. tech. folks on Scott’s list all tend to comment on each other’s blogs and they all tend to communicate with each other via Twitter and other communication media. There are LOTS of REALLY valuable reasons for that, mostly that there is lots of co-learning going on that way. [NOTE: I've noticed that the same sort of "echo chamber" exists for the ed. policy bloggers, although the echoes there are less frequent and more muted (IMHO), and I don't know that there's much electronic communication going on between them beyond the blogs].

So, maybe rather than a closed space, then, a problem as I see it is that there are very few nodes connecting the unfortunately and unnecessarily distinct networks (i.e. the ed. tech. networks and the ed. policy networks). When Gary Stager called out the ed. tech. bloggers for failing to write about the Reading First study, he (purposefully or not) served as a node/bridge but also pointed out just how little overlap there is between the tech. and policy networks. A few commenters thanked Gary for pointing out the “story”.

I’m problematizing this disconnect because, for me, it is symptomatic or an extension of the sort of compartmentalization that plagues the education sector generally. I believe that one of the reasons our public schools have not experienced rapid technological change is because the ed. tech. folks are treated as separate. Leaders and decision makers don’t really understand the issues, so they delegate it to the geeks and wireheads (I use those terms lovingly). It’s the same way with, for example, special education. Leaders and decision makers don’t really understand all of the laws, regulations, etc., so they delegate authority to THAT department and THOSE educators. We have the same problems in schools and colleges of education. For example, I work in a department of educational leadership. Separately, we have a Department of Special Education and Disability Policies. So, let’s say an educator wants to pursue a doctorate with the idea of serving as a special education policymaker or an assistant superintendent for special education. To which program does that educator apply?

We need more integration in education and less compartmentalization; fewer echo chambers. After all, everything is miscellaneous, right?

Tags: blogs, echo chamber, education, networks, nodes, policy, technology