One of my posts from almost four months ago has been resurrected by comments from Tina K. and Amir.  In that post, I suggested I’d do some more digging.  So, I dug.

Some background…these are NAEP data with tables and statistics generated by the NAEP Data Explorer.  The NDE is an awesome (free!) tool for analyzing NAEP data.  It’s kinda amazing to me that more folks, including the media, haven’t picked up on this tool to do some really quick and easy data analysis.  Anyway, to satisfy the inquiries of Amir and particularly Tina, I analyzed 8th grade data from the 2007 NAEP administration.  The 8th grade assessment includes the best “type of computer” use data; i.e. we can break down computer use by some specific applications.  That’s what I did.  The math achievement results disaggregated by response category follow:

[NOTE: click on images to enlarge them]

So, quite clearly, the same results appear as with the 4th grade data in my earlier post.  The group of students who never or hardly ever use computers score significantly higher than the other groups, across all applications.  Again, I don’t know anything about those students demographically.  But, still…

Going one step further, I ran a regression analysis with four of the independent variables (i.e. the “types” of uses).  The NDE would only allow me to use four; it’s a statistical/psychometric thing…don’t ask.  So, I took out word processing and drawing as those seemed likely the most remotely associated with math achievement.  The results are as follows (again, click on the image to enlarge):

Make sense? Yeah, I know, unlikely. Unless you are well versed and regularly practiced in regression analysis, there’s no reason that would make any sense to you. So, let me try to summarize some key results:

• Of all the variance in math achievement, differences in these four types of computer use for math account for 16%. That’s not that high; not terrible, but it’s safe to say that, overall, computer use for math does not explain much of why kids differ on their math scores.

The independent variables are “contrast coded” which is the right way to do this analysis. But, it limits what we can say. That being said,…

• The average score for a student who never or hardly ever uses computers in any of those ways is 291.
• Students who use the Internet for math once every few weeks score a bit higher than the previously mentioned student (i.e. never or ever uses in any of the ways).  That is, by simply adding Internet use for math once every few weeks adds a little bit to the average score of the non-computer using student.
• Same story for using graphing programs for charts.
• Adding Internet use once every few weeks AND graphing programs once ever few weeks has a cumulative positive effect on the non-computer using student (again, though, VERY small positive effect).
• The more frequently kids use math programs to drill on math facts, the lower they score.

So, there you have it.  I’ll likely play around a bit more with the NDE to see what else I find with respect to other subjects and other uses of computers.  Fun times!

Tags: achievement, math, NAEP, technology

There’s quite a bit of really good edublogging and commenting that lies at the intersection of “change” and “21st Century Schools” (see e.g. Chris Lehmann’s blog and Will Richardson’s blog).  As one who has studied extensively the research and literature on school reform, school change, policy implementation, etc., I’m having a hard time with a few aspects of the conversations that are going on.

First of all, what are we changing?  Are we talking about a classroom (i.e. changing one teacher’s pedagogy?)?  Are we talking about changing multiple classrooms?  Are we talking about changing a whole school?  Are we talking about changing the whole institution of public schooling?  Those are all very different scenarios and require very different approaches.  And, it gets back to the macro vs. micro distinction I made in my last blog post.

More importantly, though, let’s please consider that there is a HUGE, DEEP pool of research and literature on school change. There’s not a huge need to re-think this stuff; there’s a lot to be learned from what has already been learned.

In the mid 1970’s, the Rand Corporation conducted a national study of four federally funded programs “intended to introduce and support innovative practices in the public schools.”  The Rand researchers examined a sample of 293 local projects funded by these four federal programs in 18 states.  This so-called “Change Agent” study remains the paragon of all “implementation” studies.  According to Milbrey McLaughlin (1990), one of the principal investigators on the Change Agent study:

“…the following strategies generally were seen to be ineffective:

• reliance on outside consultants
• packaged management approaches
• one-shot, pre-implementation training
• pay for training
• formal, summative evaluation
• comprehensive, system-wide projects

The following strategies generally were effective, especially when applied in concert:

• concrete, teacher-specific and extended training
• classroom assistance from local staff
• teacher observation of similar projects in other classrooms, schools, or districts
• regular project meetings that focused on practical issues
• teacher participation in project decisions
• local development of project materials
• principals’ participation in training (p. 12)”

Since the 1970’s we’ve learned even more about change and policy implementation.  McLaughlin revisited the study in an article in 1990.  Her main conclusion there was that some of the findings of the original study needed to be reconsidered, but mostly, things remained the same.  “A general finding of the Change Agent study that has become almost a truism is that it is exceedingly difficult for policy to change practice, especially across levels of government (p. 12)”  In one study I conducted, I was able to determine that of all the variance in student computer use across the country, less than 2% could be accounted for by differences in state-level policies.  An additional study of the effects of state-level policies on pedagogy showed that 3% of the variance in teaching practices could be attributed to state-level polcies.  As the Change Agent study taught us, effective change in schools doesn’t happen by “adoption,” it happens by mutual adaptation; the adaptation of a project or policy and the organizational setting to each other.

So, what does this all mean?  Well, I think we’d all to well to internalize the bullets above as a list of what works.  First, one-and-done, sit-and-git PD doesn’t work; it has to be ongoing and as close to the classrooms/teachers as possible.  Second, teachers must be included in the change process, especially as learners.  Third, leadership must be involved at all stages and at all levels.  Finally, change is interpreted locally and the context of the institution adapts along with the change.

I also want to bring our attention to the final bullet in the list of ineffective strategies: comprehensive, system-wide projects don’t work.  We can’t change the system all at once.  So, it seems to me that School 2.0 is going to have to come about in one of two ways.  Either we get enough teachers and leaders to understand why change needs to happen in their schools and HOW that happens effectively (and what not to do).  Or, we go outside the “system.”

Tags: change, Rand, reform, school, systems

Well, other than “technology,” if I had to choose five tags to describe myself, sports, education and law would be in the top 5.  So, imagine my surprise when I was pointed to this video of a Malcolm Gladwell speech/presentation (what is it that he does exactly?) covering those three areas.  The main topic of his speech is the mismatch problem; the idea that in making hiring decisions employers regularly use metrics that are very poor predictors of success within their particular area of employment.  The substance of the presentation is certainly interesting, but here’s what I want to do with this video:

I want to use it as part of a major project for a doctoral level educational research course.  It’d be like a fact-checking exercise.  Students would have to listen to/watch the segment about hiring teachers and note each claim that Gladwell makes which is presumably research-based (i.e. that reducing class sizes from 22 to 16 will lead to increases in achievement of 5 percentile points).  Then, for each claim, they would have to find the research that either supports or refutes his claim.  The students would synthesize the research and write up their findings.  That would be fun/cool, right?

Tags: education, Gladwell, law, research, Sports

Scott McLeod deemed today Leadership Day, and so it is!  And so I go…

If you haven’t watched the video of Chris Lehmann’s presentation at NECC, there’s no question that it’s a must see.  I’m sure I’ll have lots of occasions to use it as a pedagogical tool with my ed. leadership students, especially as a model of instructional leadership.  The reviews of Chris’ preso have been through-the-roof high, and deservedly so.  Will Richardson used Twitter to suggest that we need to clone Chris, and Bud Hunt (aka Bud the Teacher) replied that he had secretly taken a few of Chris’ hairs for exactly that purpose.

For those who don’t know, Chris is the principal of the Science Leadership Academy; a magnet high school in Philadelphia that he founded/started a few years ago.  Because he is extraordinarily transparent (want to visit SLA; just ask!) and collaborative, and for at least one other reason I shouldn’t disclose, I’ve learned quite a bit about Chris and SLA.  And, as best I can tell, we really do need to clone Chris; we can’t have enough principals like him.

THAT ALL SAID, here’s the question…what would happen if we suddenly made Chris the principal of Frederick Douglass High School (NOTE: the school doesn’t even have it’s own website) in Baltimore (the subject of a recent HBO documentary which has been written about by me and others)?

You see, Chris admittedly had the luxury of starting a brand new school according to his (and presumably others’) incredible vision.  He got to self-select a whole faculty.  The school’s magnet status means that the students that attend, at some level, want to be there.  in fact, according to the school’s website,  “[a]dmission to SLA is based on a combination of a student interview at the school with a presentation of completed work, strong TerraNova scores, As and Bs with the possible exception of one C, teacher or counselor recommendation and good attendance and punctuality.” I know many, many principals who would drop everything to be able to select an entire faculty and work with already accomplished students.

But, there’s another thing that separates Chris from the vast majority of his principal peers.  Chris is an unrelenting progressivist and he has a true global, future-oriented vision.  Just read his recent blog post about progressive pedagogy for 21st century schools.

I know that not all schools like Douglass High are destined to fail.  I’ve seen and read parts of this book.  And, I know about the Achievement Alliance’s efforts to document success stories.  But, even there, if you read about the high school they spotlight, the school is unique in its geography and the “success” is having gone from 26% proficiency in one subject (ELA) to 42% proficiency over the course of 6 years.  That’s steady, but slow, improvement; but 42% is not exactly superior.

I’ve also followed closely the research and documentation of the 90/90/90 schools (90% low income, 90% minority, 90% proficiency).  Just about everything I’ve read about those schools (including this by Douglas Reeves) points to a blinding focus on standards, assessment, data-driven decision-making, etc.  For better or worse, there’s NOTHING progressive about those schools.

So, I wonder what would happen if we put Chris Lehmann in the hardest-to-staff schools; schools consistently failing to make adequate yearly progress.  I guess the question I’m asking is: Who wins?  The extraordinary progressive leader or the system?  Can a brilliant, extraordinary leader WITH A PROGRESSIVIST BENT truly reform a severely struggling school within the existing system of public education?

Personally, I think Chris, or someone like Chris, would do wonders in a school like Douglass High.  But, unfortunately, I think that remains an open (empirical?) question.  And, I’d love for us to be able to do that empirical work.  I would love to document the experiences of bright, extraordinary, progressive leaders who have proven successful in more comfortable situations attempting to completely turn around a failing school.  Please note, my interest is not how “good” someone like Chris is.  I want to know what effect “the system” has on someone as “good” and particularly as progressive as Chris.  If you know of any such experiences, let me know.

Tags: education, equity, leadership, leadershipday2008, progressivism, schooltechleadership

By now you’ve perhaps read the Time magazine article, seen the New York Times ad, and/or read/heard elsewhere about this “movement” dubbed “The Broader, Bolder Approach to Education” spearheaded by Lawrence Mishel, Helen Ladd, Pedro Noguera and Tom Payzant. The list of signatories reads like a who’s who of educational and/or social policy and is notably diverse (hey, even Deborah Meier and Diane Ravitch signed on together!).

If the full statement that this task force created looks familiar, it may be because these folks are clearly regular readers of my blog and ripped off almost all of my ideas from THIS POST nearly a month ago. OK, that’s probably not true, but I must admit that I feel sorta proud to have written a summary of my beliefs about education that overlaps so extensively with this “broader, bolder approach”. Consider the following comparison:

So, other than the fact that I’m a brilliant scholar of educational policy, what does this mean? Not much. And, overlap notwithstanding, I think this “bigger, bolder” approach does not go nearly far enough. For example, they write that the approach, “assigns value to the new knowledge and skills that young people need to become effective participants in a global environment, including citizenship, creativity, and the ability to respect and work with persons from different backgrounds.” Those are definitely important and worthy skills and dispositions (especially “creativity”), but what about 21st Century Skills such as digital literacy? I think it’s worth pointing out that there are no signatories that represent the ed. tech. community. No Tim Magner. No Don Knezek.

Also, I believe the approach is purposefully broad about outcomes, but I don’t see enough of an emphasis on learning.

I could go on, but I’ll stop now and wallow in my brilliance (-:

Tags: education, equity, outcomes, policy