Continuing my sniffing through the NAEP Data Explorer, today I “explored” differences in digitally-infused pedagogy by race.  One of the items on the background questionnaire of the 8th grade NAEP in 2007 was as follows: “When you are doing math for school or homework, how often do you use these different types of computer programs?”  One of the listed programs was “A program to practice or drill on math facts (addition, subtraction, multiplication, division).”   Looking at the results for that item disaggregated by race, we get the following (click on image to enlarge):

Overall, African-American students are much more likely to use computers to practice or drill on math facts than White students.  Given the significant achievement gap that exists, these differences partly explain why, overall, the there is a negative correlation between using computers to practice or drill on math facts and math achievement.  I can’t be entirely sure about the degree to which race confounds that overall relationship without access to the raw (restricted-use) NAEP data.

But, more importantly, is the figure above problematic?

Tags: computers, drill, math, NAEP

One of my posts from almost four months ago has been resurrected by comments from Tina K. and Amir.  In that post, I suggested I’d do some more digging.  So, I dug.

Some background…these are NAEP data with tables and statistics generated by the NAEP Data Explorer.  The NDE is an awesome (free!) tool for analyzing NAEP data.  It’s kinda amazing to me that more folks, including the media, haven’t picked up on this tool to do some really quick and easy data analysis.  Anyway, to satisfy the inquiries of Amir and particularly Tina, I analyzed 8th grade data from the 2007 NAEP administration.  The 8th grade assessment includes the best “type of computer” use data; i.e. we can break down computer use by some specific applications.  That’s what I did.  The math achievement results disaggregated by response category follow:

[NOTE: click on images to enlarge them]

So, quite clearly, the same results appear as with the 4th grade data in my earlier post.  The group of students who never or hardly ever use computers score significantly higher than the other groups, across all applications.  Again, I don’t know anything about those students demographically.  But, still…

Going one step further, I ran a regression analysis with four of the independent variables (i.e. the “types” of uses).  The NDE would only allow me to use four; it’s a statistical/psychometric thing…don’t ask.  So, I took out word processing and drawing as those seemed likely the most remotely associated with math achievement.  The results are as follows (again, click on the image to enlarge):

Make sense? Yeah, I know, unlikely. Unless you are well versed and regularly practiced in regression analysis, there’s no reason that would make any sense to you. So, let me try to summarize some key results:

• Of all the variance in math achievement, differences in these four types of computer use for math account for 16%. That’s not that high; not terrible, but it’s safe to say that, overall, computer use for math does not explain much of why kids differ on their math scores.

The independent variables are “contrast coded” which is the right way to do this analysis. But, it limits what we can say. That being said,…

• The average score for a student who never or hardly ever uses computers in any of those ways is 291.
• Students who use the Internet for math once every few weeks score a bit higher than the previously mentioned student (i.e. never or ever uses in any of the ways).  That is, by simply adding Internet use for math once every few weeks adds a little bit to the average score of the non-computer using student.
• Same story for using graphing programs for charts.
• Adding Internet use once every few weeks AND graphing programs once ever few weeks has a cumulative positive effect on the non-computer using student (again, though, VERY small positive effect).
• The more frequently kids use math programs to drill on math facts, the lower they score.

So, there you have it.  I’ll likely play around a bit more with the NDE to see what else I find with respect to other subjects and other uses of computers.  Fun times!

Tags: achievement, math, NAEP, technology

The full RFP has not been issued yet, but the feds put out a pre-solicitation notice for: “Conducting a Special Study in Technological Literacy for the 2012 National Assessment of Educational Progress.”  In other words, as part of the 2012 administration of the NAEP, there will be an assessment of technological literacy (presumably of 4th, 8th and 12th graders).  The National Assessment Governing Board, the lead NAEP agency, will be looking for a contractor to develop the measures. 

I was asked by a colleague if I’d be interested in bidding when the RFP is released.  I’m hesitant for one main reason.  First, the RFP isn’t released yet and the notice indicates that proposals will be due July-August 2008.  So, that’s a 3-month turnaround at best.  That would indicate to me a less-than-open competition; nobody not totally in-the-know could put together a winning proposal in such a short period of time.  My guess is that the proposal will be written with certain folks in mind.  Who?  Well, any or all of the members of this committee.  Those are the folks responsible for compiling this really interesting resource: Tech Tally: Approaches to Assessing Technological Literacy.

  You can read the whole book online at the link above (or buy it, of course).

I also wonder, will “technological literacy” mean something different in 2012 than it does today?  If so, how do we develop measures of something unknown?

Tags: assessment, NAEP, technological literacy

This is what I get for playing around with the NAEP Data Explorer: 

So, the use percentages are a bit depressing.  But, it gets worse…much worse…

This is 4th graders, 2007, and, yes…the score for the group of students who report daily or almost daily use of computers at school for math is (statistically significantly) lower than all of the other groups.

The smart thing may have been to tuck these findings in my back pocket and walk away.  But, I’d bet that if we controlled for a bunch of demographic variables and even a baseline math score, those differences would disappear.  I’ll have to dig a little further…

Tags: achievement, computers, math, NAEP

OK, one more “finding” (see post below).  The figure below points out a trend that shows up in all of the data I’m analyzing, and the trend that led me to the phrase “digital horserace.”  You’ll notice that the states that are above the national average in student use of the Internet for math are all in the Southeast.  Additionally, most of the states in the Northeast are below the national average.  My interpretation is that, in large part, one or two states jumped on the ed. tech. bandwagon early and then their “neighbors” developed a need to “keep up” (hence the horserace).  I know, for example, that West Virginia made huge commitments to ed. tech. a long time ago.  That led Kentucky to jump aboard the bandwagon along with, eventually, other states in the Southeast.

Anyway, some numbers to go with the graphic.  Among the sample of fourth-grade students across the nation, 42% said that they did use the Internet to learn things about math (58% said no).  The state with the highest percentage of students who reported using the Internet to learn things about math?: Louisiana (56%).  That state with the lowest percentage?: Vermont (26%).

Tags: computers, education, Internet, math, NAEP, technology