Educational Insanity

One of my posts from almost four months ago has been resurrected by comments from Tina K. and Amir.  In that post, I suggested I’d do some more digging.  So, I dug.

Some background…these are NAEP data with tables and statistics generated by the NAEP Data Explorer.  The NDE is an awesome (free!) tool for analyzing NAEP data.  It’s kinda amazing to me that more folks, including the media, haven’t picked up on this tool to do some really quick and easy data analysis.  Anyway, to satisfy the inquiries of Amir and particularly Tina, I analyzed 8th grade data from the 2007 NAEP administration.  The 8th grade assessment includes the best “type of computer” use data; i.e. we can break down computer use by some specific applications.  That’s what I did.  The math achievement results disaggregated by response category follow:

[NOTE: click on images to enlarge them]

So, quite clearly, the same results appear as with the 4th grade data in my earlier post.  The group of students who never or hardly ever use computers score significantly higher than the other groups, across all applications.  Again, I don’t know anything about those students demographically.  But, still…

Going one step further, I ran a regression analysis with four of the independent variables (i.e. the “types” of uses).  The NDE would only allow me to use four; it’s a statistical/psychometric thing…don’t ask.  So, I took out word processing and drawing as those seemed likely the most remotely associated with math achievement.  The results are as follows (again, click on the image to enlarge):

Make sense? Yeah, I know, unlikely. Unless you are well versed and regularly practiced in regression analysis, there’s no reason that would make any sense to you. So, let me try to summarize some key results:

• Of all the variance in math achievement, differences in these four types of computer use for math account for 16%. That’s not that high; not terrible, but it’s safe to say that, overall, computer use for math does not explain much of why kids differ on their math scores.

The independent variables are “contrast coded” which is the right way to do this analysis. But, it limits what we can say. That being said,…

• The average score for a student who never or hardly ever uses computers in any of those ways is 291.
• Students who use the Internet for math once every few weeks score a bit higher than the previously mentioned student (i.e. never or ever uses in any of the ways).  That is, by simply adding Internet use for math once every few weeks adds a little bit to the average score of the non-computer using student.
• Same story for using graphing programs for charts.
• Adding Internet use once every few weeks AND graphing programs once ever few weeks has a cumulative positive effect on the non-computer using student (again, though, VERY small positive effect).
• The more frequently kids use math programs to drill on math facts, the lower they score.

So, there you have it.  I’ll likely play around a bit more with the NDE to see what else I find with respect to other subjects and other uses of computers.  Fun times!

1. Dave says:

   July 24, 2008 at 5:03 pm

   “The more frequently kids use math programs to drill on math facts, the lower they score.”

   Does that data show causation or correlation?

   It seems EXTREMELY likely to me that low-scoring students would be the ones that educators are trying new techniques with…like computer. If students are already performing well, there’s no need to try to bring computers into the mix.

   To get the numbers you’re looking for, I would think you’d need to talk a population of students, split it down the middle into control group or computer users, and monitor both group’s progress. Even then, it would be hard to balance out the fun factor of using computers.
2. Dave says:

   July 24, 2008 at 5:05 pm

   computer -> computers
   talk -> take
   group’s -> groups’
   …and probably others.

   Will computers help me proofread before I submit?
3. Jon Becker says:

   July 24, 2008 at 5:56 pm

   Actually, I’m doing some proofreading of my own…my interpretation is not exactly right…I’ll update now and then comment, Dave…
4. Jon Becker says:

   July 24, 2008 at 9:21 pm

   Now that I got the interpretation more accurate…

   Dave, first…regression analysis is a form of multivariate analysis. So, it shows multiple correlations simultaneously. You will, never or hardly ever know about causation unless there’s a true experiment (which is really hard to do in education). Secondly, I don’t know about trying new computer-related pedagogies with those who’ve scored the lowest on prior math measures. I often see quite the opposite. Kids are struggling so the assumption is that they need the basics; i.e. not those new-fangled computers. But, it is an empirical question that I think needs to be answered.
5. Marshall says:

   July 24, 2008 at 10:38 pm

   Jon
   Long ago, I called you “smart” for some reason. Now you throw this statistical analysis at me and I know I under rated you. Although I’m not really a stat person, I like the graphs and charts and enjoy seeing the trends. Whether statistically significant or not, the data you have indicate a reverse relationship (causation not identified, I understand). Since we are tossing out hypothesis for debate, could the raw scores be negatively effected because our students (and WE) get lazy because we have tools and forget the basics that may cause us errors. On math assessments this can be “forgot to carry the 1″ type of thing, or Dave noted an example of our reliance on spellcheck. Been there, done that. If so, do we need to keep this in mind and legitimize long-hand math and proofreading. If not, thts fin as longg as yoo dont minnd my typnig.
6. Ed says:

   July 25, 2008 at 10:17 am

   Dumb question: Is the test taken on a computer, or paper?
7. Jon Becker says:

   July 25, 2008 at 11:19 am

   NAEP is paper-based.

   I have some more data I’ll share today that might shed some light on these findings…
8. Ruben says:

   November 6, 2008 at 12:28 am

   Interestingly was, but there is someone who does not quite agree with the author?
9. romonoeroetoko says:

   July 8, 2009 at 12:26 pm

   Hm that sounds good but I would like to know more details.
10. amenodimeno says:

    July 31, 2009 at 2:05 pm

    That’s good man, keep it going.
11. amenodimeno says:

    August 5, 2009 at 7:28 pm

    Good story for me but please more details.
12. queroeropoo says:

    August 6, 2009 at 7:30 pm

    Good information to me.
13. John Easo says:

    August 15, 2009 at 4:23 am

    There is a great set of Math study material here… more importantly the online collaboration learning platform allows group contribution to create useful flashcards quickly and seamlessly!
14. John Easo says:

    August 15, 2009 at 4:24 am

    Here’s the link to the correct website – online collaboration learning platform
15. adamoerikom says:

    September 19, 2009 at 5:38 pm

    Stunning blog and good article. High 5 for u man !