If you’ve ever taken a college or graduate level course, surely you’ve completed some kind of summative evaluation form at the end of the semester.  At Hofstra University, where I worked for 5 years before this past academic year, we called them CTRs (Course and Teacher Ratings).  They consisted of a bunch of Likert scale items (strongly disagree to strongly agree) and a few open-ended questions.  For the most part, students hated doing them and faculty members hated having to use them.  I didn’t love the wording of many of the items, but I always asked my students to please take them seriously as an opportunity to let me know how I was doing.  I told them that I would receive an analysis of the data and their actual responses to the open-ended items.

As part of applying for tenure at VCU, I have to demonstrate growth as an instructor.  So, I plugged the CTR data from my 5 years at Hofstra into EXCEL and discovered some very interesting things.  The graph below represents the data from a scale (composed of 5 items) that purports to be an overall measure of the course and the instructor.  The x-axis represents the time points from Fall 2002 to Spring 2007.  The y-axis represents the range of scores (which can range from 1 to 5).  For this particular scale, the lower the number the better.  But, I flipped the y-axis so that it looks like “better is higher;” a more standard look for such a line graph.  The blue line represents my ratings; the red line represents the average score of the other faculty members (including adjuncts) within the program area.

[NOTE: click on image for larger view]

I entered the professoriate with NO teaching experience.  I guest lectured once while I was getting a masters degree, but that was it.  Hofstra took a bit of chance on me in that respect and I am eternally grateful to them for that.  But, the graph clearly shows that my ratings were not as good early in my teaching career as they were last year.

I should also add that in my first couple of years as a professor, i was asked to teach a few sections of an undergraduate foundations of education course.  I thought I would really enjoy working with undergraduates considering a future as an educator.  But, after teaching a few semesters, I began to really dislike it.  I had a hard time dealing with the students’ limited understanding of and experiences with education.  Seemingly simple concepts such as “charter schools” were completely foreign to them.  My ratings were not terrible for those course sections, but my department chair and my colleagues and I decided that my time and energy was better spent working with graduate students.

Overall though, I think the graph tells an accurate and interesting story.  Quite simply, I’ve improved significantly as an instructor.  The more comfortable I’ve become in my own skin and the more I’ve been able to find my own voice, the more I’ve been able to engage my students.  That’s my interpretation of the data.

Academics bemoan the use of “quantitative” ratings of their work as instructors.  But, I think it’s critically important that we ask our students to reflect on their experiences in our classes and to provide us with data about our work.  I wonder how many of my P-12 colleagues/readers have similar systems in place to collect and analyze summative or formative data about their performance directly from their students.  Do you?

Tags: data, evaluation, teaching

There’s an argument/discussion/meme bouncing around the blogosphere that is RIGHT in my wheelhouse.  I learned of it through eduwonkette and she provides the links in the chain.  Since I’ve written about this before, I’ll chime in again…

The discussion is based on the premise that judging teachers using a value-added analysis is the equivalent of a system that judges baseball players using a sabermetric analysis.  Actually, even that summary is not entirely accurate because what Michael Lewis wrote about in Moneyball is not so much a system or form of analysis as it is an orientation.  Billy Beane, J.P. Ricardi, Theo Epstein and all of the other new baseball G.M.’s who’ve espoused Moneyball-like approaches, believe that there are inefficiencies in the market.  That is, there are data available that, when subjected to appropriate statistical analyses, might surface players who are undervalued based on more traditional forms of judgment.  These forms of player valuation, when added to the more traditional/standard methods, allow teams with less resources and less payroll flexibility to find players who are most cost-effective. 

The key there is the italicized segment.  Critics of the Moneyball/Sabermetric approach (including Murray Chass) have the wrong impression that it is intended to replace more traditional/standards of player valuation.  That’s simply not true.  Sabermetric analysis is an additional weapon in the arsenal of baseball decision-makers. 

Applying that back to the proposal by New York City proposal, I have no problem using value-added analysis as an additional measure to hold teachers (and principals?) accountable.  It should never be the single criterion for any important decision about an individual teacher.  But, the data are there, so let’s look at them.  As one principal says in the NY Times article:  “This should simply be one more way to think about things,” said Frank A. Cimino, the principal of P.S. 193 in Brooklyn, who said he was participating in the experiment. “It is going to tell you some things you don’t know, but it will miss the other things that go on in a classroom.”

Tags: accountability, education, evaluation, Moneyball, sabermetrics, teachers

 In the evaluation work I do, I’m occasionally asked to study/assess the cost effectiveness of a “product.”  This makes some sense where the product is hardware or some kind of proprietary digital object.  And, there, COSN’s work on Total Cost of Ownership is exemplary.  But, when it comes to many software programs, even comprehensive programs, things have changed.  In the day and age of Moodle and other free, open-source programs, cost effectiveness is moot.  I’m not tapped in to the open source in education community, but I’d love to know what prevents open source from making a bigger impact in education.  At a time when the economy is slow and budgets are tight, how do schools/districts continue to justify ed. tech. expenditures where equally good and free alternatives exist?

Tags: budgets, cost effectiveness, education, evaluation