Pedagogical Improvement
assessment, higher ed., teaching July 21st, 2008
If you’ve ever taken a college or graduate level course, surely you’ve completed some kind of summative evaluation form at the end of the semester. At Hofstra University, where I worked for 5 years before this past academic year, we called them CTRs (Course and Teacher Ratings). They consisted of a bunch of Likert scale items (strongly disagree to strongly agree) and a few open-ended questions. For the most part, students hated doing them and faculty members hated having to use them. I didn’t love the wording of many of the items, but I always asked my students to please take them seriously as an opportunity to let me know how I was doing. I told them that I would receive an analysis of the data and their actual responses to the open-ended items.
As part of applying for tenure at VCU, I have to demonstrate growth as an instructor. So, I plugged the CTR data from my 5 years at Hofstra into EXCEL and discovered some very interesting things. The graph below represents the data from a scale (composed of 5 items) that purports to be an overall measure of the course and the instructor. The x-axis represents the time points from Fall 2002 to Spring 2007. The y-axis represents the range of scores (which can range from 1 to 5). For this particular scale, the lower the number the better. But, I flipped the y-axis so that it looks like “better is higher;” a more standard look for such a line graph. The blue line represents my ratings; the red line represents the average score of the other faculty members (including adjuncts) within the program area.
[NOTE: click on image for larger view]
I entered the professoriate with NO teaching experience. I guest lectured once while I was getting a masters degree, but that was it. Hofstra took a bit of chance on me in that respect and I am eternally grateful to them for that. But, the graph clearly shows that my ratings were not as good early in my teaching career as they were last year.
I should also add that in my first couple of years as a professor, i was asked to teach a few sections of an undergraduate foundations of education course. I thought I would really enjoy working with undergraduates considering a future as an educator. But, after teaching a few semesters, I began to really dislike it. I had a hard time dealing with the students’ limited understanding of and experiences with education. Seemingly simple concepts such as “charter schools” were completely foreign to them. My ratings were not terrible for those course sections, but my department chair and my colleagues and I decided that my time and energy was better spent working with graduate students.
Overall though, I think the graph tells an accurate and interesting story. Quite simply, I’ve improved significantly as an instructor. The more comfortable I’ve become in my own skin and the more I’ve been able to find my own voice, the more I’ve been able to engage my students. That’s my interpretation of the data.
Academics bemoan the use of “quantitative” ratings of their work as instructors. But, I think it’s critically important that we ask our students to reflect on their experiences in our classes and to provide us with data about our work. I wonder how many of my P-12 colleagues/readers have similar systems in place to collect and analyze summative or formative data about their performance directly from their students. Do you?
Tags: data, evaluation, teaching
This is an interesting topic! I loved getting to do professor evaluations at the end of semesters in college because I felt like I was making a difference and actually making the university better for future students. I was honest and constructive. This is where things get a bit trickier in the P-12 world - can feedback be taken as seriously from younger students? I’m NOT saying I don’t respect the opinion of young students, because I do - I’ve had students do evaluations in every class I taught and had students do ongoing reflections on the overall class experience on our class blog this past year.
However, I think that anonymous feedback in the younger grade levels might not be as useful as in the university setting. I hate how this sounds, because I KNOW there are a lot of great kids out there, but I do think that university students are more apt to take the evaluations seriously.
Am I wrong on this? Completely open to debate and would love to hear numbers telling me I’m TOTALLY off base!
@ Kate,
I worked in a district where everyone had to do this, K-12, and it was included in the teacher’s evaluative instrument. Obviously the instruments varied along with the questions depending upon the level. We staffed this with people for each level who would compile the data and send the reports to the teachers and the administrators. In addition we sent surveys to random parents to assess how they felt about their child’s teachers(and administrators). We based it on the idea you need 360 degree feedback to determine how you are REALLY doing the job. For the most part it told us the things we already knew from direct observation. Effective teachers are evaluated higher by their students, unless it is a PE teacher and then everyone hates them for some reason.
We stopped the program after three years due to monetary, teacher associations and administrative concerns. Our really excellent teachers would obsess over the one negative comment and just not be able to move on and realize you cannot please everyone. The one consistent item was how valuable everyone found the feedback regardless of the grade level of the student. The customers always know the answer the only question was did we have the courage to ask.
Jon, as a student, I ran the Faculty/Course Evaluation system at Carnegie-Mellon. I got the job as a service project for Alpha Phi Omega, the national service fraternity, and passed it on as a university run system. It has been interesting to me to note the changes from the system I inherited to the one (now rebranded) today. Then it was voluntary and for students’ use; now its part of the evaluation/promotion/teaching quality improvement process.
Yet, a professor at the University of Oklahoma informs me that even today, they have no such system there!
In thinking about this applied to k12, I’m recalling that normally, instructors did not see the raw data. They were to place them unseen in the envelope, and return them. We would summarize comments for publication, but generally not provide all the gory details.
And, perhaps in K12, parents should fill them out?
Just got done listening to a press conference from Michelle Rhee, where she again describes her non-stop efforts to hear what students, parents, and teachers in her 50,000 student charge have to say. Customer feedback–What a concept!
Thanks, Ed, for that information. I’d never really considered the history of the system. I do, however, regularly think about the politics of it. At Hofstra, faculty were part of a collective bargaining unit. So, the CTRs were very much a negotiated instrument and process. There are no unions here in VA though. I do imagine, then, that teacher unions would be very much part of any student-teacher evaluation process.
It’ll be interesting, as it always is in D.C., to see how Rhee does as Supt.
Parent input in the elementary survey could replace that of young students. Teaching staff would also be very beneficial in survey data. If done professionally a survey could be a worthwhile tool for the teacher involved but also for the administrator and school system. Not as a negative entity but as a building up and reaffirming of a pedagogaical process.