Learner and Self Ratings
Cashin (1995) shows that learner’s ratings can correlate well with external measures of their learning and with the instructor’s self-ratings. Student ratings are statistically reliable (they have internal stability and are consistent over time), are more statistically reliable than are "colleague ratings" and are not easily or automatically manipulated by grades. Also see L'Hommedieu, Menges, and Brinko, (1990); d’Apollonia & Abrami, (1997); Ory, Braskamp, & Pieper, (1980); and Centra, (1993).
Imagine, your learners are better able to rate you than your fellow instructor (Dancer, & Dancer, 1992)…
Donnelly and Woolliscroft (1989) reviewed student ratings, using 12 descriptive items over a period of a year and concluded that the learners’ evaluations were reliable and that their judgements were sophisticated and well thought-out.
Also, intellectually challenging classes average higher ratings than do easier courses with light work loads (Cohen, 1981). This closely relates to Snow’s (1988) research in which a learner's future potential depends upon his or her current cognitive state and that we can increase the learner’s potential by increasing the standards (aptitude x treatment interaction). Even the learners know when they are being challenged and they appreciate it!
What Should We Ask?
Part of the problem is that we do not always know what to ask. These types of questions produce the most reliable results (Abrami, 1989 & Cashin, 1992):
 |
Overall rating of the instructor. |
|
Giving an overall rating of the course. |
|
Estimating how much a learner has learned in the course. |
|
Rating the effectiveness of the instructor in stimulating a learner’s interest in the subject. |
|
Rating the effectiveness of this course in challenging the learner intellectually. (This is important if we are supposed to stimulate continued learner interest and provide positive influence on the ways learners think and act.) |
Do not ask questions on teaching methods -- one might get high marks on "how much the learners have learned" (which tend to be valid) and low marks on "the course was carefully planned and well organized" (which tend not to be valid). Even if these "Method or Process" questions were valid, they do not tell us anything that the "Result" questions cannot.
In the questions above, note that the word "rating" and not "evaluation" is used. Rating implies a source of data, while evaluation implies that we have an answer. That is, the learners provide us information and then we combine it with other sources of information to arrive at a total evaluation. Learners are not always on target, thus their ratings can provide valuable information, but they cannot always tell evaluators everything needed in order to make a valid assessment of the training.
Perhaps the most unreliable question is, "How much did you enjoy the class?" Learners generally enjoy courses that are the most intellectually challenging and meaningful. Yet, they will also report that they may enjoy a class that contributes little to their learning. Nevertheless, when the same learners are asked to assess their learning, provide a rating of the instructor and/or course, or to assess its intellectual contributions, the students, as a whole, are able to distinguish "fluff" from "substance" (Kaplan, 1974; Naftulin, 1973).
Learner Bias
Prior learner interest in a subject does influence the outcome of student ratings of effectiveness (Marsh & Dunkin, 1992). For example, a trainer taking a train-the-trainer class will probably give a higher rating, than if she was taking a class in which she had no real interest.
Also, learners do not give higher ratings to classes in which they receive the highest grade (Howard & Maxwell, 1980). Again, the highest marks often go to the most challenging courses. However, a learner’s ratings tend to be slightly higher if a learner "expects" to receive higher grades -- the research suggests that the differences is due to the learner being highly "motivated" and he or she is "learning more" and can thus expect to get higher grades (Howard G. & Maxwell, 1982).
Immediate Feedback
To collect immediate feedback, end the session five minutes early and ask: 1) What major conclusion did you draw from today's session? 2) What major questions remain in your mind? (The Searle Center for Teaching Excellence Northwestern University).
These two questions 1) assist learners to ask themselves extremely valuable questions (what have I learned; what do I need to learn now) and 2) provide the trainers with feedback, such as discovering that learners are drawing conclusions quite different from the ones intended. If so, you can begin the next session with responses to the patterns that emerge, or make adjustments in the way you train.
Also, learners who have no previous experience have the most inconsistent feedback. This is partially because they have nothing to base their initial feedback on. By using the two questions above in multiple training sessions, you help them in "scaffolding" their feedback so that they may improve upon it (same principle as in scaffolding instruction).
Self-Assessment
Traditional testing methods do not fit well with such goals as lifelong learning, reflective thinking, being critical, the capacity to evaluate oneself, and problem-solving (Dochy & Moerkerke, 1997). For these, self-assessment plays an important role. Self-assessment refers to the involvement of learners in making judgements about their own learning, particularly about their achievements and the outcomes of their learning (Boud & Falchikov, 1989). It increases the role of learners as active participants in their own learning (Boud, 1995), and is mostly used for formative assessment in order to foster reflection on one's own learning processes and results.
"Overall, it can be concluded that research reports positive findings concerning the use of self-assessment in educational practice. Students who engage in self-assessment tend to score most highly on tests. Self-assessment, used in most cases to promote the learning of skills and abilities, leads to more reflection on one's own work, a higher standard of outcomes, responsibility for one's own learning and increasing understanding of problem-solving. The accuracy of the self-assessment improves over time. This accuracy is enhanced when teachers give feedback on students' self-assessment" (Dochy, Segers, & Sluijsmans, 1999).
Boud (1992, 1995) developed a self-assessment schedule in order to provide a comprehensive and analytical record of learning in situations in which students had substantial responsibility for what they did. The main guidance was a handout that suggested the headings a student might use -- goals, criteria, evidence, judgements, and further action.
Weaker learners often overrate themselves. Adams & King (1995) identified a 3 step framework to help develop self-assessment skill: 1) Learners work on understanding the assessment process, such as: discussing good and bad characteristics of sample work, discussing what was required in an assessment, and critically reviewing the literature. 2) Learners work to identify important criteria for assessment. 3) Learners work towards playing an active part in identifying and agreeing on assessment criteria and being able to assess peers and themselves competently.
Another assessment framework looks at the various dimensions (Garfield, 1994):
- WHAT to assess, which may be broken down into: concepts, skills, applications, attitudes, and beliefs.
- PURPOSE of assessment: why the information is being gathered and how the information will be used (e.g., to inform students about strengths and weaknesses of learning, or to inform the teacher about how to modify instruction).
- WHO will do the assessment: the student, peers (such as members of the student's work group), or the teacher. They need to be given opportunities to step back from their work and think about what they did and what they learned.
- METHOD to be used (e.g., quiz, report, group project, individual project, writing, or portfolio).
- ACTION that is taken and the nature of the action.
- FEEDBACK given to students. This is a crucial component of the assessment process that provides the link between assessment and improved student learning.
Trainer Bias
"Negative attitudes toward student ratings are especially resistant to change, and it seems that faculty and administrators support their belief in student-rating myths with personal and anecdotal evidence, which [for them] outweighs empirically based research evidence" (Cohen - reported in Cashin, 1992).
"The research on student SETEs [Student Evaluations of Teacher Effectiveness] has provided strong support for their reliability, and there has been little dispute about it" (Hobson & Talbot, 2001).
The learner’s rating will serve their purpose if 1) you learn something new from them, 2) you value the information, 3) you understand how to make improvements, 4) you are motivated to make the improvements (Centra, 1993).
Page 6 Changing Behaviors
