Right now, thousands of teachers across the country are grading writing assignments or providing feedback on student work. Contrary to what you might think, these grades and feedback are not solely based on the quality of the student’s work, but also on the teacher’s unique life experiences, personal beliefs or personal interpretation of what constitutes “good writing.”
For instance, one teacher may prioritize grammar, while another cares more about the overall effort than the right answer. Some teachers may feel everyone should get an “A”, whereas others think “As” are reserved for the most exemplary assignments. This phenomenon is known as “rater effect”—or the effect of the individual doing the rating on the resulting score.
The rater effect also comes into play when teachers make subjective judgments during the gifted and talented identification process. In those cases, teachers are asked to rate their students on specific characteristics or behaviors. However, what emerges from this evaluation is not merely an objective assessment but rather a subjective perception filtered through the teacher’s individual lens.
Recently, as part of a five-year grant from the Institute of Education Sciences to evaluate the impact of teacher rater effect on gifted and talented identification, I worked amongst a team of researchers to collect data from over 50,000 students from five districts across the country. All the teachers in these districts rated their students using some form of gifted and talented teacher rating scale. The research team evaluated how much of a student’s rating depended on their teacher.
In an ideal world, none of the ratings should be teacher-dependent. But we suspected that wouldn’t be the case. And we were right—between 10% and 20% of a student’s rating on these identification scales could be attributed to the teacher conducting the rating rather than the student’s actual abilities.
To explain how big of an effect this is, we modeled what would happen if a student in a higher-rating teacher’s class was hypothetically rated by a lower-rating teacher. Overall, after combining the teacher rating data with ability and achievement scores, approximately 20% of students would no longer be identified as gifted. That is a major difference. Whether or not students are identified as gifted contains another element of chance: if they were assigned a teacher who tended to give out higher or lower ratings.
What can schools do about this? First, schools should critically evaluate whether teacher judgment is necessary to include in the identification process. Is there an alternative, more direct and less biased method to gather essential information about students’ abilities?
Second, if some level of teacher input is required, it should never serve as the sole data point in placement decisions. Human diversity is a wonderful thing, but it does make consistent ratings difficult to achieve. Additionally, districts using a two-phase identification system, should not use a rating scale as a universal screener.
Third, when using such rating scales as part of the identification system, try to get all teachers on board. Similar to any grading rubric, the more consistent raters are in how they’re interpreting the criteria, the less the individual doing the rating influences the resulting score. Offering a quick workshop on how to complete the rating scale can help.
If there are teachers who consistently score most of their students noticeably higher or lower than their colleagues, it may be useful to work with them to understand how to assign more consistent ratings. Similar to tuning an instrument, teachers need to be aligned in their understanding of the rating scale.
Teachers can be a great source of information about students, but they do bring their unique perspective. Usually, this is the point. However, when completing rating scales on students, the teacher’s uniqueness should take a back seat so that unique student qualities can shine through.
