Metacognition of Performance Raters.

Sixteen Family Practice faculty members completed ratings on 59 senior medical students after a 6-week primary care clerkship. Each student was rated by seven to ten faculty members and the chief residents who worked with them, resulting in a total of 353 ratings. The rating scale covered: (1) attainment of learning objectives; (2) progress during the clerkship; (3) overall performance, (4) frequency of contact between student and rater (number of patients discussed); and (5) confidence in the rating, to indicate raters' metacognition. A two-factor analysis of variance was performed on the results to explore the relationships among rater accuracy, level of contact, and rater confidence in the score assigned. It was concluded that confidence in the validity of a rating was not related to the accuracy of that rating. Level of rater-student contact was, however, related to accuracy, with the most accurate ratings based upon discussion of seven to eleven patients. Low levels of contact were associated with overly stringent ratings, and high levels of contact were associated with lenient ratings. Individual raters differed in the leniency of scores, the tendency to make extreme judgments, and confidence in each rating. (GDC)