cover image: Detecting Intrajudge Inconsistency in Standard Setting Using Test Items with a Selected-Response Format. Research Report.

Detecting Intrajudge Inconsistency in Standard Setting Using Test Items with a Selected-Response Format. Research Report.

In judgmental standard setting experiments, it may be difficult to specify subjective probabilities that adequately take the properties of the items into account. As a result, these probabilities are not consistent with each other in the sense that they do not refer to the same borderline level of performance. Methods to check standard setting data for intrajudge inconsistencies are thus of paramount importance to setting meaningful standards. This paper presents a method of consistency analysis for standard setting experiments in which judges specify probabilities for each response alternative of the items. The method is based on a residual diagnosis of the subjective probabilities under the hypothesis of a consistent judge to the probabilities. An empirical example shows how the method can be used to identify sources of inconsistency in response alternatives, items, or judges. (Contains 19 references.) (SLD)

Authors

van der Linden, Wim J., Vos, Hans J., Chang, Lei

Related Organizations

Chinese University of Hong Kong
Authorizing Institution
Twente Univ., Enschede (Netherlands). Faculty of Educational Science and Technology.
Peer Reviewed
F
Publication Type
Reports - Descriptive
Published in
United States of America

Table of Contents