Measurement precision at the cut score in medical multiple choice exams: Theory matters

Abstract
Introduction In high-stakes assessment, the measurement precision of pass-fail decisions is of great
importance. A concept for analyzing the measurement precision at the cut score is conditional reliability, which describes measurement precision for every
score achieved in an exam. We compared conditional
reliabilities in Classical Test Theory (CTT) and Item
Response Theory (IRT) with a special focus on the cut
score and potential factors influencing conditional
reliability at the cut score.
Methods We analyzed 32 multiple-choice exams from
three Swiss medical schools comparing conditional
reliability at the cut score in IRT and CCT. Additionally, we analyzed potential influencing factors such as
the range of examinees’ performance, year of study,
and number of items using multiple regression.