Abstract
This project demonstrates the utility of introducing an Item Response Theory (IRT) methodology, specifically the Rasch model, to complement Classical Test Theory (CTT) in the development and calibration of a multiple-choice exam at a large State Civil Service Department. Given the psychometric developments and advances within IRT-methods and the prevalence of CTT in test development efforts conducted in State civil service, the current project was conducted to demonstrate how modern IRT-based methods could be used alongside CTT to address current limitations in practice and raise the standard of exam measurement quality for State-level multiple-choice exams. Using data from two administrations (N = 342; N = 202) of multiple-choice exams for the classification of Criminal Intelligence Specialist I, in addition to CTT-based item analyses, IRT-based analyses using the Rasch model were conducted to assess dimensionality and model fit to provide support for various sources of validity evidence, localize exam measurement precision, equate test forms, optimize exam length, and to identify items exhibiting Differential Item Functioning. Conclusions on the Rasch model’s feasibility, comparative ease of use, and challenges in implementation were addressed in order to advocate further use of the model.