Abstract
The purpose of this project was to create an electronic system for collecting ratings during a standard setting workshop and assess its worth. The electronic system was compared in time and efficiency to traditional methods to evaluate the electronic system as an acceptable alternative, and a travel cost analysis was conducted to show potential savings. This research conducted in-person workshops using the modified-Angoff standard setting procedure with a connected system of Google sheets. One workshop was conducted using typical protocols for the organization and was included as a comparison group. Primary measures were time used, reliability of ratings, number of raters, and informal feedback. Results showed the electronic system was equivalent to established methods in time and reliability. Social distancing requirements imposed by the COVID-19 outbreak further demonstrated the utility of the electronic process as it can be conducted with more distance between participants. Finally, the travel cost analysis estimated savings up to 250K for the Department of Consumer Affairs.