Abstract
To meet the needs of America’s growing science industry, national reports have called for reform in science education, including the way instructors assess their students. Research performed in the 1920s and 1930s encouraged the use of the then-new “choice-type” assessments (e.g. multiple-choice), showing them to be equally reliable as open-ended formats in assessing student learning. With the emergence of newer perspectives in cognition in the 1980s, researchers also began to question not just the reliability of an assessment, but an assessment’s ability to accurately evaluate multiple aspects of aptitude, including critical thinking. This prompted the development of more vigorously validated multiple-choice assessments known as concept inventories. Despite vigorous validation, concept inventories may be more limited in their ability to assess measures, such as critical thinking, than other formats, such as the oral exam. This study compared three assessments formats – multiple-choice, written, and oral – in an undergraduate anatomy and physiology course. The study (1) revisited the question of reliability by quantitatively comparing student scores across the three formats, (2) determined if any demographic variables were associated with potential differences in student performance, (3) identified student difficulties in the written and oral formats, and (4) examined student attitudes towards the written and oral formats through the use of a Likert-style survey. Question sets on the topics of hemodynamics and hemoglobin dissociation were validated and developed to be administered in either a multiple-choice, written, or oral format. Quantitative analyses – Kruskal-Wallis, Mann-Whitney U, correlations, and mean absolute differences – showed differential scoring between the multiple-choice and either of the open-ended formats, despite identical questions. Most quantitative analyses showed equal performance between the written and oral formats, although mean absolute differences showed slight, yet significant, differences. Two-way ANOVA analysis incorporating student demographic data detected no significant associations with most demographic variables. The only variable that showed a significant association was self-reported confidence level in the students that answered the hemoglobin questions. Qualitative analysis of the written and oral exams revealed student difficulties and misconceptions for both the hemodynamics and hemoglobin topics, with some of these difficulties revealed exclusively through the oral exam. Difficulties included the inability to define basic terms and the inability to appropriately apply mathematical models in explaining physiological concepts. Likert-style surveys revealed a diversity of attitudes among students. Most students felt uncomfortable, unable to organize their thoughts, and rushed on the oral exam. However, other students preferred the oral exam, indicating that the act of speaking helped them better organize and elaborate on their thoughts. This study demonstrates that not all assessment formats are equal and highlights the importance of different formats in accurately assessing a diverse student population. The study also offers data on misconceptions that could lead to the development of a validated concept inventory in anatomy and physiology.