On the Effect of Question Ordering on Performance and Confidence in Computer Science Examinations

Most computer science examinations tend to start with the easiest questions and progress towards the more difficult material. Whether this is because of the highly scaffolded nature of the course, an attempt to 'ease students in', or simply by convention, is unclear. However, there is a great deal of data from the psychology literature to suggest that human perception of the difficulty or discomfort of a task is disproportionately affected by the last part of the task completed. Therefore, is it possible that by structuring our exams in an easy-to-hard fashion, we are causing students to perceive the test as more difficult than it actually is? Could changing the question order allow us to change students' perception of their own achievement? What effect could this have on actual performance? This paper attempts to answer these questions by randomly assigning students to write exams ordered either easy-to-hard (referred as 'Easy-Difficult') or hard-to-easy ('Difficult-Easy), then ask them to predict their marks on per-question basis. We find that the question ordering has a small but not statistically significant effect on the performance, and virtually no effect on predicted marks when treating the entire class as one unstratified sample. However, the effect was significant for certain subgroups created via stratification. In particular, swapping the order of the questions may have hurt the performance of international students, but significantly raised both the performance and confidence of female students.