Assessing the Causal Impact of Online Instruction due to COVID-19 on Students’ Grades and its aftermath on Grade Prediction Models

The COVID-19 pandemic forced many educational institutions to transition to online learning activities. This significantly impacted various aspects of students’ lives. Many of the studies aimed at assessing the impact of the online instruction on students’ wellbeing and performance have mainly focused on issues such as mental health. However, the impact on student grades—a key measure of student success—has been given little attention. The handful existing studies are either focused on primary schools—where the dynamics are different from higher education—or based on statistical correlations, which are usually not causally rigorous, therefore, prone to biased estimates due to various confounding variables. There are many variables associated with students’ grades, thus, to assess the causal impact of the online instruction on students’ grades, there is a need for a causally-grounded approach that can control for confounding variables. To that end, we use a causal tree to investigate the impact of online instruction on the grades of the general population as well as different demographic subgroups. Our analysis is based on the demographic and engagement data for the 2019 (offline/control) and 2020 (online/treatment) cohorts of 3 mandatory courses in an Australian university. For all 3 courses, our results show that for any given student in the population, the average grade they would have gotten, had they studied offline, reduced by 3.6%, 4.7%, and 14% respectively. Further analyses show that among students with similar level of (low) engagement with the virtual learning environment, the average grade international students would have gotten, had they studied face-to-face, reduced by 19.9%, 36.6%, and 46.9% more than their domestic counterparts despite having similar engagement for the 3 courses respectively. These subgroup disparities have the potential to exacerbate existing inequalities. Given the current concerns about algorithmic bias in learning analytics (LA), we trained grade prediction models with the data and investigated for algorithmic bias. Interestingly, we find that by simply changing citizenship status, a student gets a new predicted grade, entirely different from what was initially predicted given their actual citizenship status. This implies that researchers must be careful when building LA models on COVID-19 era data.