A comment on "Pandemic Schooling Mode and Student Test Scores: Evidence from US School Districts"

Jack et al. (2023) estimated the impact of district-level schooling modes (in-person, hybrid, or virtual learning) during the 2020-2021 academic year on standardized test pass rates for grades 3-8 across 11 U.S. states. Between 2019 and 2021, average pass rates declined by 12.8 percentage points in mathematics and 6.8 percentage points in English Language Arts (ELA). By leveraging within-state and commuting zone variations, the study found that districts with full in-person learning experienced significantly smaller declines-13.4 percentage points in math and 8.3 percentage points in ELA. Furthermore, the benefits of in-person learning were particularly pronounced in districts with higher proportions of Black students. The study highlights the potential long-term effects of pandemic-related schooling disruptions and highlights the need for targeted policy interventions to address learning losses. In this report, we computationally reproduced all the main results from Jack et al. (2023) using the replication package provided, including code and data. Our replication confirmed that the estimates, directions, and significance levels were identical to those reported in the published study. Additionally, we conducted two robustness tests: (1) comparing balanced versus unbalanced panel datasets and (2) examining the influence of the distribution of the independent variable (e.g., the share of in-person versus online or hybrid learning) on student performance. Our analysis shows some evidence of outliers and state-level heterogeneity in the distribution of in-person learning, which led to some deviations from the original results in the coefficient magnitudes, but the direction and significance remain similar to original results.