The Algorithm and the Randomized Trial

The Global Signal

England's exam regulator, Ofqual, used a statistical model in 2020 to standardize grades after in-person exams were cancelled, weighting a school's historical performance heavily against individual teacher predictions. When results were released, nearly 36 percent of A-level grades came in one grade lower than teachers had predicted and 3 percent came in two grades lower, a pattern that disproportionately affected strong students at schools with weaker historical results. Public backlash forced a government reversal within days of the August 13, 2020 A-level results, and Ofqual's chief regulator, Sally Collier, resigned on August 25, 2020.

Georgia State University took a different path with a narrower tool. Its Pounce chatbot, piloted from 2016 to reduce "summer melt," the phenomenon of admitted students failing to enroll, was tested over multiple years before being trusted at scale. Reported outcomes include summer melt falling from roughly 19 percent to roughly 9 percent in the pilot population, with first-generation students reported to benefit disproportionately, plausibly because they are more reluctant to ask questions in person.

The Hidden Signal

Ofqual's model was applied to every affected student in the country simultaneously, with no smaller-scale test run to reveal how it behaved on real students before the entire system depended on it, and with no mechanism for an individual teacher's judgment to override a specific grade. Georgia State's chatbot was piloted for years on a single institution's incoming class before broader claims were made about it, and it was built specifically to route a student toward a human advisor when a question exceeded what the tool could answer, not to make a final decision about that student on its own. Consider what the alternative looks like at Georgia State's scale: a university deploys a similar tool nationally in its first year, skips the multi-year pilot, and discovers only after the fact whether the tool actually helps the students it was meant to help.

Why the Visible Metric Misleads

A model's statistical soundness in aggregate, correctly predicting the national grade distribution, correctly reducing the average student's likelihood of a specific bad outcome, says nothing about whether any single decision made about a specific individual was fair or correct. Ofqual's approach was defensible at the level of overall distribution and indefensible at the level of any one student whose own performance did not match their school's history. The more revealing test for any algorithm applied to individual people is not whether the aggregate looks right; it is whether the system was piloted at a smaller scale first, and whether an individual's own circumstance can still override the model's output.

Implication for Operators

Any organization applying a statistical model to decisions about individual people should assume that the model's soundness in aggregate says nothing about its fairness in any one case, and that only a genuine pilot at smaller scale, plus an explicit override path to a person, closes that gap. Ofqual's national, unpiloted rollout and Georgia State's multi-year, human-routed pilot are the same category of tool built on opposite assumptions about what individual fairness requires.