John Bald is a former Ofsted inspector and has written two books on the history of writing and spelling. He is Chairman of the Conservative Education Society.
Ofqual’s appearance at the Education Select Committee on Wednesday showed more clearly than anything to date just how far the organisation’s faith in statistical modelling and lack of understanding of education led it into error – and the education system into chaos.
Roger Taylor, its Chairman, started confidently, saying that Ofqual had wanted examinations to continue, but had been overruled by the Secretary of State. A second option had been to delay the examinations, and the third to find “some form of calculated grades.”
Gavin Williamson wrote to Ofqual on March 31 to say that students should receive “calculated results based on their exam centres’ judgements of their ability in the relevant subjects, supplemented by a range of other evidence.”
He went on to say that the approach should be “standardised across centres”, and that steps should be taken to maintain a similar grade profile to previous years. Ofqual then used “statistics and teachers’ rankings” to produce something which, said Taylor, was as fair as it could be.
The first error was to advise that examinations continue. This was impossible because some schools, following trade union advice, stopped direct online teaching as soon as lockdown started, while others – only a handful in the state sector – did not.
Stopping teaching when it would have been perfectly possible to continue it for A level classes would have put the affected pupils at a serious disadvantage. The same issue would have affected delayed examinations.
Ofqual’s statisticians could not have been expected to understand these considerations, but ministers did. Ofqual’s Board, which has highly experienced and expert practitioners, would have been able to explain the position but, according to its official records, did not meet between 26th September 2019 and a late-night session on 15th August, when it put its collective foot down over the botched appeals process. Why not?
What seems to have happened instead is the delegation of the work to a technical group, which did not standardise teachers’ assessments, as instructed, but ignored them completely by applying a statistical model to their rankings. Michelle Meadows, Ofqal’s “Executive director, strategy, risk and research”, justified this by saying that teachers’ grades were not accurate, but that their rankings were.
There is some research evidence to support this view, notably from Daisy Christodoulou, but to ignore teachers’ grades completely was a victory for statistics over reality. Dr Meadows told the committee that 0.2 per cent of grades were “potentially anomalous” and that the statistical model – which I will not flatter with the term “algorithm” – was fair and unbiased.
Furthermore, as teachers were often unsure whether to enter candidates for lower or higher tiers in some subjects, Ofqual had removed any limitation on grades for foundation candidates. That sounds fair – until we see pupils with very limited skills awarded grade 9 on the basis of work they’d never even seen.
Conservative committee members Jonathan Gullis and Christian Wakeford made the case for reality, Gullis pointing to the unfairness of the model to large entries from FE colleges, and Wakeford echoing a pupil’s lament, “I’ve got somebody else’s D”.
The consequences of not applying the model to entries of fewer than five candidates, which favoured private schools and some subjects had clearly blind-sided Ofqual, as did the question why they did not run this year’s results, which they had had since June, through the model to see how far it worked.
Dr Meadows evaded this question, saying they had done all sorts of trials. The point is: why not this one, which would have allowed problems to be identified in advance? It is hard to see how a system that only claimed 60 per cent accuracy could result in only 0.2 per cent of potential anomalies, but Dr Meadows was undaunted. Analysis did not show any bias in the system.
Robert Halfon concluded by asking whether Ofqual was fit for purpose, to which the witnesses, all of them Ofqual officials, predictably replied in the affirmative.
I do not agree with them. Assigning children’s futures to a statistical model, without considering the quality of their work, or even looking at it, is not the action of a reasonable body, acting reasonably, and would have brought a well-deserved hammering on judicial review.
If Ofqual had moderated teachers assessments sensibly, perhaps, as suggested by Bob3142 in response to my previous article, by requiring schools to justify any overall change from past performance, we could have had a fair outcome. As it is, we have had to swallow the grade inflation, and leave schools and universities to sort out the mess. Ofqual should be wound up.