The scandal
In 2011, a state report to the Governor’s office in Georgia claimed that there was “organised and systemic misconduct” in 44 of Atlanta’s 56 schools. 2 years later 35 teachers, head teachers and administrative staff were brought to trial and charged with racketeering and other offences.
The smoking gun
The accusations revolved around instances of “wrong-to-right erasures” in the Criterion-Referenced Competency Tests (CRCT).
The CRCT is an annual test of academic ability taken in elementary and middle schools. It tests student’s ability in Maths, English, Art, Social Studies and Science.
A wrong-to-right erasure is a correction in a multiple choice test from a wrong answer to a right one.
Examiners always see these corrections as students go back over their answers and check them. But not this many. The state prosecutor, Fani Willis, told the jury that a person was more likely to be hit by lightning twice in the same week.
Do you know what the odds of such wrong-to-right erasures are? One in a quadrillion. That’s 15 zeros. A quadrillion. ~ Fani Willis
The investigation
In 2008 the Atlanta Journal-Constitution published a story about rises in test scores in an Atlanta school. The journalist questioned the likelihood that they had really happened. When presented with the allegations Georgia Governor, Sonny Perdue, set up the investigation.
It was a major task. Cheating was wide-spread. By the time the enquiry was complete, 51 investigators had interviewed more than 2,100 people and looked at over 800,000 documents.
They reported a litany of facts and figures:
- 178 teachers were involved in cheating
- 38 of these were head teachers
- 82 teachers confessed to the allegations
- 61% of students who took the test had their answers changed
- The worst school had an 89.5% wrong-to-write erasure rate
Numerous stories backed the statistics.
When the tests were delivered to schools a teacher would steal a single paper. They would take the test themselves and prepare an answer key on a transparency sheet. Once the exams were over, the teachers would use the key to make corrections.
One group explained that they took the papers to the home of a teacher and had a “changing party”. Another teacher admitted that she had changed test results in 2006, 2007, 2008 and 2009. Yet another teacher described how students who fell asleep or refused to complete the exam still somehow met or exceeded expectations.
Why did they do it?
The CRCT test scores were an important part of Georgia’s compliance with the “No Child Left Behind Act”. It was passed by the US administration in 2002. The legislation was trying to improve education levels across the States. Schools that improved their test scores year on year continued to receive funding (a ratchet target). Those that didn’t were subject to a series of remedial actions. These ranged from school improvement plans to staff “replacement” and potential closure.
Three years before the act, Atlanta had appointed a new head of education — the Atlanta Public Schools Superintendent — Beverley L. Hall. Ms Hall was well-respected for her work turning around schools in New York and New Jersey. She promoted a “data driven” system. Her mantra was “no exceptions, no excuses”.
Her approach to appeared to work. To the outside world the Atlanta school system went from strength to strength. In 2009, the American Association of School Administrators named Ms Hall the “National Superintendent of the Year”.
A rotten system
Inside the Atlanta schools system the reality was different.
Ms Hall ran a closed hierarchical system. Most of her employees had to have special permission to enter her central offices. Those that did and bumped into her knew to keep their mouths closed and greet her with just a smile or a nod.
To make sure she met government targets, Ms Hall linked teacher evaluations to test scores. She warned principals that they’d be fired if they didn’t meet their targets. Eventually, ninety per cent of those head teachers were replaced. One principal claimed that “The way principals keep their jobs in Atlanta is they make targets.”
Whistle blowers who reported the cheating to Hall’s aides were ridiculed and punished. Staff at schools that beat the exam targets were given cash bonuses. Even the bus drivers and dinner-ladies. At the same time Hall’s own pay grew to a base salary of nearly $300,000 p.a and a bonus that, over ten years, totalled more that half a million dollars. These bonuses were also largely linked to test score results.
The Governor’s investigators concluded their report by stating that “Hall created an atmosphere that rewarded cheaters, punished whistleblowers and covered up wrongdoing”. Whilst they stopped short of claiming that Ms Hall had directed anybody to cheat, they did claim that Ms Hall “created a culture of fear, intimidation and retaliation” that had allowed “cheating — at all levels — to go unchecked for years.”
The trial
The trial took six months. Ultimately the court convicted 11 teachers of racketeering. 21 others pleaded guilty to receive shorter sentences.
Beverley Hall was not sentenced, she was too ill to face trial and died of cancer before the trial ended.
It would never happen here
It is easy to explain this all away as a single event caused by a bad apple. However, there have been reports of cheating to hit “No child left behind” targets across the US,in cities including: St Louis, Houston, Cincinnati, Baltimore, El Paso and Philadelphia.
In the UK, where targets have also been widely used by government to drive performance there have been accusations of:
- Teachers bribing pupils before school inspections
- Elderly people being sent home from hospital too soon to hit waiting targets
- Police officers giving cautions to children in playground fights to meet “offences brought to justice” targets.
Solving the problem
Following the Atlanta case and criticism of the act, the US government has replaced “No child left behind” with “Every student succeeds”. This still maintains the “hallmark annual standardised testing requirements”.
Only time will tell if this has prevented any further dysfunctional behaviour.
I know what my fiver says, though I suspect I won’t find a bookmaker who is prepared to take it.
When a measure becomes a target, it ceases to be a good measure ~ Goodhart’s law
If you enjoyed this post click here to receive the next
Read another opinion
Image by Kevin Dooley
Annette Franz says
A good reminder that we need to be careful how we incentivize, i.e., what behaviors we’re driving with the metrics and rewards…
Annette :-)
Adrian Swinscoe says
Professor Goodhart’s insight and subsequent ‘law’ was first posited in 1975. I wish that we owed him a much bigger debt of gratitude.
James Lawther says
It is a shame how few people know about it Adrian.
Alex Morrish says
A consultant once told me that targets are the work of the devil. This helps to explain why.
James Lawther says
I hope you paid him a lot of money Alex :)
Ian Mackay says
Incentive schemes are devilishly hard to construct as they do what they say; they incentivise certain actions, the consequences of which may not be what was anticipated nor to the benefit of the originator of the scheme.
Brian Field says
Many pitfall. Make it unfair, too complex or over competitive and you get bad behaviours and dubious results. But we never seem to face up to this.
James Lawther says
It is a bit lazy of us Brian, I agree