You could make it up

In today's collective strop by the newspapers about regulation there's one theme, a supposed threat to investigative reporting. One story being pointed at as potentially being threatened by 'government control of the press' is the 'scandal' of hundreds of 'avoidable deaths' at Stafford hospital.

What's funny about the press using this example to make their argument is that their Stafford reporting has, universally, from the BBC all the way to the Daily Star, been terrible, shockingly terrible.

The number of 'avoidable deaths' - the figure commonly reported is 1200 - is completely made-up. Any reporter doing their, you know, job could figure it out.

Blogger Steve Walker explains at length how '1200 deaths' was pulled out of someone's ass thin air and how its basis is undermined by just a cursory understanding of statistics. Here's a key passage:

As I explained in my post on the real story of Mid Staffs, Prof Jarman’s HSMR (hospital standardised mortality ratio) system rates English hospitals according to where they sit according to the average death rate for the nation. A hospital hitting exactly the average rate would receive a ‘score’ of 100. A hospital doing better than average would have an HSMR below 100, and one doing worse would be over 100.

So far, so clear – I hope. But here’s the key fact: every year, the system is ‘rebased‘ – the averages are re-measured and ’100′ is re-calibrated to the new average.

This leads to 3 key problems:

1. Because of how averages work, unless by some miracle every single hospital in the country got exactly the same score, you will always have some hospitals above 100 and some below. This does not mean the ‘extra’ deaths in those hospitals were avoidable – it just means that somebody has to be above the line because it’s an average.
2. Because the ’100-line’ moves every year, a hospital can maintain exactly the same standard in one year as it achieved in the previous year – and yet can score below 100 one year and above 100 the next. The performance of the hospital did not get worse. The line just moved. It’s not only wrong but ridiculous to extrapolate ‘extra’ or ‘avoidable’ deaths from a position above or below a line that moves every year.
3. Leading on from number 2 – and it’s impossible to overstate this – there is no ‘standard’ rate of deaths from a particular illness. No expert clinicians are sitting down together and saying ‘Yes, we agree that out of every 100 patients with an intracranial bleed, this many are going to die’. No. All that happens is that the average for the previous year becomes the re-calibrated ‘expected death’ figure for the following year. This means that HSMRs are measuring the success/failure in achieving/beating/failing to bear a target that moves every year – and has no basis in clinical expertise. It’s just a number.

In his testimony to the Francis inquiry, Prof Jarman claimed that he had to present the figures this way because the English are ‘simple-minded‘ – but in doing so he has committed a fundamental error of logic worthy of a simpleton.

These fundamental logical errors mean that even if everything else was perfect, HSMR scores over 100 cannot be used to calculate avoidable deaths.

Walker explains how behind the media's scary headlines lies 'rubbish in, rubbish out'. The data wasn't being properly recorded at the hospital and the woman brought in to fix that, who he has met, wasn't allowed to give evidence or speak to the media.

That women's work on correcting the errors produced the exact opposite outcome to the screaming headlines:

When Ms Kirkbright arrived at Mid Staffs, she carried out a re-coding exercise on past deaths. This re-coding corrected the absent Z51.5 code and used the case notes to add in the co-morbidities (what is known in the jargon as ‘depth of coding’) that was missing. This brought down Mid Staffs’ HSMR to 88 – well below the national average death rate.

Further:

HSMRs are a statistical device. If you want to be sure whether deaths were avoidable or not, you need to look in detail at the case notes for each patient. The doctor in charge of the Independent Case Note Review (INCR) was asked by the inquiry how many ‘excess’ deaths he had discovered among all of the cases for which families asked for a review. His answer was telling – but has been almost completely ignored:

    Perhaps one such death.

And further, Walker reports evidence that hospitals are gaming this bonkers system because of funding pressures - to increase their incomes.

Plus Walker gives examples of how reporting on supposed bad treatment of patients at the hospital could simply not be true.

The man behind the headlines, Professor Sir Brian Jarman, has a clear conflict of interest. He is financed by a company which sells services that allow NHS Trusts to use and monitor HSMRs. That has not been reported.

What is also not being reported is that the entire system is based on the most ridiculous way of recording and therefore supposedly monitoring morbidity rates.

HSMRs are based on ‘first diagnosis’.

If a patient arrives unconscious and the ambulance driver reports ‘He fainted’, fainting would be the first diagnosis. If that patient is then discovered to have had a serious stroke and dies, that death will be recorded against ‘fainting’ – a condition with a very low death-rate – resulting in an inflated HSMR.

There is no recording of post mortum results. The system is utterly broken. You couldn't make it up!

It's plain once you read Walker, or the handful of other bloggers, or this lonesome letter to The Guardian, or Computer Weekly, all similarly demolishing the media's reporting, that the headlines about '1200 avoidable deaths' aren't wrong because the media are all idiots, they are the result of an agenda. A stealth-privatisation, anti-National Health Service agenda.

Says Walker:

The threat is serious, because a lot of people will believe what’s being said and written just because it’s in the media and has apparent statistical/scientific support. Very few will look beyond the soundbites to see whether they stand up to scrutiny.

So please, if you’ve read this article and agree with its premise, spread the word. We need good information out there to counteract the absolute tripe that’s being force-fed to the British public.

Happy to oblige

• Here's a postscript