Postscript: You could make it up

Somewhat of a validation for my post on how media reports about '1200 deaths' at Stafford Hospital was pulled out of someone's ass thin air. It's Paul Taylor in the august London Review of Books on 'Rigging the Death Rate'.

Taylor, a reader in health informatics at University College London, in a lengthy piece, pretty much demolishes the basis for those media reports and their source, one Professor Jarman. In fact, in an extremely detailed exposition on measuring stuff in health care, he comes back to the points raised by Steve Walker, the blogger on which my post was based and who has been battling away, somewhat on his lonesome, against the use of Jarman's mishandled (to be generous) data by those who wish the NHS ill.

On the controversial source of info in the NHS which we're talking about, here's something I did not know:

The coding, however, is notoriously prone to error (last year’s HES [Health Survey For England] summary statistics record, for instance, that of the 785,263 in-patient episodes coded under Obstetrics, 16,992 were recorded for male patients) and the system wasn’t designed to measure the quality of care, only the quantity.

The basis of Jarman's business (DFI) is something called the 'Good Hospital Guide', which is entirely based on ...

League tables – applied to surgeons, hospitals or schools – do not give the person reading them any sense of how much variation would be expected for a given sample size and instead impose a rank that the data may not support. There were good reasons for a hospital to regard a poor placing in DFI’s tables as merely an irritating bit of bad publicity.

As Walker and some others have noted, the recording of data isn't the problem, it's how it gets used:

Governments and regulators tend to make drastic interventions with the worst-performing institutions in mind, but the more effective strategy is probably to aim at improving average hospitals, since the ideas developed along the way will work almost anywhere. Lilford and Mohammed [Birmingham University critics of DFI] so criticised the use of the HSMR [Hospital Standardised Mortality Ratio] to measure quality of care. The HSMR shares the weaknesses of the HES data on which it based, but Lilford and Mohammed had a more fundamental criticism: mortality is unlikely to be a good measure of how good care is in a hospital. Typically, 98 per cent of in-patients survive their visit, so nearly all the data about what happens in a given hospital are ignored if we concentrate on mortality rates. Not only that, but of the 2 per cent who don’t survive, only a few will have died an ‘avoidable’ death, and it is only those deaths that can be used to measure the quality of care.

Was Stafford 'broken' because of useless nurses, as you may have heard in the MailExpressTelegraphSun, or ...

The chart above shows the distribution of waiting times in A&E in Stafford Hospital. There is an extraordinary peak at 3 hours 50 minutes: patients were being admitted in a hurry to avoid breaking government targets for a maximum of four-hour waits in A&E. That the system encouraged such sleight of hand is unsurprising – the data from many other hospitals would show a similar peak – but the problem at Stafford was that the care in A&E was poor. It was chronically understaffed, initial assessments were carried out by receptionists with no medical training, and essential equipment – cardiac monitors, for example – were missing or broken. Many patients admitted just before the deadline were sent to units where the care was even worse, some of them to an unstaffed ‘clinical decisions unit’, others to the so-called emergency assessment unit (EAU), a large ward described by the Healthcare Commission as poorly designed, busy, noisy, ‘chaotic’. Nurses in the EAU were inadequately trained and it was common for a patient’s condition to deteriorate unnoticed or for necessary medication not to be provided.

After running through the similarities uncovered in a report on a past case of poor care at another hospital with the Mid Staffs report (staff/patient ratios, inadequacy of systems for reporting incidents, whistleblower victimisation, disinterested senior management in which consultants weren't involved), Taylor turns to the latest idea from the No. 10 brains trust, the ‘friends and family’ test:

Given what we’ve learned about how managers respond to targets, it would be surprising if trusts weren’t already looking for wards or services which, given some extra attention, might yield more than their fair share of grateful respondents. If the test worked, it might help identify hospitals where nursing is poor, but it will be of less help in cases where patients are treated with a combination of kindness and technical incompetence – as at Bristol – or, as with Harold Shipman’s patients, ostensible kindness but actual malice. Just as important, most NHS scandals have affected patients who aren’t likely to voice their complaints, aren’t in a position to make choices and who often don’t have friends or family to speak up for them.

Here's Taylor's concluding point, a sad but true one given the investment much of the media and a certain political party have made in misusing data:

A different approach is used in Australia. Queensland Health was the subject of two public inquiries in 2005 over its employment of a surgeon who became known to journalists as Dr Death. In response, it has adopted an approach which bears some similarities to the work of DFI: data are obtained from hospitals across the state and actual outcomes compared with expected outcomes in order to generate alerts. The focus is on performance over time rather than on a single number and, more important, the data are used differently. A broader range of outcomes is considered, not just mortality, and because it was argued that the threshold should be set very low, many hospitals are flagged so that investigations become a normal part of an institution’s commitment to safety. So long as NHS trusts that have a high HSMR continue to be named in the papers and the statistical concept of ‘avoidable death’ is interpreted as providing grounds for criminal prosecution, the argument for a similar system won’t be made here, let alone won, any time soon.

Walker has just reported on how Mid-Staffs is demonstrably improving, yet is being threatened with closure -- for blatantly political reasons. The proposal is to give £30m to a nearby hospital, and close Stafford. As opposed to:

Save £53m over 5 years in order to keep a hospital that serves around 300,000 people and which has been adjudged by the Care Quality Commission (CQC) as ‘meeting all the essential standards of quality and safety‘.