You are hereBlogs / dr-no's blog / Nature Cooks the Books

Nature Cooks the Books


Posted on 03 February 2012

cheshire_cat_2.jpgEven at the best of times, epidemiology can seem as dry as old biscuits, and when it starts counting stiffs – as it so often does – it can smell not just dry and old, but musty too. But it is an important ology, and when done well, which is surprisingly difficult, it can tell us useful things.

This morning on Radio 4’s Today programme, we had Professor Bruce Keogh, a Department of Health chief pongo, putting the icing on an epidemiological cake he had himself helped bake. Having started soundly enough, he suddenly veered off-piste, gathered speed, and delivered a shopping list of deliverables that bore no relation to the study he and Wingnut had only a moment ago been discussing. “For me,” Prof Cough said, “the big prize …is about our NHS catching up with other service industries and offering a routine six or seven day week…where people can have routine operations over the weekend…at their convenience…access to expert advice…that’ll put the compassion back into the NHS”. Compassion? It sounded more like commerce to Dr No. It was, of course, a government inspired vision for the 24/7 hospital, where every day is Wednesday, every patient a consumer, and all the clocks strike thirteen.

The study, available here, appears sound in its data collection and analysis. Unlike recent comedy research by Dr Foster, which relied dodgy DIY trust data to show that a lower hospital consultant presence at weekends was linked to higher patient death rates, Prof Cough’s study relies on national data (albeit the hospital admissions data is self-reported by trusts), and the analysis was, so far as it was able to go, commendable. The researchers found that, for every 100 deaths to patients admitted on a weekday (they chose Wednesday), there would be 116 deaths to patients admitted on a weekend day. Put bluntly: you are more likely to come out of hospital feet first if you go in at the weekend. So far, so good – or bad, if you happen to be admitted over a weekend.

But – and is a big but – this nicely baked epidemiological cake is only good so far as it goes. One can’t go around layering it with speculative icing, and expect the icing to look good. And yet that is just what Prof Cough did: he leapt from an established association, via a speculative (he himself described the situation as “complex”) causation (not enough senior doctors around at weekends), to an imaginary solution: the 24/7 hospital.

There are at least two reasons why the cake cannot take the icing put upon it. The first is that none of the data collected measured actual clinical activity - number of consultants and other grade staff present, ward round frequency and diagnostic and operative activity, for example - so the study is incapable of showing an association between lack of clinical activity and subsequent mortality, even if one existed. But even if that data had been collected, and used in the analysis, there is another potential flaw, the bane of the epidemiologist’s life: bias. Unlike random error, which can be largely dealt with by increasing sample size, bias causes a regular, systematic error. It does so because something is wrong with the conduct and/or analysis of the study.

Much of the art and science of epidemiology is directed towards spotting and countering bias. Unspotted, un-countered bias remains an ever present danger; the black ice on which even the soundest epidemiologist can slip, and leave the road of truth and unwittingly end up in a duffer’s ditch of despair.

The varieties of bias are as they say legion, but it seems to Dr No that – just for starters – there is a common form of bias, selection bias, that could very easily apply to this study. Selection bias occurs when something causes - selects - patients with differing and, crucially, relevant characteristics to be unequally distributed between differing groups in a study. More patients with more severe disease, for example, end up in group A, and it is this, rather than differences in treatment between group A and group B, that explains why there are more deaths in group A.

Now, if we think about our own behaviour when we are ill, we can see all too easily how just such a selection might happen in the present study. Many people, especially the elderly, who are likely to figure large in this study, are reluctant to call the doctor out of hours, all the more so at the weekend; those that can will hold out until Monday morning, as any GP knows only too well. Those with more severe illness, however, will not stand the wait, and so will call the doctor straight away and so be admitted to hospital there and then. And so it is that more patients with more severe illness may be selectively admitted at weekends; and this in turn may account for more deaths in those admitted at weekends.

None of the data collected in the study measures of the severity of actual illness giving rise to admission, and so we cannot allow for it in our analysis. We do not know whether Sunday’s chest pain turned out to be indigestion, or Wednesday’s pain the start of a massive heart attack, or, of course, the other way round. If, as seems very possible, Sunday’s pain is indeed more likely to be cardiac, then it is equally possible that greater severity of illness, rather than differences in staffing, at weekends could accounts for the higher mortality following weekend admission.

Not for the first time, and certainly not for the last time, we may find that nature, through bias, has cooked our books.

17 comments:

What makes the study of particular interest is its propaganda potential in justifying Lansley's market driven turd burger.

I can remember having a conversation with a friend in a radiology (after 24hr imaging had been mooted for routine investigations).
Unsurprisingly the prospect of appointments at 3 o'clock in the morning was not thought to be terribly popular with the vast majority of patients?

As to the difference in mortality rates - well, there always has been a sense that out of hours services are less robust than during office hours, Monday-Friday - "the sickest being treated by the thickest", etc.

The National Confidential Enquiry into Patient Outcome & Death (NCEPOD) reports are always instructive
http://www.behfuture.nhs.uk/archive/docs/appendix_17.pdf

http://www.ncepod.org.uk/2009report2/Downloads/DAH_report.pdf

... she said, without even looking round
"I'll fetch the executioner myself,"

Only, that was today, and it was Andrew's head ... but then DC said NO! I think it's because when they asked Lansley to name his last wish, he said " I want my HSCB passed!" Ah, well ...

What will happen now, I Wonder ... ;-)

http://www.telegraph.co.uk/health/healthnews/9066483/Downing-Street-does...

(S)AM - I think we are at a tipping point. Lansley has never looked more fragile, his supporters more divided.

They 'need' that bill Dr No for without it ideologies would collapse too - domino effect

Plus, there are many god bits in that new shaped and much more improved that original bill too. For example; empowering all docs not just GPs, reducing red tape so eventually, more money in the hands of 'docs' not bureaucrats for patient care, properly scrutinising and holding GPs to account [KPMG], and lots more opportunity to innovate for willing people with ideas like you - breaking the monopoly on the way too ... besides, the new shape bill is more flexible to allow more adjustments if needed as they arise too, not bad!

Meant to say 'good bits' of course :-D

"For example; empowering all docs not just GPs, reducing red tape so eventually, more money in the hands of 'docs' - this is the primary myth that the whole croc of shit is based on.

Over to Ben Goldacre
"In case u don't understand NHS bill: GPs know they're being set up to fail by being given commissioning powers - those are specialist skills.
After GPs fail, private commissioning expertise will be needed: large private corps, which will come to operate like health insurers.
These large bodies, like public/private insurance co's, will be able to pick & choose patients. Note no geographical responsibility in bill.
Small differences will emerge in what services they offer. Top up plans will become available. And that, kids, will be that.
It is so very obvious that GPs are being set up to fail at the specialist task of health service planning that it's clearly not an accident.

Those last 5 tweets are what will obviously, predictably, happen to the NHS after this bill. If you missed them, they were a bit important".

http://bengoldacre.posterous.com/what-will-happen-with-the-nhs-bill-in-5...

""In case u don't understand NHS bill: GPs know they're being set up to fail by being given commissioning powers - those are specialist skills."

Only the GPs will not be left, or allowed, to 'fail'. Because this causes headache, like 'what to do now?', and no government would want that, specially with such new initiative as it needs to be seen to work or government would suffer! That's why the likes of KPMG are being sought, to ensure just that! KPMG makes fool proof 'tailored' programmes, ones that fit like a glove, then it monitors using the built in audit capabilities and provides specialist advice and expertise whenever needed - this is precisely to avoid that 'failure', while at the same time holding those using that programme to account too, to the dot. Could this be the reason why GPs are now opposing the bill?

"Only the GPs will not be left, or allowed, to 'fail'. Because this causes headache, like 'what to do now?', and no government would want that" - oh, it won't be spun as failure - the propaganda line will be something along the lines of GPs being freed up for clinical work while the poor old corporations are left to fret over pesky commissioning stuff.

Support or opposition to the bill (amongst NHS clinical staff) is broadly divided into x2 camps, those who prefer to maintain the NHS's founding principles (the overwhelming majority, by the way) and those who think some form of privatisation is overdue.

My guess is most support for privatisation (amongst doctors) will arises amongst specialties where doctors already have an established list of private punters - things like production line surgical procedures might be especially attractive to money orientated clinicians.

Anonymouse and A&E CN - as it happens, by pure coincidence, Dr No was putting together a post on Snatcher Commissioning as you were kindly commenting here. Note that, although Ben Goldacre says it much more succinctly, we are pretty much saying the same thing. But then, fools seldom differ...so we must be wrong!

Of course, 5 tweets are will be, predictably, happen to the NHS after this bill. If you missed them, they were a bit important.Great.taxi zürich

This is what I have been searching in many websites and I finally found it here. Amazing article. I am so impressed. Could never think of such a thing is possible with it...I think you have a great knowledge especially while dealings with such subjects.

I would like to thank you for the efforts you have made while writing this post. I am hoping for the best work of the same from you in future. click me

How could I have missed this blog! Its incredible. Your design is flawless, like you know exactly what to do to do make people flock to your page! I also like the perspective you brought to this subject. Its like you have an insight that most people havent seen before. So great to read a blog like this.
london tantra

I have been meaning to write something like this on my website and you have given me an idea. Cheers. Lipozene

Well somehow I got to read lots of articles on your blog. It’s amazing how interesting it is for me to visit you very often.
Slim Quick Reviews

Outstanding blog post, I have marked your site so ideally I’ll see much more on this subject in the foreseeable future.
Slimquick

It is really a beautiful and creative blog having some vital information over the subject. Thank you for share.


Add a comment...

Will show as anonymous if no name added

If added, your name will be a link to the address you enter

If left blank, first few words of comment will be used

• Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li>
• Web page addresses and e-mail addresses turn into links automatically
• Lines and paragraphs break automatically
 

NOTE: Dr No's spam filter can be somewhat overzealous. If your comment has been wrongly rejected, Dr No apologises, and asks that you let him know (via Contact Form in side-bar). Many thanks.