Just a few hours ago, David Tuller published the latest installment of his excellent Trial by Error blog on the PACE trial in which he includes the briefing notes provided by Prof Michael Sharpe (MS) to the Westminster Hall debate on ME/CFS on 21st June 2018.
I can quite understand Dr Tuller’s reluctance to provide yet another debunk, but I disagree that it is a waste of energy, and that’s why I’m going to attempt to do so here.
First of all, MS refers to the trial as “the so-called PACE trial”. Eh? So-called? That implies that it is colloquially known as the PACE trial, when in fact, that is the name that the investigators gave it. I could now go into the reasons why this name has always been problematic – but I’ll save that for another time (or maybe a footnote later). (Or maybe you can work it out for yourselves!)
What was the PACE trial?
The PACE trial was a large clinical trial including 600 patients that was funded mainly by the Medical Research Council and published its main findings in 2011 in the Lancet.
That, at least, is true (mostly). It was a large clinical trial, and not, as some would have it (including Simon Wessely, who was their key clinical trials advisor), a “randomised controlled trial”, because it was both unblinded and uncontrolled. One of the problems with comparing 4 treatments against one another is the issue of control. And this was particularly problematic, because, as MS is so keen to remind us on Twitter, each treatment had its own “model”, its own explanation for what it was doing and how it was supposed to be helping patients. Ideally, you would want each of those “models” to be controlled for in some way. The PACE trial did not even attempt to do this.
What were the findings?
The findings of the trial were clear. CBT and GET were superior to APT and SMC in improving both fatigue and physical function one year after entering the trial.
That’s self-reported fatigue and physical function, and that is important. And this is the first thing to be included as a “false allegation” made by “activists”:
The trial finding that CBT and GET are superior to APT and standard medical care is false because it used patient rated outcomes.
Many of us who have criticised the trial have simply not made this allegation. We accept that the trial found that CBT and GET were superior to APT and SMC on the patient-rated outcomes, but that’s where there’s a problem – not that they are false, but they are unreliable because they were subjective, and therefore vulnerable to bias.
When you have treatments (CBT and GET) that consistently encourage patients to downplay their physical symptoms, you will produce bias in the results, particularly if the other interventions (APT and SMC) do not do this.
And we have testimony from the trial itself that shows exactly this effect:
“After repeatedly being asked how severe…my symptoms were…I started to feel like I had to put a…positive spin on my…answers. I could not be honest about just how bad it was, as that would…tell the doctors I wasn’t trying and I wasn’t being positive enough.”
Were the differences meaningful?
The next point that is not quite correct is that “the differences were clinically meaningful.” They weren’t. They might have been statistically significant, but they were not clinically significant. This is demonstrated by the lack of any difference in the more objective outcomes used in the trial.
Were the outcomes changed?
The next allegation is:
The investigators changed the trial outcomes to make CBT and GET look better.
MS claims that they used the “originally registered primary outcomes to report the trial findings.” This is only true in as much as the outcome measures themselves were not changed. However, the thresholds for those outcomes very definitely were changed.
MS concedes that the “way the outcomes were used in the analysis was changed from the initial protocol following statistical advice.” However, this advice was clearly flawed or misunderstood, since it now meant that the thresholds were lowered to a point that rendered the results clinically meaningless.
We are also mystified as to how this was signed off by the Trial Steering Committee, since we can find no evidence in the meeting minutes (obtained by a FOIA request) that this happened.
What about harm?
The study looked very carefully for harms and found no excess of harms with the rehabilitative-type treatments (CBT and GET).
Trying to assess harms when the therapist is telling patients to actively disregard harms is always going to be problematic.
What about recovery?
Allegation:
The proportion of patients regarded as recovered was inflated and no better with CBT and GET than with the other treatments.
Actually, we don’t know how many patients “recovered”, because we don’t really know what “recovery” was. Latest info from the authors is that recovery was simply “recovery from the latest episode of the illness”, but even this is poorly defined and doesn’t accord with what was reported in their recovery paper.
The measures for “recovery” used in their recovery paper were even lower than the criteria they used for “improvement” in the original protocol. Hmmmm.
Wilshire et al.’s reanalysis of the data according to the original protocol found no statistically significant difference between the groups on this criteria.
What about outside influences?
Allegation:
The trial was fraudulent or in some way influenced by the DWP or the insurance industry.
It didn’t need to be. The aims of the trial and the investigators already chimed with the aims of the DWP and the insurance industry. There were no “conflicts” between the “interests”! They all seemed to be singing from the same hymn sheet.
Was data hidden?
Allegation:
The PACE authors refused to share or hid the trial data.
They have refused to share trial data. A partial release of data allowed for Wilshire’s reanalysis, but this would not have happened without a Freedom of Information request, and the investigators are still refusing to release the whole dataset for a more complete reanalysis. MS claims that Wilshire’s reanalysis is flawed. However, it only uses the pre-specified criteria from the PACE investigators original protocol. (I’m pretty sure that White et al. published their own reanalysis on this basis on the PACE trial website at one point that agreed with this.)
As far as we can tell, the investigators are only willing to share data with their friends, and not with their critics. The Cochrane group mentioned includes members of the PACE trial team, and is headed by those who are sympathetic to their aims and methods.
They still refuse to release data related to the more objective measures used in the trial, including employment data or follow-up data. Follow-up studies were planned at 5 and 10 years, but have yet to be completed. This may be because the long-term follow-up at 2.5 years showed no differences between the groups.
Flawed and discredited?
Allegation:
The PACE trial is universally regarded as flawed and consequently discredited.
MS Response 1:
Very lengthy critiques have been produced alleging myriad flaws in the trial. Some of these include quotes from scientists, none of whom are experts in clinical trials.
You don’t have to be an expert in clinical trials to see how flawed the PACE trial is. I only have an MSc in Medical Statistics, and even I can see how flawed it is.
MS Response 2:
It is important to note that experts in clinical trials, including the many peer reviewers of the paper when it was published in The Lancet, have not found the trial to be flawed. Indeed, it is regarded as a high quality trial.
How many peer reviewers? How many of them at least have an MSc in Statistics? I very much doubt that they were experts in clinical trials, as you say, or they would have spotted the errors. Maybe if Richard Horton could tell us who they were, we could better assess that. I suspect that the trial was reviewed by authors of similar trials that have found similar findings because they made similar mistakes. The key arbiter of science may well be to replicate findings, but it is certainly not to replicate mistakes.
Other points
The finding that CBT and GET are helpful has led some to believe that the study proves an unwanted proposition, that “ME is a psychological condition”. Not only does a treatment study not do that, but the paper also explicitly says that this is not the case.
But it is the case that the investigators treat the condition as “psychosomatic”. This is the source of much of the stigma surrounding the condition.
Another reason is that patient experience of CBT and GET outside the trial is reported as often being unhelpful or even harmful. This is an interesting and worrying observation probably explained by the treatment being given to people dissimilar to those in the trial or being given incorrectly.
Quite a few of such reports have come from patients who were treated at the same centres that were part of the trial.
“[F]ound no excess of harms” is hardly the same as “found no harms”. What is the definition of “excess”? How much harm did a trial participant have to endure before it crossed the threshold of “excess harm”?
How can it be that PACE “found no harms”, however it is defined, while numerous patient surveys are unequivocal: GET causes harm. Sharpe claims to be concerned about possible post-trial harm caused by his treatments, yet he refuses to withdraw those treatments.
There have been many pharmaceutical and surgical treatments that were found to be harmful post-approval and subsequently withdrawn. Why aren’t Sharpe’s treatments subjected to the same official scrutiny as any other medical treatment?
The reason for hiding the trial data is obvious to me. I predict a proper investigation of the trial’s raw data, including the participant interviews, will reveal just how much harm the participants actually suffered. That is why they have spent 250,000 pounds to hide the data – that is small potatoes compared to their liability if the truth about harm to PACE participants ever sees the light of day.
Thanks are due to Lucibee.
In discussing the many fatal flaws in the method of this pseudoresearch, we can all too easily help Sharpe et al by overlooking, even temporarily, that the population “studied” was of people who may or may not have had ME. Recruits were required only to experience persistent fatigue, so it is anybody’s guess what illnesses they actually had. I think that this needs to be mentioned every time PACE is criticised.
One point I think worth adding is that QMUL argued at the tier 1 tribunal that the cochrane group wasn’t independent they seem to want to have it both ways, which ever is most convenient to them.