Bizarro World at the World Congress of Physical Therapy

I have to admit, I’m a massive Seinfeld fan. So my apologies for this blog post title as it reflects my constant need to infuse daily life with Seinfeld references. For those of you not obsessed with Seinfeld, Bizarro World is an alternate reality discussed in the Seinfeld episode, Bizarro Jerry, when Elaine (one of the main characters) meets a new group of friends that are exactly like her normal group of friends (Jerry, George, and Kramer), except they are exactly opposite. And at the end of the episode, both groups of friends have a random chance encounter and we experience the extent of the Bizarro-ness.

Well, the World Congress of Physical Therapy (WCPT) was a bit like Bizarro world for me. A colleague of mine, Steven Kamper, has already provided a great post summarizing the conference, but I wanted to build on it a little. So Bizarro world… Usually at conferences, many of the biggest trials (like randomised controlled trials [RCTs]) are presented as platform presentations, thus allowing the greatest opportunity for dissemination. However, at WCPT, I found this to be opposite – some of the biggest trials were presented as posters. Now before I discuss this further, I’m not meaning to insult anyone who had a platform presentation – a lot of the presentations were really good and very interesting. Also, I’m not meaning to imply that all the posters were completely amazing (I had a poster[1], so that’s just poor form!)…I’m only speaking in general terms.

But I found this oppositeness fascinating. On the good side, it allowed platform presentations of studies with interesting findings (but that weren’t large RCTs) to be discussed in front of a larger audience. It also allowed numerous one-on-one discussions in the poster sessions with authors of some of the larger studies. Both of these things have merit. The only thing I found a bit tough to come to terms with was that these large RCTs, that are the highest quality of evidence we have for physiotherapy interventions and for guiding treatment, weren’t being highlighted to the physiotherapists attending the conference. The poster sessions were great, but there were a lot of posters and a lot of poster sessions, resulting in a decent chance that many practicing physios missed the results of these important studies.

At the end of the day, I felt a bit like I was experiencing Bizarro World and wasn’t exactly sure how good I felt about its Bizarro-ness. Has anyone else experienced conferences like this and what did you think?

About Tasha

Tasha Stanton Researcher

Tasha Stanton is a postdoctoral research fellow working with the Body in Mind Research Group both in Adelaide (at University of South Australia) and in Sydney (at Neuroscience Research Australia). Tash has done a bit of hopping around in her career, from studying physio in her undergrad, to spinal biomechanics in her Master’s, to clinical epidemiology in her PhD, and now to clinical neuroscience in her postdoc. Amazingly, there has been a common thread through all this hopping and that common thread is pain. What is pain? Why do we have it? And why doesn’t it go away?

Tash’s research interests lie in understanding the neuroscience behind pain and its clinical implications. She also really likes nifty experiments that may have no clinical value yet, but whose coolness factor tops the charts. Last, Tash is a bit mad about running, enjoying a good red with friends and organizing theme parties.

References

ResearchBlogging.org

[1] Stanton TR, Fritz JM, Hancock MJ, Latimer J, Maher CG, Wand BM, & Parent EC (2011). Evaluation of a treatment-based classification algorithm for low back pain: a cross-sectional study. Physical therapy, 91 (4), 496-509 PMID: 21330450

Comments

  1. Neil Tuttle says:

    Thanks Tasha
    I’ll comment on some aspects individually, then expand more globally. The fact that therapists often individualise treatments is only one aspect of the lack of reproducibility. Suzanne Snodgrass and others have clearly demonstrated the wide variation in forces used when therapists are intending to do the same grade of the same technique on the same individual. We also took some ultrasound images of the movement of the vertebra underlying the application of a treatment technique and found very large differences in the movement produced with quite small alterations in technique. I do not have any difficulty with RCTs, in fact on contrary our studies have been RCTs, the difficulty is using results from parallel designs – which can by their nature only compare group effects – to be the determinant of selection of treatment for an individual. The Kent, et al article only considered group effects when either a preselected or other treatment was selected. Similarly, although patients improved with both treatments, our recent study did not find a difference in the group effects when two treatments were applied (one thought to be relevant and the other not). We did find that individuals responded consistently and differently to the two treatments. Regarding crossover trials being of most value when the condition is static. Although not necessarily true as time, order and carryover effects can be controlled for, our recent studies looked at multiple repetitions of 2 minutes of treatment in a single one hour treatment session – a duration wherein the symptoms would not be expected to fluctuate too much. Also note repeated crossover trials are capable of providing very different information than single crossover trials. We did fairly straightforward full factorial ANOVAs with Pain or Range of movement being the dependent variable and Treatment, Axis of movement(rotation or LF), direction (towards or away from the treated side), and subject as fixed factors. I would suggest that it is the several patient by treatment interactions that are of particular interest. The preliminary results were presented at WCPT and abstract is available online. The full results which were very similar are being presented at the upcoming Australian Physio Conference with publication hopefully in the nearish future.
    As regards interpretation of someone responding, or appearing to respond, to a given treatment once, I agree that this is pretty much meaningless (unless without the intervention absolutely no one would improve). That is the reason why repeated crossovers are necessary to determine if there are consistent individual responses. Also vital to remember that experimental evidence is only 1 of three equally important ‘pillars’ that make up evidence informed practice with Clinical experience /judgement and Patient preferences and values. What I am advocating is far from open slather, but that there is evidence for using the patient’s response to guide clinical decision making. Furthermore if in clinical practice one intersperses a couple of alternate repetitions of two treatments we can create individual N or 1 trials which provide a much stronger support (or not) for a given treatment selection than just continuing with what works on the first attempt.
    Cheers
    Neil

  2. Tasha Stanton says:

    Hi Neil,

    Thanks for your comments. I take your point about physiotherapy treatments being hard to test using a rigorous RCT design as it is often very difficult to blind patients and therapists to the treatment at hand. In addition you brought up the problems of heterogeneous patient populations and treatments that aren’t reproducible (I’m assuming you are meaning that often treatments are not cookie cutter treatments, but individualised to the patient). However, I do not think that the presence of these attributes suggests that RCTs are inappropriate study designs to evaluate the effectiveness of physiotherapy treatment. Speaking more about individualised treatment, there is some evidence to suggest that the outcomes may not go the way we expect. In a large meta-analysis by Kent and colleagues,1 it was found that when therapists had their choice of treatment technique for manual therapy, outcomes were no better (in fact, they were actually a bit worse) than when therapists provided standardised manual treatment. I also agree that cross-over studies do have value in that patients can act as their own controls. However, these cross-over studies are of most value in patients in which the condition is static (eg, will not just get better on its own) and thus are not the best study designs for things like acute pain. Problematically, even chronic low back pain is known for its fluctuating nature.

    I didn’t get to see your presentation unfortunately, so I’m curious as to how you interpreted the individual responses. Were the individual responses just analysed within one arm of the trial? (For example, with treatment A, 14 patients had a very large improvement in pain, with no reference to what outcomes any patients receiving treatment B had). If this is the case, there is not an adequate control for the fact that this person receiving treatment may have just gotten better due to the natural history of the condition. So we can pretty much ‘prove’ every treatment works because someone out there ‘responds’ to it. However, this may not have been how you analysed your results, so this point may be moot. I do agree that when you apply treatment A to a group of people you get a variety of responses. Some people get heaps better, some not so much, and some might actually get worse. However, if we are comparing this treatment to something that has no active component whatsoever (eg, inert ultrasound) or a no-treatment control, we would expect that if this treatment had some effect there should be a response at the group level.

    This is an interesting discussion indeed and thanks for joining Neil.

    1. Kent PM, Marks D, Pearson W, Keating J. (2005) Does clinician treatment choice improve the outcomes of manual therapy for non-specific low back pain? A meta-analysis. Journal of Manipulative and Physiological Therapeutics. 28:312-322

  3. Neil Tuttle says:

    Hi Tasha et al
    Just came across the discussion and was pleased to see the amount of interest in the validity or otherwise of RCTs versus other types of evidence. I think RCTs work best for the situations they were designed for – homogeneous conditions and reproducible and blindable treatments. I would suggest that most of what we do doesn’t fulfill either of these conditions. Clinical prediction rules were an attempt to refine subgroups, but haven’t proven terribly successful in making the groups more relevantly homogeneous. RCTs can also be performed using, for example multiple repeated crossover designs that evaluate individual treatment responses (several articles by Senn provide an interesting discussion). Also it is not necessarily just a question of whether one type of treatment is better than another for a group, but questions of individual responses. We also presented (a platform so it may not have been much chop) preliminary results which demonstrated that consistent individual responses to treatment occur in the absence of group responses. In other words it may not make any difference to the outcome if everyone in a group receives the same treatment whether they receive A or B, but it does make a difference to an individual – which, I think after all is what we, and most of our patients, are interested in.
    On another note many studies that compare two treatments separately and in combination frequently suggest that multimodal treatments (e.g. exercise and manual therapy) produce better results than each alone are interpreted as each patient should receive multiple treatments. Since everyone in the group doesn’t respond to either treatment, I think it is important to recognize that these results are just as easily explained by some individuals responding to one treatment and other individuals responding to the other. Again , the problem is not with the use of science, but with knowing what can be determined from what type of study design and simplistic interpretation of results.
    Cheers
    Neil

  4. Interesting and disturbing observations. How many of us are really able to analyse either why or what we do? If the “research” was done by a completely disinterested party where would we be?

    ANdy

  5. Tasha Stanton says:

    Steve – spot on comment. Following your line of thought: I think one of the most important things about RCTs is that they control for the unknown variables that affect treatment outcome. I think this is particularly relevant for health conditions where we do not know what is causing the pain (eg, such as back pain). It follows then, that if we don’t know what’s causing the pain, the mechanism of action of our treatments is an educated guess. Now if we don’t know what causes the pain or how our treatments really work, how can we possibly know what variables (patient variables, environment variables, etc…) we need to be controlling for (eg, because they could plausibly affect treatment outcome). To me this is the real strength of the RCT study design, because through proper randomisation (and a sufficient sample size), it makes us more confident that these unknown variables will be approximately equally distributed between groups. This to me is the real danger of only taking into account single arm study designs or case studies, because it does not allow us to control for the unknowns. (Gosh I think I’m up to putting in $0.08 now…is my money still good here?)

  6. David Nolan says:

    I agree Paul,

    One of the things physiotherapy should be proud off is the critical appraisal of what we do. We are in danger of sounding like Homeopaths and Chiropractor.

  7. Griping about RCT limitations “followed by loud cheering and applause”? That’s a striking observation! What’s next? Tear down a statue of a scientist? Burn a pile of research methodology texts?

    It’s not really a surprise, sadly: on many occasions I have witnessed groups of professionals circle their wagons to defend against the implications of the evidence. Their livelihoods are at stake, after all. More and more, their fear and anger is directed at the process and principles of science itself — “if I don’t like what it has to say, there must be something wrong with this science stuff.”

    All this defensiveness is par for the course in alternative medicine. It’s really sad to hear about it so clearly on display at WCPT.

  8. Steve Kamper says:

    I just don’t know if I buy the line that (insert any non-pill therapy including physiotherapy) is not amenable to testing in the context of RCTs because RCTs control for non-specific effects, and these effects are an important component of the intervention. If both treatment arms include the non-specific effects (interaction, context, meaning, relationship, placebo etc) and the between group difference is small/non-existence doesn’t that show that the index intervention is ineffective? In the best case, any improvements seen after its implementation are due to the non-specific stuff. Does adopting this line mean we accept that the effective mechanism of action is due to non-specific effects, and the ‘treatment’ just serves as a vehicle?

    For the record, I agree that RCTs don’t answer every question relevant to a clinician with a patient standing in front of them. I do however share Paul and Tasha’s view that we shouldn’t ignore a body of knowledge that shows a coherent pattern.

    Geoff Bostick Reply:

    Your comments are bang-on. Of course the RCT is the gold-standard in testing interventions and randomization is likely to control for context in both arms. However, in cases when process is not well understood perhaps more background work is needed before jumping into the RCT to avoid the common findings of ‘inclonclusive’ or ‘modest effects’. Put another way, are the right questions being asked?
    There was a really nice commentary in IASP’s clinical updates highighting the challenges of non-pharmacological trials in pain by Michael Bennett (Vol. XVIII, Issue 2 May 2010. He gave the analogy of nonpharmacolgical interventions using an imprecise shotgun pellet that hits the target but is all over the place. Contrasted to pharmacolgical interventions where the process is understood resulting in repeated and precise marks on the ‘bullseye’. The suggestion is that the process of nonpharmacological treatments of pain are not well understood making it likely to be imprecise. Thus, perhaps more feasibility studies need to be performed to understand the context and complexity of a treatment before undertaking an RCT. Qualitative research and case studies could certainly fit here. In addition, adjunctive data collection within RCTs could be helpful to better understand process, explaining results and to assist in follow-up refining future studies.
    Like most things, ‘truth’ seems to be somewhere in the middle – the pendulum has been at the biomedical end, and one session at WCPT seemed to suggest it swing entirely the other way. As Bennett suggests, the RCT in nonpharmacological studies needs to have a slightly different set of rules coupled with adjunctive data collection (pre-RCT and within) to capture the unique complexities faced.

    Geoff

  9. Tasha Stanton says:

    Great discussion!! First, I definitely agree that other study designs besides randomized controlled trials are important to consider when making clincal decisions. I’d argue that a well-designed case series (tested in an appropriate health condition) gives us a lot of strong and really important information about treatment effectiveness. And Diane, adding my $0.02 to your $0.02, I also often think of case studies as good ‘informers’ for future, larger research studies. (So now that there is $0.04 behind this idea of case studies informing larger studies, I reckon it has stronger evidence via face validity! …just kidding!).

    In all seriousness, I just felt that with the combination of: 1) heated discussions about the limitations of the RCT study design when testing the effectiveness of physiotherapy intervention (followed by loud cheering and applause) AND 2) the lack of RCT findings presented in the main forum, made me a bit uneasy.

  10. Diane Madras says:

    Interesting discussion! I was unable to attend WCPT, and appreciate the thoughts presented here. Personally, as a PT (MS) and cardiovascular physiologist (PhD), I find myself vacillating between the desire for RCT results, and the enjoyment of reading case studies. My hope would be that the personal nuances observed through case studies lead to larger well designed RCTs to try to elucidate more of the why of the nuances, and not report on just the ‘what’. Unfortunately, in the US, the push in PT programs is to train ‘consumers of research’ and not ‘producers of research’. I currently teach PT at a small liberal-arts university that does not have research facilities. This only allows case-study type research, or small sample-size studies, but my hope is something is better than nothing. (my $0.02)

  11. Geoffrey: who wants to “judge PT treatment exclusively on evidence (traditional RCT) derived from a biomedical model”? This is the most common straw man about EBM that there is, and it is scarcely possible to mention EBM in public without someone trotting this out and warning us not to throw the baby out with the bathwater. I think Tasha’s post was about an observed pattern of dismissal of evidence in place where one would hope not to see that, and not about how RCT results should be the only thing anyone ever considers.

    Anonymous Reply:

    Hi Paul,
    My post was more directed at Keith’s comment that seemed to suggest case reports and qualitative research are not valid forms of evidence. In my experience, there are many who are frustrated that physio research has not lived up to the standard of the many successes of research derived from the medical model. Based on the sessions I attended and the people I spoke to at WCPT there seems to be support for a model of EBP that meets the unique challenges of PT research, as opposed to trying to fit our research within the medical model. I don’t disagree with any of the posts, my interpretation of WCPT had more to do with illuminating other forms of evidence as opposed to being dismissive. Maybe there is no distinction there.
    Geoff

    Neil O'Connell Reply:

    I spend a week away and a really juicy discussion comes up! What worried me was the underlying driver behind most of the discussions Keith and Tasha have mentioned – “we know physio works”.

    While human and understandable it is plain wrong – we know no such thing. To take that argument then we must know that all of physio works because we continue to use it. Who would be prepared to bet that all approaches in the physio canon are valid and effective? – it really would be a remarkable turn-up if that were true, particularly given that many approaches are effectively underpinned only by arguments from an often charismatic authority figure who just discovered it in a beautiful data-free world.

    The assumption that in healthcare “something is better than nothing” does not stand, there are many ways in which an ineffective treatment might be detrimental but that this effect is difficult to measure, and all the time we would observe through our rose tinted specs that our treatment seems to “work”.

    Other methodologies may be able to put flesh on the bones of an observed effect (or lack of) but only an RCT can tell you whether the effect is there. And if a hundred RCT’s in a given area, all done differently, all with different strengths and weaknesses, all tell you the same message – then really it might be time to believe the RCTs.

  12. Tasha, alternative medicine is full of enthusiasm for “prescribing placebo” these days! It’s all the rage!

  13. Tasha Stanton says:

    And Keith – brilliant Seinfeld quote!

    Perhaps, ‘Serenity Now!’ is in order?

  14. Tasha Stanton says:

    Hi Keith and Paul,

    Thanks for the replies and I’m happy to hear that I wasn’t the only one leaving the conference with a bit of a sinking feeling.

    I worked as a clinician for a bit and so I can appreciate that feeling of ‘knowing’ a treatment works. You apply a treatment to a patient (who has long-standing pain) and immediately after the treatment the patient’s pain is gone. Therefore your treatment (and your treatment alone) caused this and the knowledge that this treatment ‘works’ is set in place. However, I think it is massively important to realize that the improvement in pain still might not be due to the treatment. I chatted about this with a colleague the other day, and our thoughts were that taking part in an RCT where one of the treatments is a placebo, may be a good experience. Then you experience that absolutely shocking realisation that patients, whom you applied the placebo treatment to, are telling you that their pain is completely gone. If we follow the same line of thought as above, then we should start prescribing placebos, because that treatment ‘works’.

    As you do Keith, I find it worrying that there is no consideration that the truth might be that treatment is no better at improving patient outcome than placebo. If we are happy to prescribe treatment based on anecdotal history and case study findings alone, then I think we are taking a step backwards as a profession.

  15. That is quite a disappointing report. Unfortunately, it gives comfort and support to a pet theory I have: that there’s much more bad science, pseudo-science, and even outright quackery going on in physical therapy than most people realize.

    I hope some others who were present will comment as well.

  16. lorimer says:

    I can see the disappointment and, also being a Seinfeld fan, was waiting for someone to use a knife and fork on their chocolate bar. However, I would also like to offer an alternative view – I seldom try to get to many oral presentations at such big conferences – I go for the posters. I actually request posters with my submissions for these things because it gives an opportunity to provide enough information to satisfy the lunchtime visitors and enough unsaid to get the keen ones over to chat. I find the posters have the most recent stuff and they are no less robust than the talks. I agree with Tasha that some seriously good studies were posters but I don’t think this is necessarily bad. Unlike you Keith, I was encouraged by how far PT has come in the 8 years since Barcelona. The standard of research is HEAPS higher and I think that will continue. I have sympathy however for your frustration – how long can one look for the pony? (reference to “There must be a pony” by Jim Kirkwood.)

  17. Keith Smart says:

    Hi Tasha.
    I agree, it was a bit bizzare.
    I’m also sorry to say that i left the conference more than a little disappointed.
    I attended various sessions and heard a number of the chairs/platform presenters (i won’t name them) who would generally be considered ‘big cheeses’ in the world of physio explicitly criticise clincal trials on the grounds that they did not demonstrate efficacy for treatments which we (menaing they) know work because we ‘see’ it in the clinic. That the evidence might be an accurate reflection and opposite true was not even considered.

    And this from leading academics! I was dismayed. Just as valid as forms of evidence, they suggest, were case studies, case series and qualitative evidence (e.g. the treatment was no better then placebo/a comparative intervention but the patients were more satisfied – so that’s ok and evidence enough for us to keep using that intervention).

    I heard one presenter make reference to ‘evidence-biased practice’ which is simply absurd in my view. They went on to suggest that, and i quote, “if you want a scientific answer look at the p-value, but if you want a real answer, ask the patient”!

    Well, if this is the position of some of our most highly regarded professional leaders i fear greatly for our profession’s future credibility and standing.

    Turming our backs on robust scientific methods of enquiry would be a betrayal of the enlightenment, and could, worse case (study!) scenario, herald ‘an age of endarkenment’, to quote David Colquhoun. http://www.guardian.co.uk/science/2007/aug/15/endarkenment

    In another episode of Seinfled, George Costanza says to Jerry:
    “Jerry, just remember, it’s not a lie if you believe it”. Indeed.

    Geoff Bostick Reply:

    Hi Keith,
    I believe I was at the same session. My impression was a little different though. What I heard was traditional RCTs tend to control things that are emerging as important contributors to PT outcomes such as context, therapeutic relationship, meaning responses etc. That RCTs are the gold-standard may lead to spurious assumptions that potentially effective treatments are deemed non-efficacious. This is not to say the traditional RCT is not valuable, but considering the contextual effects believed important in PT treatment, perhaps alternate views on evidence need to be considered.
    As much as it does not make sense to make clinical decisions on case studies and qualitative research alone, it also does not make sense to base decisions on RCTs alone. However, this does not mean either forms of evidence are not valuable.
    I do not believe most PT professionals and academics would suggest abandoning the traditional RCT, but rather elevating other forms of evidence to address the fact that illness experience is multi-faceted necessitating a plualist approach to research and treatment. This opens the door to important contributions of designs such as pragmatic RCTs, qualitative research and case series.
    The point I took was that PT treatment, more so than pharmaceuticals, have important contributions from ‘non-specific’ effects of treatment. These effects are not easily measured and mounting evidence suggests they are not to be regarded as nuisance. Thus, to judge PT treatment exclusively on evidence (traditional RCT) derived from a biomedical model seems to suggest contextual factors are not valid.

    My $0.02. All of us need a little Bizzaro to keep us balanced. Upon reflection, organizers might agree with Keith and Tasha, though I commend them for taking an atypical approach.