Critical Appraisal of a RCT (February 2018)

Critical appraisal of a RCT (February 2018).

Exercise has been shown to be as effective as surgery for subacromial pain syndrome (SAPS) in a number of different studies [1,2,3].  While this is encouraging for proponents of exercise based physiotherapy, it also raises more questions than it answers.  What type of exercise? How much of it? For how long? Does it matter if it's painful or not? How, why, or even does either treatment actually help?  This month we examine a RCT designed to try to shed some light some of these questions by investigating whether one type of exercise (in this case non-painful eccentric training of the shoulder external rotators) produces better results than another (general exercise) in this patient group.

This is the third in a series of blogs (the first two can be found here and here).  Again, this blog will not provide a systematic or comprehensive critical appraisal of the chosen paper, or employ one of the many critical appraisal tools available, but will highlight what we consider to be three important elements to consider when trying to interpret the results of the trial and apply them to our patients.  This blog will consider various aspects of RCT design and implementation, an overview of which can be found here.

This month we will consider the potential sources of bias and the degree of confidence or doubt we have in results of the following RCT:

Shoulder external rotator eccentric training versus general shoulder exercise for subacromial pain syndrome: a randomized controlled trial. The International Journal of Sports Physical Therapy, 12(7), 1121-33.

This study was designed to investigate whether eccentric training of the shoulder external rotators (ETER) or general exercise (GE) produces better clinical outcomes in those with SAPS.   This study defined SAPS based on the presence of at least three of the following; a positive Neer, Hawkins-Kennedy, or empty can test, painful resisted external rotation, palpable tenderness of the supraspinatus or infraspinatus insertion, or a painful arc of abduction. The primary outcome measure was the Western Ontario Rotator Cuff Index (WORC), a patient reported outcome measure which considers physical symptoms, sports and recreation, work, lifestyle, and emotions.   Secondary outcome measures were a Numerical Pain Rating Scale (NPRS) for best, worst, and average pain, isometric strength, active range of movement, Y balance test, and Global Rating of Change (GROC).  48 participants were randomised into two groups (25 in the ETER group and 23 in the GE group) and underwent a six week exercise program including four visits to a physical therapist (the study was conducted in the USA).  The ETER group performed non-painful eccentric external rotation (3 sets of 15 with a 3 second eccentric phase), resisted scapula retraction (2 sets of 10), and a cross body stretch (3 reps with 30-45 second holds).  The GE group performed active flexion and abduction with no resistance (2 x 10 reps of each), and the same resisted scapula retraction and stretching excises as the ETER group. All outcomes were measured at baseline, 3 week, 6 weeks, and 6 months.  The study found that ETER produced statically significant superior results compared to GE at 3 weeks, 6 weeks, and 6 months according to WORC score, NPRS score, and isometric muscle strength. There were no statistically significant differences in active range of movement, Y balance, or GROC. The authors conclude that eccentric training may be efficacious to improve self-reported pain, function and strength in those with SAPS.

It is often easier to find faults when critically appraising a study, so let's start with the strengths of this RCT.  It asked a clinically relevant question and used a design appropriate to answer it, and the study protocol was published before the trial began which guards against bias.  Additionally, the interventions and outcome measures were well described.  This may sound simple but the quality of description of interventions and outcome measures in RCT’s, especially those including therapeutic exercise, is often poor limiting interpretation, application, and replication of results [4,5].

However there are a few aspects of the trial which we must consider before deciding how confident we can be in the reported superiority of ETER exercises over GE.  The three areas we feel are most important to consider are:

1.     Differential dropouts and statistical analysis.

2.     The choice of comparator.  

3.     Methods of randomisation and baseline differences between groups.

1. Differential dropout rates and statistical analysis.

Dropouts and the resultant incomplete data cause investigators a number of problems when it comes to analysing and interpreting RCT results. One of the key characteristics of RCT’s that reduces their susceptibility to bias is randomisation.  Random allocation of participants to different treatment groups aims to ensure that the groups are comparable at baseline in relation to both known and unknown factors that might influence the outcome of the trial.  This increases our confidence that any differences in treatment effectiveness are related to the intervention of interest rather than any baseline differences.  In order to maintain this control of bias, the random treatment assignment must be preserved through the whole trial including in the statistical analysis.  This form of statistical analysis is known as an intention-to-treat (ITT) analysis.  This analyses participants based on the group to which they were assigned at randomisation irrespective of what treatment they actually received or whether they completed the trial. This is considered the most appropriate method of analysis when comparing the effectiveness of treatments in RCT’s [6].

The authors in this study did not employ an ITT approach because they were concerned that the asymmetrical dropouts between groups may cause type I error (finding a significant difference when one does not exist). Instead they analysed only participants that completed the trial which is termed a completed cases analysis.  Whether asymmetrical dropouts cause error in the results of an RCT depends on why the data are missing (rather awkwardly termed ‘missingness’) and how it is handled in the analysis [7].  If those that dropout do so completely at random, then a completed cases analysis is reasonable because the two groups available for analysis are still based on chance alone.  However, if the data is not missing completely at random, a completed cases approach analyses a non-random subset of those that entered the trial and compromises the initial randomisation process. In this trial the differential dropouts between groups (39% in the GE group dropped out compared to 12% in the ETER group) increases suspicion that that the reasons for this may not have been completely random [8].  Using a completed cases analysis therefore increases doubt that the differences in treatment outcome can be attributed to the intervention of interest confidently.

Author Response:

Thank you, this is a great point and one that we deliberated over for quite some time.  There are pros and cons with ITT and we had concerns that week 3 between group comparisons could be inflated with the 2 subjects in the control group reporting a slight worsening in subjective outcomes and subsequently dropping from the trial early.  This will end up lending to your comparator group interventions discussion but generally speaking my biggest concern was that the active range of motion exercises used as a comparator may have slightly increased symptoms in the early phase of treatment for some subjects in the control group and carrying over week 1 data could falsely inflate between group differences in favor of the experimental group.  If the 6 month dropouts were the only issue using ITT to carry over week 6 data to 6 month data would be an easier decision but the early dropouts from the control group were a big factor in the decision.

2. The choice of comparator.

To accurately test how effective a treatment is it needs to be compared to something.  Studies where treatment outcomes are measured without a comparison group can show that a patient had a particular treatment and got better, but cannot show that they got better because of the particular treatment.  Controlled studies (both RCT’s and other non-randomised controlled studies) use a control group to demonstrate what would have happened if the participants had not had the treatment of interest, either by doing nothing (no treatment control), making patients think that they have had the treatment of interest but without administering the active components (placebo control), or by comparing to another treatment (active control). In this case an active control was chosen.  While this is reasonable because the alternative to using eccentric exercises would be to provide an alternative exercise-based treatment, the most appropriate comparator would be representative of current practice (so we know whether changing to this ‘new’ or ‘different’ treatment is better than what we already do). This study uses range of movement exercises (with one resisted exercise that was standardised across groups) to represent general exercise.  The authors themselves identify that this may not be representative to a typical exercise program used in clinical practice.  Unless this reflects our current practice, it makes it very difficult to know what these results mean.

The choice of control also introduces doubt as to whether we can be sure that it was the type of exercise that was the decisive factor in determining the results of this trial.  Both groups performed exercises that involved both concentric and eccentric phases.  This makes it less clear whether this was a true comparison between two distinct types of exercise. The control group also performed lower dose and lower resistance exercises than the ETER group.  Previous studies have suggested that exercise protocols that include resisted exercise may be more effective that those that do not [9], and that higher dose exercise may be more effective than lower dose exercise [10]. Even if we accept the reported differences in outcomes between the two groups, can we be confident that it was the type of exercise that caused them?

Author response:  This is another great point.  I am not confident that the comparator exercise in this trial was representative of what a PT would do in practice.  Simply having a patient actively move the shoulder through an elevation movement without load may not be a typical general exercise program.  Its possible that the various differences between exercise programs, ie load, specific isolated movement, arm position etc could be the reason for between group differences rather than the fact that the experimental group utilized an eccentric exercise.

3. Methods of randomisation and baseline differences between groups.

As described, the benefit of randomisation is that it theoretically balances both known and unknown factors that could potentially influence the outcome of the trial between the groups. This increases our confidence that any difference in outcome is due to the intervention of interest and not some other known or unknown difference between groups.  In this trial the researchers randomised patients by asking them to blindly place a pen on a table of random numbers. Manual randomisation methods such as this (or the use of a coin toss, drawing lots, of shuffling cards) introduce more doubt than more robust methods like using computer generated or remotely generated random numbers because either the participant or investigator could theoretically influence the process.  For example what happened if a participant landed equidistant between 2, 3, or even 4 numbers in the table? This raises an important point in the assessment of the risk of bias; we are not saying that the participants or investigators did unduly influence the randomisation process in this trial, just that the method used increases doubt because they theoretically could have done so.  We as readers will never know for sure if the results were unduly influenced, and that is why we asses for the risk of bias rather than actual bias itself.

If there was bias in the randomisation process this would mean that there were systematic differences between the two treatment groups.  However, the fact that there were systematic differences between the two groups does not necessarily mean that there was bias in the randomisation process. Randomisation can only maximise the probability that known and unknown factors are balanced between the two groups; it cannot not guarantee that this is the case. The bigger the sample size the more likely they are to be balanced (the reasons for this were discussed in a previous blog here).   In this study there were statistically significant differences in favour of the ETER group in strength (ABD/ER ratio) and Y balance. There were also non-significant (but not necessarily non-important) differences in all other baseline strength measurements, most range of movement measurements, best pain, and younger age.  We do not really know how, why, or even if exercise really does help patients with SAPS so we cannot know how, why, or if these baseline differences affected treatment outcomes.  If it is feasible that younger, stronger, patients with better range of movement  and balance are more likely to benefit more from exercise based treatment,  then we have to consider that it could have been differences between groups rather than the differences in treatment effectiveness that caused the differences in outcomes.

Author response:  I also agree with this, hindsight is 20/20, If we did a similar trial again the use of computer generated randomization would be much preferred.  The topic of baseline variables that could be affiliated with improved outcomes is very important.  I would love to have collected more baseline variables in a larger sample and run a regression off the responders to determine the patient characteristics that are consistent with a positive outcome.  In this case we are examining the between group mean but some participants have dramatic improvements over others.  It would be interesting to know which patients respond best to heavy load exercises and which do not respond as favorably.

Conclusion

This study reports that eccentric training of the external rotators of the shoulder produces significantly better results that general exercise in those with SAPS. However, as with any RCT, it is important to appraise the methods used before accepting the results.  Differential dropouts between groups and the way the data were analysed might increase risk of bias and hence decreases our confidence in the reported results, and baseline differences between groups and the choice of comparator increases doubt as to whether any differences in outcome can be specifically linked to the type of exercise performed. 

Author response:  One last point about this topic is that the progression of exercise mode, load and volume dosing is critically important.  The level of tissue irritability is also an important factor to help dictate exercise prescription and in clinical practice I wouldn’t arbitrarily prescribe eccentric exercises to any patient with chronic sub-acromial pain.  Progressions in arm position, type of movement (ie. Isometric vs light isotonic vs eccentric) and load/dose increases respective of patient tolerance and baseline strength will be important to integrate into future trials.  A pragmatic design that allows the clinician to manipulate these exercise prescription variables based on patient presentation will be important in future studies.  Thank you all for your interest and review of this topic.

Eric Chaconas

_______________________________________________________ 

Thanks for reading, hope it was useful; thoughts gratefully received.

Paul Regan, Chris Littlewood, Tomas Parraguez, Brian Cho, Sijmen Hacquebord

[1]

Haahr JP, Østergaard S, Dalsgaard J, Norup K, Frost P, Lausen S, Holm EA, Andersen JH, (2005). Exercises versus arthroscopic decompression in patients with subacromial impingement: a randomised, controlled study in 90 cases with a one year follow up.  Annals of Rheumatic Diseases, 64(5), 760-4.

[2]

Haahr JP, Anderson JH, (2006).  Exercises may be as efficient as subacromial decompression in patients with subacromial stage II impingement: 4–8-years’ follow-up in a prospective, randomized study.   Scandinavian Journal of  Rheumatology, 35(3), 224–228.

[3]

Ketola S, Lehtinen JT, Arnala I, (2017).  Arthroscopic decompression not recommended in the treatment of rotator cuff tendinopathy: a final review of a randomised controlled trial at a minimum follow-up of ten years.  The Bone and Joint Journal, 99-B(6), 799-805.

[4]

Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, Altman DG, Barbour V, Macdonald H, Johnston M, Lamb SE, Dixon-Woods M, McCulloch P, Wyatt JC, Chan AW, Michie S, (2014).  Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide.  British Medical Journal, 348:g1687.

[5]

Page P, Hoogenboom B, Voight M, (2017).  Improving the reporting of therapeutic exercise interventions in rehabilitation research.  International Journal of Sports Physical Therapy, 12(2):297-304.

[6]

Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org.

[7]

Bell ML, Kenward, MG, Horton, NJ, (2013).  Differential dropout and bias in randomised controlled trials: when it matters and when it may not.  British Medical Journal, 346:e8668.

[8]

Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC,  Devereaux, PJ, Elbourne D,  Egger M, Altman DG, (2010).  ConSoRT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials.  British Medical Journal,340:c869

[9]

Littlewood C, Malliaras P, Chance-Larsen K, (2015).  Therapeutic exercise for rotator cuff tendinopathy: a systematic review of contextual factors and prescription parameters.  International Journal of Rehabilitation Research, 38(2), 95-106.

[10]

Østerås H, Torstensen TA, Østerås B, (2010). High-dosage medical exercise therapy in patients with long-term subacromial shoulder pain: a randomized controlled trial. Physiotherapy Research International, 15(4), 232-42.