Many clinicians are interested in becoming involved in research but the path is not always clear and the opportunities available are not always evident. In this blog I will reflect on my own experience and share some ideas about how you might begin to tread this path and take advantage of some of the opportunities out there.
Suggestion 1: Reach out
There are now many clinical academics who have beaten the path into research; I recommend that you reach out and contact a researcher who is active in your area of interest. There might be opportunities to offer input to a project that is ongoing, for example undertaking some data extraction for a systematic review being undertaken by a PhD student. Involving yourself in projects, where available, enables you to begin to understand some of the processes involved, maybe enables you to be named on a publication and importantly helps you to develop your networks.
I often reflect of this TED talk 'My year of saying yes to everything' : the opportunity might not be exactly the project you would want to be involved with the potential spin-offs are invaluable.
Suggestion 2: Develop a healthy thick skin
Not everyone you reach out to will have the time to give and not everything you do will result in immediate success. Someone once said to me that resilience is the key to becoming a successful researcher - I think I agree. Rejection or failure once, twice or three times is not the end of the road; reach out to others, submit that paper or grant application elsewhere. I would suggest that you don't just carry on regardless though; learn from rejection and develop your approaches and ideas, a thick skin is necessary but it is healthy to know when to go back to the drawing board too.
Suggestion 3: Undertake some research training
Just as clinicians undertake clinical training, training in research is also important to help develop competence. Although I undertook some research training as part of my undergraduate training as a physiotherapist, I did not have the understanding or confidence to go further into research on the back of this. So, I looked to one of my mentors, Stephen May, to learn from his path into research and, as well as working incredibly hard on top of full-time clinical commitments, Stephen had undertaken an MSc in Health Services Research at the University of Sheffield. I enrolled part-time on the MSc in Health Services Research between 2003 and 2005 while working clinically and becoming a dad for the first time. The MSc exposed me to a different way of thinking and also enabled me to undertake a research project in a supportive environment that was eventually published.
This was only the start of the path for me, but an invaluable start. Now, in the UK, we have a research infrastructure that offers opportunities to some clinicians to to develop clinical academic careers. The details, including information relating to internships, MSc's in Clinical Research and PhD Fellowships, are available on the National Institute for Health Research (NIHR) website and I would strongly encourage you to read through the website and become familiar with the opportunities available. Also, again in the UK, as part of the NIHR infrastructure we have access to the Research Design Service (RDS). The RDS is accessible across the UK and local advisors will meet with you to discuss options relating to research projects and/ or research career development opportunities. The RDS also hosts events to disseminate information relating to these opportunities; details on the RDS website.
Suggestion 4: Expose your ideas to a wider audience
Again reflecting on my own experience, an important early hurdle to overcome was exposing my ideas to a wider audience. This might not seem such a big thing to some of you but this is a significant barrier to many in my own experience, with thoughts such as 'What if I am rejected?'
'What if I am criticised?', being commonplace. I was fortunate to meet another mentor, Julie Shepherd, who had a significant impact on me. I met Julie, an incredibly experienced and insightful physiotherapist, through clinical training and fortunately at the time Julie was an editor for a small newsletter/ journal. She encouraged me to write down some of my ideas and reflect on my clinical experience and publish them in the newsletter/ journal. This early support and encouragement was pivotal for me. I understand that many of you reading this might not have the opportunity to access the support and nurturing of your own 'Julie Shepherd' or 'Stephen May', and I recognise how fortunate I have been. But, exposing your ideas to a wider audience is still an important hurdle to overcome if you want to develop your research credentials. Fortunately, on average, the academic community is a supportive one so 'dipping' your toe and submitting a paper to a journal, or an abstract to a conference, might not be as traumatic as you fear.
The first peer-reviewed publication I was lead author on was a reflection on a clinical case written while working clinically. By no means was this ground-breaking research but an important part of my development and I would encourage you to think of ways to begin to get your ideas 'out-there', develop the healthy thick skin and overcome your fears/ insecurities and, also perhaps most importantly, develop your networks.
Suggestion 5: Actively seek opportunities
While opportunities are limited, they are out there but it is unlikely that they will come looking for you. As mentioned above, reaching out to researchers actively involved in areas that are aligned with your interests is a good start. One useful resource is the website jobs.ac.uk which advertises opportunities to undertake MSc's, PhD's and also advertises research/ academic positions.
Suggestion 6: Rome wasn't built in a day
I started my MSc in Health Services Research in 2003 and in the same year raised my hand at a conference to offer my help to Stephen May with a systematic review . Thirteen years on, my research career is still very much under development. Although I have been able to publish papers, present at conferences, successfully complete a PhD and secure funding for research, my learning continues. I guess the point I am trying to make is that it takes time to develop a clinical academic career and needs commitment and perseverance. I think it is important to recognise that learning never ends and that some of the principles outlined above are relevant throughout your career. To this end, I have recently taken a post at Keele University and recognise the fantastic opportunity this brings to continue to work alongside and learn from established researchers such as Professor Nadine Foster.
And finally, after sitting down to write this blog, I realise there is so much more I could say and I also recognise that there is so much more others who are developing and have developed clinical academic careers could offer. But, we need to stop somewhere; maybe we could pick up some further points and advice through the comments. I hope this is of some value to those of you looking for some thoughts and guidance. My final point; be pro active, seek a mentor, develop a plan...and persevere.
Friday, 23 December 2016
Critical appraisal of the SELF study
This is the first in a series of blogs that I hope will be a useful learning resource to help develop critical appraisal skills. We are awash with research, often with conflicting messages, and so it is important to understand which studies we can trust, which studies we should be cautious about accepting and what all this means for clinical practice. So, over the coming months, my intention is to select studies relevant to physiotherapy and write a critique of that research (apologies in advance; this is a long blog because I have taken time to explain the concepts in some detail; this won't be necessary in future blogs so I promise they will be shorter).
I thought it appropriate and timely to begin this series with a critical appraisal of some of my own work. So, this first blog will be a critical appraisal of the SELF study; a randomised controlled trial with the aim of evaluating the effectiveness of a self-managed single exercise programme in comparison to usual physiotherapy treatment. The published report of the SELF study can be accessed here; the open-access version here; the trial registration site here; the protocol here; and the pilot study here.
Some readers might be interested in the rationale and development of the self-managed single exercise programme but I won't go into detail on that in this blog but further information is available via this published paper; the open-access version is here too.
And finally, before we begin, for those of you not too familiar with the randomised controlled trial, I wrote about the basic design and rationale in a previous blog, here. This might be a useful starting point if some of the terms used seem unfamiliar or confusing.
Critical appraisal
There are numerous resources available to help us critically appraise studies. During this blog I'm going to take advantage of one such resource; the Cochrane risk of bias tool. The Cochrane risk of bias tool asks questions in relation to sequence generation (selection bias), allocation sequence concealment (selection bias), blinding of participants and personnel (performance bias), blinding of outcome assessment (detection bias), incomplete outcome data (attrition bias), selective outcome reporting (reporting bias) and other potential sources of bias, to help us judge whether we can trust the findings of a randomised controlled trial or whether we should be cautious or even reject the findings.
Sequence generation and the SELF study
Sequence generation refers to the methods used to generate the random allocation sequence. Remember, as discussed in the previous blog, random allocation to the different groups in a trial is important to ensure that the groups are comparable to begin with. If the groups are different at the start of a trial, for example in relation to pain severity, then it is likely that they will be different at the end of the trial too and so we cannot say with any confidence that the differences are due to the treatments received. Rather the difference is more likely explained by this confounding factor(s).
Valid random allocation increases the chance that groups in trials will be comparable in relation to the factors that we know about, e.g. age, gender, pain severity, but also factors that we might not know about or might not yet understand their importance, e.g. diet, sleep quality, genetics. In the SELF study we used a computer generated random allocation sequence which is regarded as a valid method. Other valid methods include coin toss or throw of a dice. Methods that would raise concern include allocation by date of birth, attendance at the clinic, clinician preference.
So, in relation to sequence generation we can conclude that the SELF study presents a low risk of bias, i.e. the method used was trustworthy.
Allocation sequence concealment and the SELF study
Once the sequence by which patients will be randomly allocated to different treatment groups within a randomised controlled trial has been generated, we can then begin to think about allocating patients. But, imagine we have used computer software, for example Microsoft Excel, to generate a list of random numbers where the number one means that the patient will be allocated to receive the new intervention and the number two means that the patient will be allocated to receive the comparator treatment or usual care. You have printed this list of numbers out and it is on the desk in your clinic. So, you can see that the next patient to be recruited to the study will be a number one and so allocated to the new intervention. But, when the next eligible patient arrives and they have multiple co-morbidities and low levels of motivation, and you do not believe that they will do well with the new intervention, what do you think the likely outcome will be? It is likely that the patient won't be recruited to the study. Then the next eligible patient arrives and they are fit and well and have high levels of motivation, what do you think the likely outcome will be this time? There has been a fair bit of research undertaken looking at this particular issue and where the random sequence is not concealed, i.e. the next allocation is known or can be predicted, or where the allocation concealment is broken, the benefit of the new or experimental intervention is likely to be over estimated. Here is an interesting review that suggests two-thirds of conclusions in favour of one of the interventions were no longer supported if only trials with adequate allocation concealment were included. Clearly then allocation concealment is an important issue to consider.
In the SELF study, we used consecutively numbered sealed opaque envelopes to conceal the allocation. The patient identification number had to be written on the envelope before opening so we could then see that the patients had been allocated in sequence and without knowledge of the treatment group to which they were going to be allocated. This method is regarded as a valid method of allocation concealment although it is preferable if the allocation is undertaken away from the recruiting clinic by a third-party, for example a clinical trials unit, using a telephone or web-based system.
In my experience it is not uncommon for authors not to report their methods of allocation concealment; sometimes this will be because the allocation has not been concealed but sometimes, especially in the older trials before reporting guidelines were introduced, simply omitted. At this point it is worth pointing out that the quality of reporting does not necessarily reflect the quality of the study. This is something I have looked at in a previous study for those of you interested. If in doubt, it is often worth contacting the authors directly to seek clarification.
The methods used to generate the random sequence and the methods used to conceal allocation, are methods to minimise selection bias. Selection bias is apparent when groups are not comparable to begin with or when inadequate methods of sequence generation and allocation concealment have been used. When we look at the baseline characteristics of the participants in the SELF study, they are comparable in relation to the data we collected. One apparent exception though is the participants in the usual physiotherapy group reported a longer mean duration of symptoms. However, on closer inspection, the duration data is positively skewed, i.e. not normally distributed, and so when this data is summarised using the median value the groups are comparable: median duration of symptoms in the self-managed single exercise group = 7 months and 6 months in the usual physiotherapy group.
So, in relation to allocation concealment and more broadly in relation to selection bias we can conclude that the SELF study presents a low risk of bias.
Blinding of participants and personnel and the SELF study
Blinding is where participants/ patients and study personnel, for example the clinicians, are unaware of which treatment the patient has received. In drug trials, blinding is readily achieved because pills can be developed which look the same but contain different ingredients. Hence, the patients and the clinicians will be unaware of whether the patient has taken the active drug or the placebo. In this context, you can probably appreciate that knowing which drug has been taken could effect the response or performance of the patient or clinician; 'my headache won't get better because I've only taken a placebo pill and my headache is real!' However, in many physiotherapy trials it is difficult or sometimes impossible to blind participants or personnel. In a trial of therapeutic ultrasound, this might be possible by using a de-tuned machine as the comparator but when comparing exercise to manual therapy, for example, this is not feasible because the differences between the interventions are clear to both patients and clinicians. However, even though blinding might not be possible it does not mean that this will not affect the results of the treatment particularly if the patients or clinicians have strong preferences or expectations. For example, consider a patient with a strong preference for the new intervention under investigation who is randomly allocated to receive usual care; this patient is likely to suffer resentful demoralisation, i.e. the patient becomes demoralised due to their non-preferred allocation which might bias the reporting of the treatment results adversely. If the patient was blinded then they would not have the knowledge of whether they received the intervention or control treatment but, as mentioned, this is not always possible. (In a future blog I will consider the impact of patient preference and expectations in terms of alternative trial designs and their impact on clinical outcomes).
In the SELF study, we initially aimed to blind the participants by describing both the self-managed single exercise programme and usual physiotherapy treatment, i.e. treatment prescribed by the treating physiotherapist as they would typically do for patients presenting with this problem, as 'physiotherapy' treatments. This approach recognised that different physiotherapists treat individual patients differently and most patients arrive at the physiotherapy clinic not knowing the specific ingredients of the physiotherapy they will receive. However, the research ethics committee would not permit this and requested that a full description of both interventions be offered to the patients as part of the process of informed consent. Clearly, we could not blind the treating physiotherapists otherwise they would not know which treatment to deliver but it was apparent that the beliefs and preferences of the treating physiotherapists did have an impact; details are available through two qualitative studies that we undertook alongside the pilot trial and main trial here and here.
Hence, although the potential impact is unknown, in relation to blinding of participants and study personnel and hence potential for performance bias, we can conclude that there is potential for a high risk of bias with the SELF study.
Blinding of outcome assessment and the SELF study
Increasingly in physiotherapy we are recognising the value of patient reported outcome measures. Previously we have placed value on measuring outcomes such as range of movement, strength, tissue status which we now recognise often don't associate well with the pain and disability that our patients complain of. In the SELF study the primary outcome was the shoulder pain and disability index (SPADI). The SPADI contains 13 items which the patient completes to detail their current level of shoulder pain and disability. Here, as with many other physiotherapy trials using patient reported outcomes, is the issue; the patient is reporting their outcome and, as we have seen above, in the SELF study were not blinded. Hence, blinding of outcome assessment was also not possible.
Thus, although the potential impact is unknown, in relation to blinding of outcome assessment and hence potential for detection bias, we can conclude that there is potential for a high risk of bias with the SELF study.
Incomplete outcome data and the SELF study
Clearly without data we cannot make a valid judgement about the effectiveness of our treatments. Hence, where outcome data is incomplete we need to exercise caution. In the SELF study only 70% of patients returned their SPADI data at 3-months, 56% at 6-months and 49% at 12-months. This is a concern because would our conclusions be different if all patients had returned their data? The answer is probably, or at the very least we could have been more confident in the conclusions drawn.
In this respect, both groups improved by what would be regarded as a clinically significant amount between 0 and three months; only the self-managed exercise group improved by a clinically significant amount between three and six months; and then the improvement between six and 12 months was not clinically significant for either group. At no time-point was there a statistically significant difference between the two groups, i.e. one approach proving superior to the other. Imagine though if we had the missing data, the changes or differences observed in the study from those participants who returned their SPADI data might be accentuated or attenuated; unfortunately we do not know.
One critical feature in relation to incomplete outcome data is whether there is a differential response between the different outcome groups, i.e. is there greater drop-out in one group compared to the other. This might be an important indicator relating to both the safety and effectiveness of an intervention, i.e. the intervention might be causing harm causing patients to drop out, so findings from a trial with differential drop-out should be treated extremely cautiously. Differential drop-out was not observed in the SELF study but bias could still be introduced if the reasons for missing outcomes differ. We do not know the reasons so caution should be exercised.
Thus, in relation to incomplete outcome data and hence potential for attrition bias, we can conclude that there is potential for a high risk of bias with the SELF study.
Selective outcome reporting and the SELF study
Researchers typically collect lots of data when undertaking research, and therefore potentially have lots of opportunity to detect whether a treatment works. As we know in research, positive findings can be a chance occurrence and if you collect enough data it increases the likelihood of attaining a significant finding. Or, for example, one section of an outcome measure might show significant findings if taken in isolation but not when treated as a whole measure.
However, if the outcome measure or measures that are to be collected and used to evaluate whether a treatment is effective are not detailed before the trial begins then we would struggle to know whether authors have only reported a biased selection of their data. This is a significant issue in health research with recent movements to require researchers to register their trials on open databases before the study begins and to also publish protocols. This ensures that the researchers do what they say they would do. Further detail is available here. In relation to the SELF study, as detailed above, the trial was registered prior to initiation and the protocol was published. The published report refers directly to the pre-specified outcomes and hence we can conclude that there is a low risk of reporting bias with the SELF study.
Other potential sources of bias and the SELF study
There are a number of other potential sources of bias that might affect the internal validity or trustworthiness of a randomised controlled trial. Some relate to the specific form of randomised controlled trial used, e.g. cluster, cross-over, and further detail is provided here.
From my perspective, beyond the potential sources of bias discussed above, there are no obvious other sources of bias to consider in relation to the SELF study but if you managed to read this far and have some thoughts then please do let me know through the comments section.
A final point to note; the discussion above has been focused on internal validity, i.e. trustworthiness, rather than external validity or generalisability. Clearly a valid/ trustworthy trial is of limited use if it is not generalisable, i.e. can not be applied more widely in practice. There are points to consider in relation to the external validity of the SELF study linked to its development and pragmatic methods and I will aim to shed some light on these issues in the coming weeks.
Conclusion
So, to conclude, based on this structured critical appraisal, it would be sensible to be cautious when interpreting the results of the SELF study. There are some clear strengths in the design of the study but, particularly, due to the risk associated with attrition bias, caution is needed. Although the pilot study reported similar results to the SELF study at three months, which is reassuring, replication of the study with particular attention to reducing attrition is indicated particularly because the simple self-managed exercise is simple and confers minimal burden on patients. Hence, if the self-managed single exercise programme proves comparable to usual physiotherapy treatment then there is pragmatic value.
In terms of future research, I am currently involved with the GRASP study which, although not a replication of the SELF study, overlaps to some degree and will provide further insight in due course.
Thanks for reading, hope it was useful; thoughts gratefully received.
I thought it appropriate and timely to begin this series with a critical appraisal of some of my own work. So, this first blog will be a critical appraisal of the SELF study; a randomised controlled trial with the aim of evaluating the effectiveness of a self-managed single exercise programme in comparison to usual physiotherapy treatment. The published report of the SELF study can be accessed here; the open-access version here; the trial registration site here; the protocol here; and the pilot study here.
Some readers might be interested in the rationale and development of the self-managed single exercise programme but I won't go into detail on that in this blog but further information is available via this published paper; the open-access version is here too.
And finally, before we begin, for those of you not too familiar with the randomised controlled trial, I wrote about the basic design and rationale in a previous blog, here. This might be a useful starting point if some of the terms used seem unfamiliar or confusing.
Critical appraisal
There are numerous resources available to help us critically appraise studies. During this blog I'm going to take advantage of one such resource; the Cochrane risk of bias tool. The Cochrane risk of bias tool asks questions in relation to sequence generation (selection bias), allocation sequence concealment (selection bias), blinding of participants and personnel (performance bias), blinding of outcome assessment (detection bias), incomplete outcome data (attrition bias), selective outcome reporting (reporting bias) and other potential sources of bias, to help us judge whether we can trust the findings of a randomised controlled trial or whether we should be cautious or even reject the findings.
Sequence generation and the SELF study
Sequence generation refers to the methods used to generate the random allocation sequence. Remember, as discussed in the previous blog, random allocation to the different groups in a trial is important to ensure that the groups are comparable to begin with. If the groups are different at the start of a trial, for example in relation to pain severity, then it is likely that they will be different at the end of the trial too and so we cannot say with any confidence that the differences are due to the treatments received. Rather the difference is more likely explained by this confounding factor(s).
Valid random allocation increases the chance that groups in trials will be comparable in relation to the factors that we know about, e.g. age, gender, pain severity, but also factors that we might not know about or might not yet understand their importance, e.g. diet, sleep quality, genetics. In the SELF study we used a computer generated random allocation sequence which is regarded as a valid method. Other valid methods include coin toss or throw of a dice. Methods that would raise concern include allocation by date of birth, attendance at the clinic, clinician preference.
So, in relation to sequence generation we can conclude that the SELF study presents a low risk of bias, i.e. the method used was trustworthy.
Allocation sequence concealment and the SELF study
Once the sequence by which patients will be randomly allocated to different treatment groups within a randomised controlled trial has been generated, we can then begin to think about allocating patients. But, imagine we have used computer software, for example Microsoft Excel, to generate a list of random numbers where the number one means that the patient will be allocated to receive the new intervention and the number two means that the patient will be allocated to receive the comparator treatment or usual care. You have printed this list of numbers out and it is on the desk in your clinic. So, you can see that the next patient to be recruited to the study will be a number one and so allocated to the new intervention. But, when the next eligible patient arrives and they have multiple co-morbidities and low levels of motivation, and you do not believe that they will do well with the new intervention, what do you think the likely outcome will be? It is likely that the patient won't be recruited to the study. Then the next eligible patient arrives and they are fit and well and have high levels of motivation, what do you think the likely outcome will be this time? There has been a fair bit of research undertaken looking at this particular issue and where the random sequence is not concealed, i.e. the next allocation is known or can be predicted, or where the allocation concealment is broken, the benefit of the new or experimental intervention is likely to be over estimated. Here is an interesting review that suggests two-thirds of conclusions in favour of one of the interventions were no longer supported if only trials with adequate allocation concealment were included. Clearly then allocation concealment is an important issue to consider.
In the SELF study, we used consecutively numbered sealed opaque envelopes to conceal the allocation. The patient identification number had to be written on the envelope before opening so we could then see that the patients had been allocated in sequence and without knowledge of the treatment group to which they were going to be allocated. This method is regarded as a valid method of allocation concealment although it is preferable if the allocation is undertaken away from the recruiting clinic by a third-party, for example a clinical trials unit, using a telephone or web-based system.
In my experience it is not uncommon for authors not to report their methods of allocation concealment; sometimes this will be because the allocation has not been concealed but sometimes, especially in the older trials before reporting guidelines were introduced, simply omitted. At this point it is worth pointing out that the quality of reporting does not necessarily reflect the quality of the study. This is something I have looked at in a previous study for those of you interested. If in doubt, it is often worth contacting the authors directly to seek clarification.
The methods used to generate the random sequence and the methods used to conceal allocation, are methods to minimise selection bias. Selection bias is apparent when groups are not comparable to begin with or when inadequate methods of sequence generation and allocation concealment have been used. When we look at the baseline characteristics of the participants in the SELF study, they are comparable in relation to the data we collected. One apparent exception though is the participants in the usual physiotherapy group reported a longer mean duration of symptoms. However, on closer inspection, the duration data is positively skewed, i.e. not normally distributed, and so when this data is summarised using the median value the groups are comparable: median duration of symptoms in the self-managed single exercise group = 7 months and 6 months in the usual physiotherapy group.
So, in relation to allocation concealment and more broadly in relation to selection bias we can conclude that the SELF study presents a low risk of bias.
Blinding of participants and personnel and the SELF study
Blinding is where participants/ patients and study personnel, for example the clinicians, are unaware of which treatment the patient has received. In drug trials, blinding is readily achieved because pills can be developed which look the same but contain different ingredients. Hence, the patients and the clinicians will be unaware of whether the patient has taken the active drug or the placebo. In this context, you can probably appreciate that knowing which drug has been taken could effect the response or performance of the patient or clinician; 'my headache won't get better because I've only taken a placebo pill and my headache is real!' However, in many physiotherapy trials it is difficult or sometimes impossible to blind participants or personnel. In a trial of therapeutic ultrasound, this might be possible by using a de-tuned machine as the comparator but when comparing exercise to manual therapy, for example, this is not feasible because the differences between the interventions are clear to both patients and clinicians. However, even though blinding might not be possible it does not mean that this will not affect the results of the treatment particularly if the patients or clinicians have strong preferences or expectations. For example, consider a patient with a strong preference for the new intervention under investigation who is randomly allocated to receive usual care; this patient is likely to suffer resentful demoralisation, i.e. the patient becomes demoralised due to their non-preferred allocation which might bias the reporting of the treatment results adversely. If the patient was blinded then they would not have the knowledge of whether they received the intervention or control treatment but, as mentioned, this is not always possible. (In a future blog I will consider the impact of patient preference and expectations in terms of alternative trial designs and their impact on clinical outcomes).
In the SELF study, we initially aimed to blind the participants by describing both the self-managed single exercise programme and usual physiotherapy treatment, i.e. treatment prescribed by the treating physiotherapist as they would typically do for patients presenting with this problem, as 'physiotherapy' treatments. This approach recognised that different physiotherapists treat individual patients differently and most patients arrive at the physiotherapy clinic not knowing the specific ingredients of the physiotherapy they will receive. However, the research ethics committee would not permit this and requested that a full description of both interventions be offered to the patients as part of the process of informed consent. Clearly, we could not blind the treating physiotherapists otherwise they would not know which treatment to deliver but it was apparent that the beliefs and preferences of the treating physiotherapists did have an impact; details are available through two qualitative studies that we undertook alongside the pilot trial and main trial here and here.
Hence, although the potential impact is unknown, in relation to blinding of participants and study personnel and hence potential for performance bias, we can conclude that there is potential for a high risk of bias with the SELF study.
Blinding of outcome assessment and the SELF study
Increasingly in physiotherapy we are recognising the value of patient reported outcome measures. Previously we have placed value on measuring outcomes such as range of movement, strength, tissue status which we now recognise often don't associate well with the pain and disability that our patients complain of. In the SELF study the primary outcome was the shoulder pain and disability index (SPADI). The SPADI contains 13 items which the patient completes to detail their current level of shoulder pain and disability. Here, as with many other physiotherapy trials using patient reported outcomes, is the issue; the patient is reporting their outcome and, as we have seen above, in the SELF study were not blinded. Hence, blinding of outcome assessment was also not possible.
Thus, although the potential impact is unknown, in relation to blinding of outcome assessment and hence potential for detection bias, we can conclude that there is potential for a high risk of bias with the SELF study.
Incomplete outcome data and the SELF study
Clearly without data we cannot make a valid judgement about the effectiveness of our treatments. Hence, where outcome data is incomplete we need to exercise caution. In the SELF study only 70% of patients returned their SPADI data at 3-months, 56% at 6-months and 49% at 12-months. This is a concern because would our conclusions be different if all patients had returned their data? The answer is probably, or at the very least we could have been more confident in the conclusions drawn.
In this respect, both groups improved by what would be regarded as a clinically significant amount between 0 and three months; only the self-managed exercise group improved by a clinically significant amount between three and six months; and then the improvement between six and 12 months was not clinically significant for either group. At no time-point was there a statistically significant difference between the two groups, i.e. one approach proving superior to the other. Imagine though if we had the missing data, the changes or differences observed in the study from those participants who returned their SPADI data might be accentuated or attenuated; unfortunately we do not know.
One critical feature in relation to incomplete outcome data is whether there is a differential response between the different outcome groups, i.e. is there greater drop-out in one group compared to the other. This might be an important indicator relating to both the safety and effectiveness of an intervention, i.e. the intervention might be causing harm causing patients to drop out, so findings from a trial with differential drop-out should be treated extremely cautiously. Differential drop-out was not observed in the SELF study but bias could still be introduced if the reasons for missing outcomes differ. We do not know the reasons so caution should be exercised.
Thus, in relation to incomplete outcome data and hence potential for attrition bias, we can conclude that there is potential for a high risk of bias with the SELF study.
Selective outcome reporting and the SELF study
Researchers typically collect lots of data when undertaking research, and therefore potentially have lots of opportunity to detect whether a treatment works. As we know in research, positive findings can be a chance occurrence and if you collect enough data it increases the likelihood of attaining a significant finding. Or, for example, one section of an outcome measure might show significant findings if taken in isolation but not when treated as a whole measure.
However, if the outcome measure or measures that are to be collected and used to evaluate whether a treatment is effective are not detailed before the trial begins then we would struggle to know whether authors have only reported a biased selection of their data. This is a significant issue in health research with recent movements to require researchers to register their trials on open databases before the study begins and to also publish protocols. This ensures that the researchers do what they say they would do. Further detail is available here. In relation to the SELF study, as detailed above, the trial was registered prior to initiation and the protocol was published. The published report refers directly to the pre-specified outcomes and hence we can conclude that there is a low risk of reporting bias with the SELF study.
Other potential sources of bias and the SELF study
There are a number of other potential sources of bias that might affect the internal validity or trustworthiness of a randomised controlled trial. Some relate to the specific form of randomised controlled trial used, e.g. cluster, cross-over, and further detail is provided here.
From my perspective, beyond the potential sources of bias discussed above, there are no obvious other sources of bias to consider in relation to the SELF study but if you managed to read this far and have some thoughts then please do let me know through the comments section.
A final point to note; the discussion above has been focused on internal validity, i.e. trustworthiness, rather than external validity or generalisability. Clearly a valid/ trustworthy trial is of limited use if it is not generalisable, i.e. can not be applied more widely in practice. There are points to consider in relation to the external validity of the SELF study linked to its development and pragmatic methods and I will aim to shed some light on these issues in the coming weeks.
Conclusion
So, to conclude, based on this structured critical appraisal, it would be sensible to be cautious when interpreting the results of the SELF study. There are some clear strengths in the design of the study but, particularly, due to the risk associated with attrition bias, caution is needed. Although the pilot study reported similar results to the SELF study at three months, which is reassuring, replication of the study with particular attention to reducing attrition is indicated particularly because the simple self-managed exercise is simple and confers minimal burden on patients. Hence, if the self-managed single exercise programme proves comparable to usual physiotherapy treatment then there is pragmatic value.
In terms of future research, I am currently involved with the GRASP study which, although not a replication of the SELF study, overlaps to some degree and will provide further insight in due course.
Thanks for reading, hope it was useful; thoughts gratefully received.
Subscribe to:
Posts (Atom)