In part 1 of this blog post, I discussed the criteria I personally use to determine the strength of a study. To summarize, there were four criteria:
- The type of study
- The number of subjects
- The p-value of the results
- Which journal the research article was published
I realize there are more formal and sophisticated approaches to assessing a study, but these criteria will often eliminate the chaff from the wheat and are a good, quick approach to determining a good study.
This discussion was provoked when I came across a great study, Acupuncture for Chronic Pain: Update of an Individual Patient Data Meta-Analysis. Let’s apply our four criteria and see how this study looks. Before we do that, I recommend reading the abstract before continuing to read this post. [As an aside, there is a great little extension for the Chrome Browser called Unpaywall (available here). This little baby is fantastic. When you look at a research abstract, it scours the internet to see if that paper is available for free at an alternate, legitimate source. In the case of the paper discussed here, it allowed me to download the complete paper from the university where the authors work. Amazing!]
1. Type of Study
This was a systematic review of randomized controlled trials of clinical acupuncture studies in the treatment of nonspecific musculoskeletal pain, osteoarthritis, chronic headache, or shoulder pain. They included studies that had sham acupuncture or no acupuncture as a control. From part 1, we know that systematic reviews of well done randomized controlled trials are the highest level of evidence in clinical studies. This study was a meta-analysis. A meta-analysis is a subtype of systematic review where the authors take the data from the studies reviewed, pool the data from those studies, apply statistics to that data, creating new, stronger results than any of the individual studies. In other words, this study employs some of the highest forms of evidence. One ding you could say about the methods in this study is they combined several different types of chronic pain (nonspecific musculoskeletal pain, osteoarthritis, chronic headache, and shoulder pain) rather than one specific condition. A critic could say combining these into one study weakens the study because it assumes the given conditions are similar enough their data can be pooled. That may or may not be the case. Since the target of the study was chronic pain and each of these conditions involve chronic pain, the results may be fine. Of course, one could say there are plenty of other types of chronic pain not included in the study, that should be. Basically, this is why looking at inclusion and exclusion criteria under the study methods is important. One could also argue there were so many patients included in the process, any substantial differences between these conditions could still be seen. Let’s look at this next…
2. The Number of Subjects
As mentioned in part 1, one cannot apply important population statistical methods to any study of 30-40 subjects or less (that doesn’t mean there aren’t useful statistics at this level, just that they cannot be effectively applied to or assessed for large populations). After that, the sky is the limit: the more subjects, the stronger the study. If I see several hundred subjects, depending on how small a change is being assessed, I start to see a more powerful study. When thousands of subjects are involved, unless they are looking at very rare phenomena, I say the study is probably pretty powerful. In terms of acupuncture effectiveness for relatively common conditions, hundreds or thousands of subjects looks really, really good. When all was said and done, this study looked at data for 20,827 patients. In other words, tens of thousands of subjects. That is a large number of subjects and the results should be incredibly powerful.
The p-value is a statistical measure, simply put, of how likely a conclusion is false. This isn’t exactly true, but it is close enough without a discussion of null hypotheses. So a small p-value means the conclusion is more likely to be true. As we discussed, a p-value of less than or equal to .05 (which means there is a 5% or lower chance the hypothesis is false), is generally considered to be acceptable evidence in support of the conclusion. In this study, there was one main hypothesis: acupuncture was superior to alternatives for each pain condition. The results when statistically examined showed this hypothesis to have a p-value of less than .001. This means, statistically, there is less than a 0.1% chance acupuncture was not better than alternatives. In other words, acupuncture has more than a 99.9% chance to be better than the examined alternatives. Wow! What was also significant was this study also looked at how acupuncture was versus sham acupuncture. This is a huge issue in acupuncture research and my next blog post will look into this issue and why it is a problem. The bottom line with this study was it showed verum (or true) acupuncture to be statistically superior to sham acupuncture, something smaller, less powerful studies have struggled to show.
4. The Prestige of the Journal
Again, as discussed, where a research paper is published is important. This paper was published in The Journal of Pain, Official Journal of the American Pain Society. So how prestigious is this journal. While imperfect, a journal’s impact factor is widely used to determine how important a journal is. The Journal of Pain has an impact factor of 4.859 in 2017. Is that a good number?
Remember, the top 100 journals ranged from 41.577 for the journal Nature to 15.04 for the Journal of Hepatology. Well, 100 journals seems like a lot and 4.859 seems well below these numbers…
In actuality, 12,061 journals have their impact factors calculated. Only 1.7% have a factor of over 10. More relevant to the Journal of Pain, 6.5% of journals have a factor over 5 and 10.4% over 4. This means The Journal of Pain‘s impact factor places it between the top 6.5 and 10.4% of all journals, and probably closer to the 6.5 % number than the 10.4% number. That seems pretty good to me and indicates this journal is fairly well regarded.
When we take all of this into account: the high level of the type of study, the number of subjects, the low p-value of the results, and the fact that it was published in a relatively well regarded journal, these results are significant. In fact, this appears to be one of the most significant studies about acupuncture I have come across. So why go through all of this? Because science doesn’t give answers. Science provides evidence and supports conclusions, but never “proves” something. That means as a consumer of research, we need to be savvy enough to interpret results and place them on a scale of how important they are. In this case, there are tons of research that says acupuncture is helpful especially in treating pain. But there are plenty of studies that say it isn’t or that it doesn’t do it any better than sham acupuncture. The fact that this study was well done, provides high-level evidence with good statistical results, and was reviewed by strong peers (given the prestige of the journal it was published in), indicates this study should rank pretty high on the evidence meter and should be considered stronger than smaller, not as well researched studies published in lesser journals.
The bottom line is, this study provides strong evidence acupuncture effectively treats nonspecific musculoskeletal pain, osteoarthritis, chronic headache, and shoulder pain.
- Gann, L. (2017, Aug 8). What is considered a good impact factor? Retrieved April 7, 2019, from: http://mdanderson.libanswers.com/faq/26159.
- Miller, J. (2015, Jan. 1). How can we define the Power of Research study? Retrieved March 21, 2019, from: https://www.researchgate.net/post/How_can_we_define_the_Power_of_Research_study.
- Rumsey, D. J. (n.d.). What a p-value tells you about statistical data. Retrieved March 21, 2019, from: https://www.dummies.com/education/math/statistics/what-a-p-value-tells-you-about-statistical-data/.
- Vickers, A. J., Vertosick, E.A., Lewith, G., MacPherson, H., Foster, N. E., Sherman, K. J., Irnich, D., Witt, C. M., Linde, K. (2018, May). Acupuncture for chronic pain: update of an individual patient data meta-analysis. The Journal of Pain, Vol. 19 , No. 5, pp. 455 – 474.