I came across a great study from last year about the use of acupuncture in chronic pain including nonspecific musculoskeletal pain, osteoarthritis, chronic headache, and shoulder pain. This was a powerful study that was published in the Journal of Pain, the Official Journal of the American Pain Society. Before I get into the results, I thought I would spend some time talking about the basics of assessing a research article and why this article in particular is quite powerful, and some reasons it may not be.
The first thing I always assess when looking at a clinical research article for the first time is what type of study it is describing. There is a hierarchy of types of research useful for making clinical decisions for humans. At the lowest end is in vitro and animal research. In vitro literally means “in glass.” It refers to research that is done in the laboratory and does not involve a live human or animal. Animal research may or may not apply to humans and is likewise of low use clinically. While both of these may be very important research, and lead to profound clinical interventions (most drug research will start at this level), by themselves, they show very little effectiveness in humans.
Next up the hierarchy, there is individual or small group case studies showing some clinical effect of an intervention in one or a handful of cases. As the hierarchy climbs up the evidence ladder, it looks at types of research like case-control and cohort studies and then systematic reviews of multiples of these studies. The most respected form of research is a randomly controlled trial (RCT) and the best evidence of all is a systematic review of many RCTs. This hierarchy is based on the Oxford Centre for Evidence-based Medicine and can be found here. Basically, I have seen so many studies all the way up to small RCTs be completely reversed when looked at with a large and well constructed RCT. So, while animal, in vitro, case and cohort studies may be fascinating and lead to something really cool, I personally don’t give them much credence in my clinical decisions. Though, in the absence of any other evidence, I may employ them clinically.
So once I have a good idea of the type study I am looking at, I look at how many subjects were involved. The larger the study the more power the study has. The power of a study is actually a technical statistical term and refers to the ability to detect a difference if a difference actually exists. It depends on two things: the number of subjects and the size of the effect (difference). So something that has a very large effect, needs fewer subjects to detect. A small effect needs more subjects to detect. So for example, if something only affects 1 in 100 people, a study needs to look at many hundreds of people to see the effect consistently. While if something affects 1 out of 2 people, it should be easily detected in a score or two of subjects. Generally, the more subjects in a study, the more power it has and the more reliable and valid (two more technical research terms that we will not go into) the results are. I was always taught that anything under 30-40 subjects cannot be reliably assessed with statistics and should be the minimal number of subjects. To go into more depth in all of these would require a further discussion of statistics, and while I am rusty, I wouldn’t mind doing that, but this article is getting too big as it is.
The number of subjects also affects another number usually reported in a study: the p-value. Again, this is a statistical number showing the likelihood that the results are significant. In research we usually use a cutoff of the p-value as .05. This basically says that the chance that the results are false are 5%. If it is higher than this point, researchers say there is not enough evidence to support the results. Below this point, we say the evidence supports the results. Notice two things about my verbiage: I do not say prove or disprove the results and this all about probabilities. There is no research that proves or disproves a hypothesis; it can only add evidence that supports or does not support a hypothesis. And since this is all about probabilities, it is ALWAYS possible the research is wrong even with a very small p-value; it is just more and more unlikely to be wrong.
Another factor that may be important when looking at a study is where it is published. Some journals are more prestigious and more important than others. While there are several ways to quantify this is, all have some flaws. Generally, the most accepted form of this quantification is known as the impact factor of the journal. Generally, the bigger the impact factor, the more important and prestigious the journal is. The impact factor is based on how many times an article in that journal is cited in other research. And this is assessed, and therefore can change, annually. Unfortunately, the average impact factor for journals in a particular field of study will differ. To give you a range of impact factors, the highest ranked journal is Nature. It’s impact factor in 2017 was 41.577. The last of the top 100 journals is the Journal of Hepatology with an impact factor of 15.04. Also inherent in this discussion is that if research has not been published by a peer-reviewed journal, it has next to no impact and generally should be discounted as too flawed to be accepted by the authors’ peers. This is is an inherent issue on the internet, anyone who has a bone to pick can publish whatever they want. And there are many online journals that have no credence whatsoever. So it is important to assess the impact factor of a publication. And this is easy to do by simply entering the name of a journal followed by “impact factor” in your favorite search engine. Why is this whole thing important? Because it is hard to get into a prestigious journal. There is high competition for limited pages, layers of editors and peers that review your article, and basically, your paper needs to be damn good to get published in one.
So let’s summarize what we have so far when interpreting the results of a research article:
- The type of study matters: in vitro and animal research have little strength while a systematic review of randomly controlled trials has the most. Others fall into the middle of these extremes.
- The number of subjects in a given study is very important. Less than 30-40 subjects lacks statistical power, otherwise, the more subjects, the more power the study has.
- The p-value shows how likely it is for the results to be wrong. A p-value of less than .05 means the results are unlikely to be wrong.
- Where a research article is published is important. Looking at a journal’s impact factor can give you an idea of how important that journal is in its field of study.
So why I am telling you all of this? Because I found a really great article on acupuncture’s impact on different forms of pain. And it hits all of these points really well, meaning it is quite useful and potentially impactful. And we will discuss this article and how it hits each of these points in part 2 of this post!
- Miller, J. (2015, Jan. 1). How can we define the Power of Research study? Retrieved March 21, 2019, from: https://www.researchgate.net/post/How_can_we_define_the_Power_of_Research_study.
- Rumsey, D. J. (n.d.). What a p-value tells you about statistical data. Retrieved March 21, 2019, from: https://www.dummies.com/education/math/statistics/what-a-p-value-tells-you-about-statistical-data/.
- Sperber, G. & Flaws, B. (2017). Integrative Pharmacology, Combining Modern Pharmacology with Integrative Medicine [Revised Edition Textbook]. Boulder, CO: Blue Poppy Press.
- The Centre for Evidence-Based Medicine. (2016, May 1). OCEBM levels of evidence. Retrieved March 21, 2019, from: https://www.cebm.net/2016/05/ocebm-levels-of-evidence/.
- Vickers, A. J., Vertosick, E.A., Lewith, G., MacPherson, H., Foster, N. E., Sherman, K. J., Irnich, D., Witt, C. M., Linde, K. (2018, May). Acupuncture for chronic pain: update of an individual patient data meta-analysis. The Journal of Pain, Vol. 19 , No. 5, pp. 455 – 474.