Did a clinical trial show the dietary supplement pill Prevagen improves memory? Only when you forget about more than half of their data.

When I watch TV, it’s hard to avoid 30-second ads for the dietary supplement Prevagen like this one and that one (on YouTube) which both claim:

“In a computer assessed, double-blinded, placebo controlled study, Prevagen improved recall tasks in subjects.”

On January 2, 2020 there was an article by Jann Bellamy at Science Based Medicine titled AARP Report: insufficient evidence that dietary supplements benefit brain health which linked to a 2019 report by Global Council on Brain Health titled The Real Deal on Brain Health Supplements. On December 9, 2019 there was another article by Ashlee Kieler at Consumer Reports titled Feds, New York accuse maker of Prevagen dietary supplement of false advertising.

I went to the research section of the Prevagen web site and downloaded the .pdf file of their clinical trial, the Madison Memory Study. But that file omitted significant details of the study, like the gender of participants. I looked around and found them in a February 2016 magazine article published in Advances in Mind-Body Research titled Effects of a supplement containing apoaequorin on verbal learning in older adults in the community. As shown above, the first thing you would look for in results of a clinical trial would be a main effect – a significant difference between two groups given a placebo and a treatment (apoaequorin). Also note the strange grouping of subjects - both by gender and between placebo and treatment. The results section from the file on the Prevagen web site instead began by stating:

“While no statistically significant results were observed over the entire study population, there were statistically significant results in the AD8 0-1 and AD8 0-2 subgroups.”

A decade ago on January 25, 2010 I blogged about Bach Rescue Remedy and Anxiety and discussed another clinical trial that did not find a main effect. What they next did instead was data dredging (also known as p-hacking) – dividing the data into subgroups until they found an apparently significant effect. As is shown above, for the Madison Memory Study you can do this if you keep just 29% or 46% of the data  - and ignore the rest. On June 21, 2018 there was another article by Jann Bellamy at Science-Based Medicine titled Prevagen goes p-hacking. A five-minute Prevagen video titled What you need to know about Prevagen eventually (at 3:22) gets around to mentioning just a subgroup.

According to a pair of professors things are even worse. There was an article by chemistry professor Joe Schwarcz on January 14, 2019 at the McGill Office for Science and Society titled Prevagen for mental clarity? that questioned whether the active ingredient apoaequorin could survive digestion, enter the bloodstream, and cross into the brain. Another article by Gary L. Wenk on May 4, 2019 at Psychology Today titled Prevagen: The 21st-century placebo also raised that same objection.