Research Papers and Peer-Reviewed Studies: A World of Difference

Note to reader: I wrote this blog on February 25, 2014 and updated it to include new information and updates one year later February 25, 2015 about a new "helmet add-on paper.

Last week, we posted to the site a group of four articles about a peer-reviewed study in the Journal of Neurosurgery showing that football helmet design affected concussion risk among a large group (or what scientists call a "cohort") of college football players.

Because the study's findings matched almost exactly the estimated difference in concussion risk in the STAR helmet ratings, a test developed by the co-lead authors of the study, Virginia Tech-Wake Forest's Stefan Duma and Steven Rowson, which is designed to measure the effectiveness of football helmets in reducing the forces that cause concussion, we also updated an existing article about those ratings to include a reference to the new study.

Just before those articles were published, we received a press release from the American Academy of Neurology about an abstract of a research paper on football helmets slated for presentation to the AAN Annual Meeting in Philadelphia in late April 2014.  The press release said that, using crash test dummies, researchers in Florida had found that football helmets do "very little" to protect kids against the rotational forces that cause concussion, and quoted the co-lead author, Dr. Frank Conidi, director of the Florida Center for Headache and Sports Neurology and Assistant Clinical Professor of Neurology at Florida State University College of Medicine, as saying that the study dispelled the assumption of "generations of football and other sports participants" that their brains are protected by their investment" in football helmets.

While numerous newspapers and reporters - with at least one notable exception (see below) - elected to take the research paper largely at face value in publishing stories (links to which then popped up repeatedly in our Twitter feed and were dutifully retweeted), MomsTEAM took a pass, which led many to wonder, on Twitter and via email, why?

For a number of reasons.

First, as many of you know, MomsTEAM has been covering the concussion beat for 14 years, longer than anyone in the mainstream media.  Since I and my staff, especially Senior Health and Safety Editor Lindsey Barton Straus, began writing about concusssions in general, and fooball helmets in particular, back in 2000, we have tried to live up to our reputation as the "trusted source" for youth sports safety information by making sure that anything we publish has been properly vetted and is based on methodologically sound scientific research.

As a result, we have become skeptics, critics and curators of credible information on the subject.  In a now very crowded space of concussion activists, bloggers, and a largely uneducated media, we continue to see our mission of keeping all sports active kids safe by separating fact and fiction, and making sure what we publish, while it may not be the first to discuss a particular development, is the most comprehensive and objective.

Second, all we had to report on was an abstract of a research paper.  With so many researchers vying for limited funding dollars and attention, the trend these days seems to be a game of jumping the gun, sending out press releases about abstracts of papers before they have appeared, if they ever do, in peer-reviewed journal papers.  While MomsTEAM occasionally reports on research papers, we are careful to point out that they have not been subject to scrutiny by the scientific community. 

In this particular instance, when Lindsey and I dug deeper, our investigation raised in our minds serious concerns that reporting on this particular research paper at this preliminary stage might give it more credence than perhaps - emphasis on the word, "perhaps" - it deserved. Without even talking to anyone with expertise in the subject, we were concerned about reporting on the paper knowing absolutely nothing about the methodology the researchers employed, especially when it involved an unspecified modification to the NOCSAE drop tests, purportedly to capture data about rotational forces.

Capturing data on rotational forces, much less setting helmet standards or designing helmets that reduce such forces, has been a monumental challenge for biomechanical engineers and helmet manufacturers for decades. We have been critical of NOCSAE in the past for its failure to test football helmets for their ability to reduce rotational acceleration (although, as we also recently reported, the organization says it will be announcing a new helmet standard at its next meeting in June which will for the first time include a standard on rotational acceleration).* Our original and updated article about the Virginia Tech STAR ratings also includes criticism by NOCSAE that it did not capture data on rotational acceleration (talk about the pot calling the kettle black!). As a result, we needed to be extra cautious in publicizing a paper employing a new methodology.

Third, that the research paper being publicized by AAN was commissioned by BRAINS, Inc., a Florida research and development company employing a co-author, Dr. John Lloyd, also raised a red flag. As we noted in our article about the new Virginia Tech-Wake Forest helmet study - which compared concussion data on an older model of Riddell helmet (the VSR4) versus a newer Riddell model (the Revolution) -  a similar conflict of interest led many in the concussion research community to discount a 2006 study reporting a difference in concussion risk between the Riddell Revolution and other helmets, peer-review notwithstanding, because it was funded by, you guessed it, Riddell.

Fourth, when we did ask respected researchers in the field to comment on the Condini-Lloyd paper, while they declined to comment on the record for MomsTEAM, we can say that they were not at all impressed by what little could be gleaned from the press release, one characterizing it, off the record, as "a train wreck."

And lastly, fifth, we weren't at all clear what the take-home message of the paper is. 

If the point of the paper is to say that current helmets don't protect very well against concussion, and, because they don't, the emphasis needs to be on coaches teaching proper tackling technique, players strengthening their neck and shoulder muscles, and officials more strictly enforcing the rules against helmet-to-helmet contact, then there was nothing new to report to MomsTEAM readers, as the fact that helmets are not designed to protect against concussions in the first place, and that proper tackling, neck strengthening, and rules enforcement are seen as more effective ways to reduce concussion risk, are positions we have been taking for years, and are part of Pillar Two of the Six Pillar approach to concussion risk management we outlined in our PBS documentary, "The Smartest Team."

On the other hand, if the point of the paper was to emphasize the point that football helmets should be tested on how well they attenuate rotational forces, which are the principal cause of concussions, and that new testing standards are needed, there was, once again, nothing new on which to report. There is already a well-formed consensus about the need for new standards, a subject on which MomsTEAM and I have reported for years. 

The bottom line is that, after considering all these factors, MomsTEAM decided not to run a stand-alone article, but to direct our readers instead to our existing, continually updated, content on football helmets and concussion risk reduction, including the following:

Virginia Tech Football Helmet Ratings: Helpful But Come With Limitations

Proper Tackling Reduces Risk of Catastrophic Injury Says Ex-Pro

Seven Ways To Reduce Risk of Traumatic Brain Injury In Sports

The Six Pillars of Concussion Risk Management: The MomsTEAM Approach

Making Sure Football Helmet Fits: A Simple, But Effective Way To Minimize Concussion Risk?

Still looking for an article about the Condini-Lloyd study that got it about right? Check out this article by Melissa Healy, one of our favorite sports health and safety reporters, which does a terrific job of at least presenting the major issues the research paper raises, albeit not in the depth that MomsTEAM does. 

FEBRUARY 25, 2015 UPDATE: A year later, we are waiting for the results of the Condini-Lloyd to be published in a peer-reviewed medical or biomechanical journal.  In the meantime, Dr. Loyd has come out with a new research paper, this one slated for presentation to the American Academy of Neurology annual meeting in April 2015, in which he concludes, apparently using the same drop test methodology as in his previous paper, that a number of helmet add-ons, including Guardian Cap, do not, according to his research, reduce the risk of concussion. Once again, a slew of news outlets have uncritically published the press release, and it is has been dutifully publicized in the Twittersphere.

But for pretty much the same reasons MomsTEAM elected not to publish an article about the new study, we have again chosen to take a pass on posting an article about Dr. Lloyd's research paper based on the AAN press release because, without knowing the details, we would essentially be "buying a pig in a poke."  

Nevertheless, in the interest of getting both sides of the story, here's what Guardian Cap had to say about Dr. Lloyd's new research paper:

Dr. Lloyd's study is concerning to Guardian Caps, a company who has devoted the past four years to providing athletes with a superior energy management solution. Guardian has been tested extensively at Oregon Ballistic Labs, Wayne State University, Veritas and at several Universities. Our test data is available for review upon request. Guardian is currently being used by over 35,000 players from youth through D1 College levels. Guardian has several concerns with the findings that Dr. Lloyd reports:

• Dr. Lloyd reported that three different add-on products "reduced linear accelerations by about 11 percent, but only reduced angular accelerations by 2 percent." This conclusion does not detail individual findings, but just averages all three products. Because he did not report individual findings, Guardian Caps may have, in fact, tested much higher than other products. Unlike the other products tested, Guardian Caps are decoupled from the helmet, further reducing rotational and angular accelerations.
• The article does not disclose that Dr. Lloyd is an inventor and is working on his own football helmet that uses non-Newtonian Fluids (starch and water mixture). In a previously published article, Dr. Lloyd states that his helmet reduces concussions up to 50%.
• A drop test is not a statistically significant way to measure rotational/angular accelerations. This test should be conducted on a linear impactor testing setup. Furthermore, researchers have yet to conclude on two other main factors that affect the end data: 1) which multi-axial accelerometer data acquisition system collects the most statistically significant data and 2) the proper mounting placement and method so as not to affect the significance of the data. Dr. Lloyd's failure to account for these key nuances in testing essentially invalidates any and all yielded data. 

The bottom line: caveat emptor.  

*Update: True to its word, NOCSAE did announce a proposed football helmet standard at its July 2014 meeting. For my blog about that meeting and the new helmet standard, click here.  

Brooke de Lench is Executive Director of the non-profit MomsTEAM Institute, Founder and Publisher of MomsTEAM.com, and Producer/Director/Creator of the PBS documentary, "The Smartest Team: Making High School Football Safer."  You can reach her via email (delench@momsteam.com) and follow her on Twitter @brookedelench.