Since its recent publication, a study by researchers at Boston University (1) finding that athletes who were less than 12 years old at the time of their first exposure to tackle football had more behavioral and cognitive problems later in life than those whose age of first exposure (AFE) to the sport was later has, predictably, garnered a great deal of attention from the mainstream media.
Just as predictably, a review of articles on the study which come up in after a brief Google search discloses that most simply reported the study's findings without asking scientists not involved in the study to comment. None examined the study's methodological soundness, and only one discussed its limitations, even those acknowledged by the study's authors.
In the interest of balanced and objective reporting, we asked a number of researchers and clinicians to review and comment on the study. Here's what they told us.
Self-selection bias
The authors acknowledged that the study cohort, because it was a convenience sample of self-selected participants, "could potentially lead to bias effects, especially if age of first exposure (AFE) play[ed] a role in selection."
While the potential self-selection bias was mentioned as a limitation in articles on the study by The Boston Globe and Time, and highlighted by the Globe's scientific publication, STAT (which, to its credit, even went to the point of pointing out that the study had limitations in the article headline), it received no mention in The New York Times article.
The scientists and researchers to whom we spoke viewed the potential self-selection bias as a significant concern.
Findings based on the results of a survey of "a convenience sample of self-selected participants could introduce bias effects if participants are familiar with the investigators work and media [coverage], where the consensus seems to be that playing football leads to long-term consequences," said Summer Ott, a sports neuropsychologist at the McGovern Medical School at UTHealth-Houston and director of the concussion program at the Memorial Herman IRONMAN Sports Medicine Institute. "Individuals who suspect they are experiencing difficulties as a result of sports participation or who are fearful of long-term issues may be more likely to participate," Ott noted.
Echoing Ott's concern, but going one step further, was sports neurologist Jeffrey Kutcher, National Director of The Sports Neurology Clinic at The CORE Institute in Ann Arbor, Michigan. Kutcher viewed the failure of the authors to disclose when the data for their study was collected as rendering all other scientific criticisms of the study essentially "moot."
If, as was almost certainly the case, the data for the study was collected after the publication during Super Bowl Week in 2015* of the BU group's earlier study (2) - one which also linked tackle football before age 12 and later life cognitive impairment - Kutcher believed that the chances were "pretty low" that the current study excluded people who had heard about, and were likely influenced by that study. As a result, the study subjects "were self-selected to be more likely to have more [behavioral and cognitive] problems," Kutcher contended.
"From a commonsense standpoint, it totally makes sense to try to reduce the potential risk for brain injury in young children," said Uzma Samadani, a professor in the Department of Neurosurgery at the University of Minnesota and Chair for Traumatic Brain Injury Research at the Hennepin County (MN) Medical Center. "But this study, like most of the work that has come out of the Boston University CTE group, is plagued by [a self-selection] bias." She likened the study's methodology to an emergency room doctor writing a paper saying that, among the patients who came into the emergency room with their tibia at right angles to their fibia, "there was an increased likelihood of having [suffered] childhood trauma. In other words, if you look for correlations in only a sick population rather than understanding them in a larger demographic you will likely find associations that are not generalizable."
Lack of control group
None of the media articles we reviewed addressed the lack of a control group as a study limitation, but the researchers we talked to all thought its absence was significant.
Not having one was "absolutely a problem," said Kutcher. "Sure, they compared two groups within the collected population. However, these people were self-selected to be more likely to be having problems."
The fact that the study did not include a control group of age-matched participants who played other contact sports associated with repetitive head impacts, such as soccer and boxing, and did not address it as a potential limitation "suggests that the authors are concerned primarily about football and not the generalized implication of pre-adolescent exposure to contact sports," said Ott. "The findings of the study would have been strengthened if results could be replicated with a randomized group and include participants who played other contact sports. With the cohort at hand, the results are suggestive, at best," she said.
Neurosurgeon Scott Zuckerman, Co-Director for Research of the Vanderbilt Sports Concussion Center Research Group, observed that, while the study only included athletes whose only sport was football, it is rare for athletically gifted athletes to only play one sport in middle school and high school, so he wondered "are we really isolating the effect of football" on long-term neuropsychiatric and cognitive outcomes. "Why not compare football players to control athletes rather than to normative data?" he wondered.
Using age 12 as cutoff
None of the media reports on the study we reviewed, except for a WBUR report containing excerpts of a "Here and Now" interview with one of the co-authors, Robert Stern, questioned whether comparing a group of football athletes who started before age 12 to the group whose AFE was later was arbitrary. Nor did any point out that 2016 study in American Journal of Sports Medicine (3), which found no association between AFE and later life neurocognitive impairment, considered age as a variable, not an absolute.
Using a cut-off of age 12 is still "somewhat arbitrary," said Zuckerman, a co-author of the 2016 AJSM study. Treating AFE as a continuous variable, Zuckerman argued, was often preferred over dichotomizing into two groups, where data can be lost. While the authors of the current study cited thirteen studies in the neurodevelopmental literature in support of using age 12 as the ideal cut-off, he noted that few specifically mentioned that specific age.
Controlling for variables
In terms of discussing controlling for potential variables which could cloud the study's findings, the best article we read - by a wide margin - was from WBUR (coincidentally, the NPR station at Boston University), no doubt due to the fact that it was based largely on an on-air interview with study co-author, Dr. Robert Stern, in which he explained how the study attempted to statistically control for a number of variables, including age, education, number of years playing football, and learning disabilities (based on criticisms that the results of the BU group's earlier study might have been due to the fact that more athletes in the AFE before age 12 group had learning disabilities, and thus had brains which may have already been uniquely vulnerable to repetitive head impacts brains).
Dr. Zuckerman, however, questioned the assertion that including a history of learning disability (LD) as a covariate had a "minimal influence" on the finding of a "robust relationship between AFE and long-term clinical dysfunction." Given that 2 of the 4 significant differences were lost when controlling for LD, the effect of LD was likely more than "minimal."
While the authors discussed and accounted for learning disabilities, noted Ott, there was no mention of ADHD, occupational and psychosocial (i.e. childhood and divorce) history, or other conditions associated with depression, apathy, and executive function deficits that could affect participant self-reporting and performance on cognitive testing. "Given the potential adverse and confounding effects of these factors on reported outcomes, more information on inclusionary and exclusionary criteria is needed to determine if these variables were included, not reported, or both."
Zuckerman wondered what comorbid medical and psychiatric conditions existed in the sample, noting that in the earlier BU study there was a significant overlap of medical and psychiatric conditions.
Limited reliability of telephone interviews and on-line testing
The study's authors candidly admitted to a host of limitations in their use of a telephone test of cognition (the Brief Test of Adult Cognition By Telephone or BTACT) and online testing of executive function and depressive symptoms and apathy.
They acknowledged that, while the BTACT was a valid, convenient, and cost-effective, telephone assessment of cognition, it was "not ideal," did not provide a comprehensive assessment of cognition; might not be sensitive to cognitive impairment in such a relatively young sample (average age of 50 years), and the global score used might not have captured the "diverse, and at times subtle, deficits associated with repetitive head impact exposure."
Further, the authors admitted that the use of telephone and online testing instruments precluded the ability to observe the participant's concentration and engagement in testing, particularly in the context of symptoms of depression and apathy, and called for future investigation of the relationship between AFE to football and cognition to use comprehensive neuropsychological testing.
Despite such admissions, only the STAT article mentioned that gathering data via telephone interviews was not as reliable as face-to-face neuropsychological examinations. The Globe and Times mentioned that the results were based on phone and online surveys, but not that such methodology was less reliable than in-person interviews. None focused on these limitations in depth or pointed out that the earlier BU study by the same group did do in-person interviews.
Not surprisingly, all of the researchers we contacted for this article pointed to these limitations as significant. Kutcher said the use of phone survey "greatly limited the overall value of the findings." While there were many reasons, the "biggest here," he said, "is that you're asking 50-year-olds what year they started playing a sport four decades ago. There's a good bit of guesswork that would go into answering that." The bottom line for Kutcher: "Having age at onset as a clinically predictive variable in this case was incredibly unprecise."
Evaluating cognition on the phone and via online testing is "not ideal," said Ott, "in that participants cannot be observed during testing nor can test administrators be sure if the former athlete completed the self-reported measures alone or with assistance. Because the authors did not report any measures that were employed to guard against malingering or suboptimal effort, it is difficult to assess whether these issues factored into the results as well."
Noting that the BU group's 2015 study used three objective neurocognitive measures, Zuckerman wondered "why now [the study] only used one objective neuropsychological test?"
No association between AFE to football and cognition
Surprisingly, none of the media reports we reviewed reported, much less considered the significance of, the fact that the study found no association between AFE to football and cognition as measured by the BTACT (Brief Test of Adult Cognition By Telephone). Such omission was all the more surprising, considering that the authors themselves admitted that the finding was "unexpected," given that they found in their 2015 study of former NFL football players that those who began playing football before age 12 exhibited worse neuropsychological test performance on episodic memory and executive function at mid-life compared with those who began playing football at 12 or older.
To Ott, the lack of a significant association between the online, self-reported measures (which found rates of impairment >40%) and the objective measure provided by the BTACT (which found only 7% of participants clinically impaired) "highlighted the potential degree of discrepancy between a patient's perception of everyday functioning and his neurocognitive ability" and suggested that "self-reported measures may not measure the same constructs as the objective measure."
Like the authors, Zuckerman said one possible explanation for the lack of an effect of AFE to football the study reported on the objective neurocognitive test (the BTACT) but positive effects on self-reported neurocognitive measures was to attribute the results on the subjective measures to their self-report nature, particularly as the convenience sample of self-selected participants may have been more likely to participate because of perceived clinical symptoms.
Number of years playing
Many of the articles pointed out that those who started playing football before age 12 but went on to play in high school, college or the pros were not found to be more likely to be at increased risk for apathy and behavioral regulation and depression based on the number of years of football played, but none questioned whether this was consistent with research suggesting that later-life neurocognitive or behavioral problems are dose-related.
"If this is a dose response process," said Kutcher, "one would have expected that if there was a difference between 11 and 12, there should also be one for 10-11, 9-10, etc. It seems that they were just searching for an effect at the 12-year cutoff."
Proceed with caution
"While this study examines an interesting relationship between youth football and its clinical implications," says Ott, "it is difficult to make a claim that exposure to contact football at a young age results in long-term cognitive and emotional problems" because any "causal effect is confounded by lack of a control group and failure to control and identify factors such as psychological and medical history, ADHD, and psychiatric illness."
While noting that the authors cautioned against using their findings to inform safety and/or policy decisions regarding youth football (a caution absent from any of the media reports we reviewed for this article, save the Globe's), and recommended that decisions about participation take into account the "important health and psychosocial benefits of participating in athletics and team sports during pre-adolescence" (noted in all of the media reports we reviewed), Ott was concerned that it was "unlikely that the findings and limitations of this study will be portrayed as such by the mainstream media. Rather, the headlines will focus on a cause-and-effect relationship instead of the association reported by the authors."
Ott feared that, as a result, "parents may blame themselves for letting their children play football. Children playing football before the age of 12 may be led to believe that they are destined to encounter future problems."
She noted that the mean age of participants in the study was 50, suggesting that some started playing over 40 years ago. Since that time, football has evolved significantly to include improved tackling techniques, reduced number of contact practices, enhanced safety equipment, and protocols for diagnosis and management of concussions." (developments mentioned in all the media reports reviewed for this article) "Rather than relying solely on information by the media," Ott said, "parents should form their decisions [about participation in youth football] after critically evaluating several sources and putting the findings into perspective. Parents should also be encouraged to have a rational conversation with a concussion specialist who possesses both research and clinical expertise and is best qualified to explain both sides of the issue."
"We don't actually know if reducing tackling before age 12 will be better or worse for the general public health," said Samadani. "It is possible that it would decrease the number of children who are obese from participating in sports at a very young age, which could potentially discourage them from participating later on, and increase their risk for all of the complications associated with sedentary lifestyle, such as hypertension, diabetes, obesity, cardiovascular disease, dementia, and thirteen different types of cancer. In addition, it is possible that it may be safer to teach tackling at a younger age than an older one. (This is hypothesized to be one of the reasons why girls are more susceptible to brain injury - it is that they have never learned to fall or be in contact the way boys do from a very young age.)"
"Ultimately," says Samadani, "we don't know if children who play football from a very young age are more likely to have had the problems described in the BU study regardless of their football exposure."
"The most dangerous aspect of most of the work done by the Boston CTE group is that they are constantly exploring the risks of brain injury from sport without at all accounting for potential benefits. We barely understand the benefits of football, but we do know that exercise in the children who are most vulnerable to obesity and sedentary lifestyle has bigger benefits than seen in their lean counterparts. In addition, the benefits of group risk-taking behaviors are poorly understood, but have implications for all adult occupations requiring risk. If our society does not allow our children to take risks, will we still be able to generate firefighters, policemen, surgeons, astronauts and other people with careers that involve individual and group risk? These are not things we understand at all," said Samadani. "To draw blanket conclusions about football exposure risk from a single uncontrolled study that does not even consider benefit is scientifically irresponsible."
** Timing the publication of the 2015 study for Super Bowl Week is nothing new for the BU group, which announced its long-since defunct "Hit Count" program just before the Super Bowl in 2012 and its long-since abandoned sensor certification program just before the Super Bowl in 2014 .
This article was originally published in Medium.com on October 3, 2017 by Brooke de Lench and Lindsey Straus
1. Alosco ML, Kasimis AB, Stamm JM, Chua AS, et al. Age of first exposure to American football and long-term neuropsychiatric and cognitive outcomes. Translational Psychiatry, 2017;7(9): e1236 DOI: 10.1038/tp.2017.197
2. Stamm JM, Bourlas AP, Baugh CM, Fritts NG, Daneshvar DH, Martin BM et al. Age of first exposure to football and later-life cognitive impairment in former NFL players. Neurology 2015;84: 1114-1120.
3. Solomon GS, Kuhn AW, Zuckerman SL, Casson IR, Viano DC, Lovell MR et al. Participation in pre-high school football and neurological, neuroradiological, and neuropsychological findings in later life: a study of 45 retired National Football League players. Am J Sports Med 2016;
