Reliability of Self-Reported Concussion History in Retired NFL Players
- 1. Department of Psychiatry, University of Texas Southwestern Medical Center, USA
- 2. Department of Neurology & Neurotherapeutics, University of Texas Southwestern Medical Center, USA
- 3. Department of Neurological Surgery, University of Texas Southwestern Medical Center, Dallas, USA
- 4. Department of Psychology, Southern Methodist University, USA
- 5. Department of Neurology, Washington University in St. Louis, USA
- 6. Center for Brain Health to Behavioral and Brain Sciences, University of Texas at Dallas, Dallas, USA
Abstract
As with other medical events, concussion history is typically dependent upon self-report, although little is known about the accuracy or reliability of such information. We assessed the reliability of lifetime self-reported concussion histories from 27 retired players from the National Football League (NFL) at baseline and one to two years later. The following data points were collected: history of any concussion with loss of consciousness (LOC), number of concussions with LOC, and total number of concussions (with or without LOC). We found perfect agreement between reports at time 1 and time 2 in terms of presence or absence of concussion with LOC. Intra class correlations revealed significant agreement between time points on number of concussions with LOC (ICC = 0.72) and total number of concussions (ICC = 0.90). Retrospective reports of concussions appear to be reliable over time.
Keywords
Concussion , Football , NFL , Head injury , Reliability
Citation
Didehbani N, Wilmoth K, Fields L, LoBue C, Strain J, et al. (2017) Reliability of Self-Reported Concussion History in Retired NFL Players. Ann Sports Med Res 4(4): 1115.
INTRODUCTION
Reliability of Self-Reported Concussion History
Approximately 1.6–3.8 million sports and recreational concussions occur each year, making this an important issue to address [1]. Potential later-in-life consequences associated with a history of concussion have gained increasing attention in recent years. In addition to the earlier onset of dementia in some cases [2], evidence suggests that a history of concussion in some populations is associated with greater hippocampal atrophy, lower memory function [3], and depressive symptoms later in life [4,5]. Concussion history reporting is generally based upon retrospective self-report; although uncertainty in the reliability of self-reported concussion information has been noted as a limitation in the literature. First, concussions often occurred years or decades earlier, which may affect recall. Second, concussion awareness and knowledge was more primitive several decades ago [6], and concussions likely went undocumented due to unawareness and/or underreporting by athletes [7-9]. Thus, in most cases, we are left with self-reported histories of concussion lacking in medical documentation, and the reliability of selfreported concussions remains unknown. The purpose of the present investigation was to examine the temporal consistency of concussion reporting within an older group of individuals with a history of concussion.
To our knowledge, three published studies have specifically investigated whether retrospective self-reports of concussion history are reliable. Kerr, Marshall, and [10] sent two surveys, 9 years apart, to retired National Football League (NFL) athletes asking them to report the total number of concussions they sustained during their professional career. They found that the retired players provided moderately reliable concussion histories over time (62.1% repaorted the same number of concussions; Cohen’s κ = 0.48). However, the authors categorized number of concussions into three groups (i.e., 0, 1-2, and ≥ 3), decreasing the variability of total reported concussions greater than three. For example, athletes who reported 3 versus 10+ would be collapsed in the same category which likely reduces generalizability of their findings, particularly in the > 3 category. A more recent study by [11] matched 130 former collegiate athletes’ self-reported concussion histories to clinically documented concussions that occurred between 1996 and 2012. They investigated total number of concussions recalled including sports related and non-sports related head injuries compared with clinical records. Intra class correlation coefficients (ICC) found low agreement (ICC = 0.21) between the number of athlete-recalled and clinically diagnosed concussions. Specifically, the athletes failed to recall 32% of clinically documented concussions and reported additional concussions, of which 77% were not clinically documented. However, the authors noted that the percentage of concussions both self-reported by athletes and clinically noted were equivalent after 2004, possibly reflecting improved concussion awareness, detection, and/or reporting. Aside from these studies, little is known about the reliability of concussion reporting, particularly among older individuals recounting injuries sustained years or decades ago. A more recent study by [12] followed children for 25 years across multiple time points with documented head injury and examined reliability of head injury recall due to car accidents, falls, and sports-related injuries. They found that recall was dependent of age at injury, time since injury, and severity of injury. Injuries that occurred in early adulthood versus childhood were more accurately recalled as was more severe injuries. The logistic regression assessing accurate reporting of head injury, only age at injury was significant. This study demonstrated the importance of assessing concussion recall and how it may differ across younger age groups.
In clinical settings, medical professional often rely on selfreport to develop treatment plans and for this reason, research this investigation aimed to examine the temporal consistency with which retired professional athletes provided retrospective estimates of concussion history. Since concussion information is differentially described in the literature, we sought to identify whether three commonly used concussion details, i.e. presence of any concussion with loss of consciousness (LOC), number of concussions with LOC, and total number of concussions (with or without LOC), were reliably reported in a sample of retired players over a period of approximately two years.
METHODS
Participants
Retired athletes were recruited between 2010-2014 through local meetings of the NFL Players Association in North Texas and word of mouth among retired players as part of a larger ongoing investigation. Athletes presented either alone or with a caregiver and were told that the purpose of the larger study was to examine the effects of playing a professional sport over time. The larger set of retired athletes included 84 former professional players, and those returning for follow-up study that underwent concussion history assessment at both time points were included in the present analyses.
Procedures
A total of 27 subjects underwent baseline testing and were seen for follow-up one to two and a half years later and completed concussion history questions at both time points (M= 20.59 months, SD= 5.05, range:11-31 months). Baseline and followup sessions consisted of a clinical interview, neurological and neuropsychological evaluation, and feedback about their results. Initial interviews were conducted on the first visit, immediately prior to the start of the neuropsychological evaluation. A second interview was completed with the athlete and often times a family member by a behavior neurologist immediately prior to the neurological examination. Self-reported lifetime concussion history was obtained during the clinical interview by both a neuropsychologist and neurologist at baseline and followup. No new concussions were reported for the 27 athletes in this study between the two time points. The majority of the concussions reported were sustained during their NFL career, and time since retirement is an approximated time since the injury. During the first clinical interview, former athletes were asked to report the number of total lifetime concussions they had sustained (including those outside of the NFL and other nonsports injuries), and for each concussion, describe the associated symptoms. A behavioral neurologist reviewed the concussion history [“Did you ever suffer any injury to your head, with or without loss of consciousness (childhood, high school, and college, pro, other)?”] with the athlete and family member, if present, and captured three concussion variables for each time point: 1). Presence of any concussion with LOC, 2). Number of concussions with LOC, and 3) Total number of concussions (with or without LOC). Subjects were followed up with more detailed questions during the neurological exam with a behavioral neurologist. Follow-up questions included more detail of cognitive, physical, and behavioral symptoms and duration of symptoms for each reported concussion. Diagnosis of cognitive impairment was made based on neurocognitive performance and the consensus of the neuropsychologist and neurologist.
Statistical Analyses
Reliability was assessed using frequency analysis for history of concussion with LOC and ICC for estimates of number of concussion (number of LOC concussions and total number of concussions) between the two reporting periods. The ICC formula is the ratio of the between-subject variance and the total variance (i.e., sum of the between- and within-subject variances) based on a general linear model of the number of total or LOC concussions. Each variance estimate was substituted into the ratio formula for ICC. In addition, partial correlation analyses examined whether time since retirement affected consistency of concussion estimates from baseline to follow-up, controlling for time since retirement.
RESULTS
Twenty-seven total participants were included in the analysis. Two athletes were not clear on number of lifetime concussions and estimated “about 5-8 every season” and noted it was “too many to count.” For these two athletes, we did not include their total concussion counts since they were uncertain (which explains missing data in results). Ages ranged from 42 to 81(M = 62.74; SD = 10.91) and years of education ranged from 15 to 18 (M = 16.26; SD = .81). Twenty-one participants were Caucasian and six were African American. A total of eight participants were diagnosed with cognitive impairment: 3 subjects had dementia, four had mild cognitive impairment (MCI), and 1 participant had a fixed cognitive deficit (i.e. static, rather than progressive in nature). Participants played a mean of 9.70 years professionally (SD= 3.74, range: 2-18 years) and had been retired from the league between 7 and 50 years (M = 32.07, SD = 12.01).
All 27participants consistently reported lifetime concussion histories for injuries with LOC from time 1 to time 2.On average, participants reported 1.59 concussions with LOC at baseline (SD = 2.04) and 1.65 at follow-up (SD = 1.70). Median number of concussions with LOC at baseline was 1.0 (range: 0–8) and at follow-up was also 1.0 (range: 0–5). A Bland-Altman plot demonstrated no bias in reporting of total number of concussions (Figure 1).The mean total number of concussions reported (with or without LOC) was 5.68 at baseline (SD = 4.13) and 6.04 at follow-up (SD = 3.46). Median total number of concussions was 5.0 at baseline and follow-up (ranges: 0–17 and 1–14, respectively). See Table 1 for the number of total concussions and concussions with LOC reported by each participant at time 1 and 2.
We also found that the mean total number of concussions for those with cognitive impairment at baseline was 5.88 (SD=4.32) and at follow up was 6.13 (SD=3.44), while the mean number of concussions with LOC at baseline was 2.00 (SD=2.62) and 1.75 (SD=1.83) at follow up, reflecting good consistency in this subgroup Table 1
ICC analyses revealed strong agreement between number of LOC concussions (ICC = 0.72, 95% CI [0.46, 0.86]) as well as total number of concussions (ICC = 0.90, 95% CI [0.78, 0.95]) which controls for years since NFL retirement Figure 1.
Cicchetti (1994) gave guidelines for interpreting the ICC. Less than 0.40 is considered “poor”, so we took that value as the lower limit to estimate power (rather than zero). Using N=27, the power to detect an ICC above “poor” is 0.77 or higher if the ICC estimate is 0.72 or higher. Our ICC estimate for LOC was 0.72 and our ICC estimate for total was 0.90.
DISCUSSION
Little is known about the reliability of concussion reporting over time, particularly among aging individuals recounting injuries from years earlier. We found that self-reported estimates of concussion histories as defined by an injury to the head were reliable across an 11–31month timeframe in our sample of retired NFL players including those with cognitive impairment. While based upon a relatively small sample of retired professional athletes, these findings provide additional support that concussion history reporting as obtained through interview appears reliable. Additionally, the injuries incurred by the athletes all occurred in adulthood which makes them more memorable than if they had occurred during childhood or early adolescence as demonstrated by Mc Kinlay et al. 2016 [13]. Our findings suggest higher reliability among retired athletes’ selfreport using the exact number of concussions (total and with LOC) compared to the self-reported and documented concussion histories found by [14] and the moderate reliability of survey responses across 9 years. The differences in the findings may
Table 1: Concussion Histories Provided at Baseline and Follow-Up.
| Concussions with LOC | Total Concussions | |||
| Athlete | Time 1 | Time 2 | Time 1 | Time 2 |
| 1 | 1 | 2 | - | - |
| 2 | 0 | 0 | 10 | 9 |
| 3 | 3 | - | 7 | 7 |
| 4* | 1 | 1 | 2 | 2 |
| 5^ | 0 | 0 | 0 | 3 |
| 6^ | 2 | 4 | 12 | 9 |
| 7† | 1 | 1 | 7 | 7 |
| 8^ | 0 | 0 | 4 | 4 |
| 9^ | 3 | 2 | 11 | 12 |
| 10* | 8 | 5 | 8 | 8 |
| 11* | 1 | 1 | 3 | 4 |
| 12 | 0 | 0 | 1 | 1 |
| 13 | 1 | 3 | 5 | 4 |
| 14 | 1 | 3 | 2 | 3 |
| 15 | 0 | 0 | 3 | 4 |
| 16 | 0 | 0 | 3 | 3 |
| 17 | 1 | 1 | 5 | 5 |
| 18 | 3 | 3 | 7 | 6 |
| 19 | 3 | 4 | 3 | 4 |
| 20 | 0 | 0 | 7 | 7 |
| 21 | 3 | 2 | 3 | 7 |
| 22 | 2 | 5 | - | - |
| 23 | 0 | 0 | 12 | 12 |
| 24 | 0 | 0 | 2 | 2 |
| 25 | 0 | 0 | 3 | 4 |
| 26 | 2 | 3 | 5 | 10 |
| 27 | 7 | 3 | 17 | 14 |
| Average | 1.6 | 1.7 | 5.7 | 6.0 |
| Median | 1.0 | 1.0 | 5.0 | 5.0 |
| Note: *Dementia, ^MCI, †Cognitively fixed, - Missing data | ||||
Figure 1: A Bland-Altman plot demonstrated no bias in reporting of total number of concussions.
be due to a number of factors including the collection of data (surveys versus in-person interviews), ways of reporting total concussions (grouping to 0, 1, 2 or, ≥ 3versus using precise numbers and number concussions during just NFL versus lifetime concussion), and time span between the time points (9 versus 2).
The largest limitation common to medical research is that we were unable to verify the accuracy of the self-reported concussion histories assessed using clinician’s interview questions asking about injuries to the head, as the retired athletes did not have medical record documentation of their injuries. Nevertheless, since concussion and medical history information typically relies upon self-report, our results are encouraging in supporting the reliability of concussion reporting. Along these lines, research on the reliability of self-reported medical history is limited, but indicates that more severe and chronic injuries are reported more consistently and show the highest reliability upon follow-up [15]. Our results add to this literature by documenting reliability of concussion history over time and provide a foundation upon which future research can expand to examine reliability in the context of longitudinal self-report along with concussion histories documented in medical or sports records.
Another limitation is that the data represent a convenience sample based on those athletes who returned for follow-up. Due to the longitudinal nature of the study, our sample may have recounted their concussion histories more frequently than other individuals with a history of concussion not enrolled in a longitudinal study. Data were also obtained from a well-educated, unique sample of retired elite athletes; thus, reliability of concussion reporting may not be representative of other groups who sustain concussions. Due to increased awareness and press regarding legal proceedings related to sports concussion, the desire to accurately recall concussion history may be more salient for the athletes in our sample than the general population. Additionally, contact sport athletes experience more opportunities for concussive injuries than non contact sport athletes and non-athletes, and more years of active participation in these activities further increases the potential for more injuries. However, it should be noted that whether this might be associated with more or less consistent concussion reporting than lower-risk groups is unclear. Another potential factor in concussion and other medical history reporting later in life could be the presence of cognitive impairment, which to our knowledge has not been previously explored. Whereas the number of subjects with cognitive impairment was limited in our sample, concussion reporting among these subjects as reflected in Table (1) showed similar consistency to the rest of the group. To further examine reliability of self-reported concussion history, future studies may benefit from utilizing larger, more heterogeneous samples and building in reliability reporting indices into longitudinal studies. Attention should also be given to methods of concussion history-taking (e.g., interview vs. selfadministered questionnaire), although in-person interviews have been suggested to be more sensitive for less severe or transient physical injuries compared to questionnaires [16]. Future investigations should also examine the role of cognitive impairment, age effects (e.g. reporting by adolescents vs. adults), and number of medically documented concussions in assessing reliability of concussion reporting.
The present study provides evidence that self-reported history of concussion with LOC, number of concussions with LOC, and total concussions were reported during clinical interview with good reliability over a period of approximately two years.
ACKNOWLEDGEMENTS
We would like to thank the retired players for their participation in the study.
