Concussion test may not be panacea

Bryan Christie Design; reference imagery: Purdue Neurotraua Group / Purdue University

Concussions have become big business in the football world. With 1,700 players in the NFL, 66,000 in the college game, 1.1 million in high school and 250,000 more in Pop Warner, athletes and families across the country are eager to find ways to cut the risks of brain injury, whose terrifying consequences regularly tear across the sports pages. And a wave of companies offering diagnostic tools and concussion treatments are just as eager to sell them peace of mind.

That's actually a slogan for one company. ImPACT, the maker of the world's most popular concussion evaluation system, offers a 20-minute computerized test that players can take via software or online to measure verbal and visual memory, processing speed, reaction time and impulse control. The idea behind ImPACT (Immediate Post-Concussion Assessment and Cognitive Testing) and similar batteries is that doctors or athletic trainers can give a baseline test to a healthy athlete, conduct follow-up tests after an injury and then compare the results to help figure out when it's OK to return the athlete to play. Selling itself as "Valid. Reliable. Safe," ImPACT dominates the testing market and has spread throughout the sports world: Most NFL clubs use the test, as do all MLB, MLS and NHL clubs, the national associations for boxing, hockey and soccer in the U.S., and nine auto racing circuits.

On a broader scale, ImPACT, headquartered in Pittsburgh, sells tests and training to more than 7,000 pro teams, colleges, high schools and sports medicine centers from the University of Alabama to St. John's College in Zimbabwe. And it's picking up corporate partners and athlete endorsements. In the past year, Dick's Sporting Goods and Wells Fargo have announced initiatives to encourage widespread use of the test. In one ad touting Dick's promise to help fund ImPACT testing of up to 1 million middle school and high school athletes, former Steeler Jerome Bettis says, "Using tools created by ImPACT … young athletes will know when to sit out."

There's just one problem. Many scientists who are unaffiliated with ImPACT don't think the thing works.

"Through amazing marketing, the ImPACT guys have made their name synonymous with testing," says William Barr, an associate professor of neurology and psychiatry at New York University and former team neuropsychologist for the New York Jets. "But there's a growing awareness that ImPACT doesn't have the science behind it to do what it claims it does."

Mark Lovell, the CEO of ImPACT Applications, Inc., the company that makes and markets the test, said in a statement to The Mag that "ImPACT has become popular because it has been extensively researched," noting that it appears in more than 110 publications. "Concussion is increasingly recognized as a very complicated and complex injury that is best dealt with using multiple modalities. ImPACT is not designed to be used 'in and of itself,' but rather as part of an overall strategy that includes a clinical evaluation by an expert, a vestibular evaluation (including visual processing and balance) and neurocognitive assessment (ImPACT)."

Yet a study -- really a study of studies -- published last year in Current Sports Medicine Reports reviewed the entire span of research on ImPACT and concluded: "[T]he false positive rate appears to be 30 percent to 40 percent of subjects of ImPACT … the false negative rate may be comparable. … The use of baseline neuropsychological testing … is not likely to diminish risk, and to the extent that there is a risk associated with 'premature' return-to-play … may even increase that risk."

Lester Mayers, who was once a captain in the Army Medical Corps and now describes himself as an "elderly clinician," retired from private practice almost 14 years ago to become director of sports medicine at Pace University. And while he had previously specialized in internal medicine and pulmonary diseases, Mayers quickly realized he would have to focus on brain injuries at Pace -- a Division II school with campuses in New York City and Westchester County -- where athletes were sustaining concussions at an alarming rate. "I've seen more than 100 concussions since I've been at Pace, and there's a mystery to them," he says. "It's frightening when your child or teammate gets a concussion, and you see an athlete in never-never land."

So Mayers hit the books, scouring research journals to make sure his program was using the latest and best practices to manage brain injuries in athletes. He knew that Lovell, a neuropsychologist, and neurosurgeon Joseph Maroon had originally developed ImPACT in the early 1990s, and that together with Michael Collins, currently the director of the Sports Concussion Program at the University of Pittsburgh Medical Center, they launched a business to make the test commercially available. But Mayers became puzzled, then disturbed, by what he found when he started to dig deeper.

"Neuropsychological testing seemed to be becoming the fallback for the people involved in football to say, 'We're doing something about concussions,'" he says. "And through skillful marketing, ImPACT was giving the public the sense that if the pros use it, it's got to be right. But I went on PubMed once a week for more than a dozen years, and I kept finding papers about ImPACT published by the same people -- the people who run the company."

As Mayers found, the vast majority of the studies evaluating ImPACT have been written by the very researchers who developed it. On the "Reliability & Validity" section of ImPACT's website, for example, 21 of the 22 research papers listed are authored or co-authored by ImPACT's inventors.

Elite sports neuropsychology is a small, densely interconnected world. Many scientists wear multiple hats as they conduct research, work with teams and invest in or partner with businesses related to their work -- all different, potentially conflicting ways of making money. And in the case of ImPACT, the people who created the test have used their various platforms to popularize it, all the while maintaining a financial interest in it.

While developing and promoting ImPACT, for example, Lovell oversaw neuropsychological testing for the NFL. That meant he was directing the NFL's testing at the same time he was chairman of a company selling tests to the league's teams. Lovell also sat on the league's concussions committee and served as director of UPMC's Sports Concussion Program, as director of the NHL's neuropsychology program and as a consultant to the Steelers. (Today, he works full-time as CEO of ImPACT Application Inc. but remains a consultant to the NFL, NHL and several other organizations.) Maroon was also on the NFL's concussions committee and still serves as an adviser to the league and the Steelers' team neurosurgeon.

These overlapping roles have sometimes led ImPACT's executives into dubious, industry-funded research. Lovell is a co-author of the notorious 2004 paper in which the NFL's concussions committee found there was "no evidence of worsening injury or chronic cumulative effects" from multiple concussions in NFL players. And Collins, Lovell and Maroon cowrote a 2006 paper that found the Riddell Revolution helmet reduced the relative risk of concussions in high school football players by 31 percent. Riddell has trumpeted that research ever since. But the helmet maker had given grant money to UPMC, its vice president of research and development cowrote the paper, and reviewers blasted the work, using phrases such as "serious, if not fatal, methodological flaw" and "substantial conflict of interest."

Moreover, Lovell and other scientists affiliated with ImPACT have often failed to identify their potential conflicts of interest when publishing research. In 2007, an ESPN.com investigation found that "on at least seven occasions since 2003, Lovell has authored or co-authored studies on neuropsychological testing, including papers directly evaluating ImPACT, without disclosing his roles in creating and marketing ImPACT." In one case, the journal Brain Injury strengthened its conflict-of-interest policy after getting a complaint about Lovell's work. In another, Maroon and Collins reviewed a paper Lovell wrote for the journal Neurosurgery without ever disclosing their roles as fellow corporate officers at ImPACT Applications.

"It is a major conflict of interest," says Christopher Randolph, professor of neurology at Loyola University Medical Center near Chicago and former team neuropsychologist for the Bears. "The people looking at this test tend to be the same few who are invested in it. What if they're offering something that turns out to not be important? What if their research results are incompatible with their perception of their product?"

Mayers wondered the same thing. He watched ImPACT's inventors publish paper after paper affirming the test's accuracy, countered occasionally by detractors. And at first, as an outsider in the twilight of his career, he had no stake in the fight. But he had trouble finding prospective, controlled studies of ImPACT's, well, impact. Even today, no research project has taken a set of concussed athletes and a tightly matched control group -- healthy athletes with a similar mix of ages, genders and playing history -- and used the current version of ImPACT over a long period of time to see what the test picks up among each set. Instead, researchers have largely focused on how subjects with very recent concussions perform on ImPACT compared to their baselines. That's much less valuable information because if athletes are still suffering from concussion symptoms, teams don't need computerized analysis to hold them out.

For any test like ImPACT to be useful, it has to detect problems with brain function when they actually occur. But research shows the test has a disturbingly high rate of false positives. For example, in a 2007 study in the Journal of Athletic Training, researchers conducted baseline tests on a group of college students, all of whom had been concussion-free for at least six months. When these injury-free subjects took follow-up tests 45 days later, ImPACT incorrectly identified them as having some element of a concussion 38 percent of the time. In research published a year earlier, a team including Lovell and Collins compared ImPACT tests on 122 high school and college athletes who had concussions with 70 who did not. They found that two days after injuries, 64 percent of the concussed athletes reported worse symptoms, whereas 93 percent had either symptoms or abnormal test results, so ImPACT seemed to improve concussion detection by 29 percentage points. (And that's what the authors of the study wrote.) But 30 percent of the control group also registered problems two days later. When you know in advance that members of a particular group don't actually have concussions, you can dismiss their test results. But in real life, it's hard to trust any test that registers something incorrectly one-third of the time. "They tried to show ImPACT tests add value beyond what you get from evaluating symptoms, but it's a mess," says Barr.

To be constructive, a test also needs to generate consistent results for the same person from one examination to another. Scientists typically measure this using the statistical concept called correlation, which examines the relationship between two variables and ranges from minus-1 (inverse) to 0 (none at all) to 1 (perfect). And to call a test reliable, they generally like to see a test-to-retest correlation of 0.6 or higher. ImPACT's various components have correlation coefficients of 0.15 to 0.86, according to three studies over the past 10 years of the test's reliability. "If you score 95 on a reliable test, a week from now you'll score close to 95," says Steven Broglio, director of the Neurotrauma Research Laboratory at the University of Michigan. "But the correlations for computerized neuropsychological tests range from moderate to poor."

The test has its defenders. "ImPACT serves a positive purpose in diagnosing and managing concussions," Tracey Covassin, associate professor of athletic training at Michigan State University, told the Magazine. "I think if these tools can help, then … you need to use what you can to help better manage a concussed athlete. As I tell administrators at high schools, 'Is $500 worth the life of an athlete, is your child worth having the best care possible, which includes a multi-faceted approach to concussion management?'"

But in practice, it's hard for any neuropsychological test to get good data. Some athletes intentionally try to perform poorly on baselines so their post-injury tests won't keep them out of play. Peyton Manning admitted to this practice, which players call sandbagging, in April 2011. Additionally, athletes often take their baselines in groups (which research has shown leads to lower scores) but take their follow-ups as individuals. And studies have found wide differences in the reliability of tests depending on how long it takes to retest an athlete after injury. All of these factors make diagnosing brain injury far more complicated than the simple pass-fail that many athletes and families imagine when they first learn about baseline testing. "We'd love it if there were a test that could simply and accurately show whether an athlete's brain is functioning abnormally even though the individual is symptom-free," says Barr. "We're not there yet."

These were the facts Mayers examined over the past few years as he tried to get a grip on concussions at Pace. He recognizes, but does not agree, with the international protocols that say computerized testing is an important component in determining when an athlete should return to play. He has come to believe that athletes with concussions have needed more time off than they were typically getting. He sees no point in testing and retesting them while they still have symptoms, and he sees no way to disentangle ImPACT's usefulness from its hard sell.

ImPACT takes a broad view of the potential audience it can squeeze for revenues. Its website declares that physicians, psychologists, neuropsychologists, nurses, athletic trainers and athletic directors can administer ImPACT tests. A UPMC presentation also insisted that ImPACT tests will give reliable readings for injured athletes even if their baseline data isn't available, because ImPACT can provide stats from the general population for comparison. ImPACT charges medical practices and clinics $10 per baseline and $20 per post-injury evaluation, while schools and teams can buy in bulk -- 300 baselines and 90 post-injury tests, say, for $500. ImPACT also stages webinars and workshops costing up to $75 a pop to train test givers. And for as much as $1,500 a year, these professionals can become a "Credentialed ImPACT Consultant," affording them not only "better quality of care for your patients," according to the company's website, but also, yes, the chance to "Become a LEADER in Sports Concussion Management!!" With the right bona fides, the company says, come "increased media coverage" and "significant revenue generating opportunities." This means that doctors or other practitioners can profit from ImPACT by turning around and billing patients and insurance companies even more for the tests.

When one considers that 3 million ImPACT tests have been administered at the middle school and high school levels alone, according to a press release promoting ImPACT's involvement with Dick's, it becomes clear that ImPACT isn't just a marketing machine, but a cash-flow juggernaut. If each of the company's partners has just one credentialed consultant, ImPACT would take in up to $10 million a year, not counting sales of its tests. And each of its revenue streams is like an annuity, generating money year after year without ever requiring more work. That's why ImPACT often gives away the first year of testing for free; once a school or a doctor has baseline data for a group of athletes, they are sure to keep paying so they can evaluate injuries and bring new players into the system. So while the relationships Lovell, Maroon and Collins had with NFL teams gave ImPACT a tremendous head start in the testing industry, its business model has accelerated its success.

Even the government is on board. Since 2000, ImPACT has received a total of $225,711 in contracts from the Department of Defense. And from 2002 to 2005, Lovell led research that received a total of more than $2.2 million in grants from the National Institutes of Health to study concussions in sports.

As ImPACT grew, eventually it arrived on Mayers' campus. The Pace athletic department told its trainers to start using ImPACT testing at the school. "The school felt the NCAA was probably going to implement this in a year or two, so we might as well get going," Mayers recalls. But Mayers believed ImPACT was providing a false sense of security, rather than accurate testing. So he assembled all the evidence he could gather and with Thomas Redick, a professor of psychology at Indiana University-Purdue University Columbus, he reviewed the scientific literature on neuropsychological testing. Last March, Mayers and Redick published a research paper in the Journal of Clinical and Experimental Neuropsychology that flatly stated: "We conclude that the empirical evidence does not support the use of ImPACT testing for determining the time of postconcussion return to play."

"We found that ImPACT is just not fully reliable," Mayers says. Now under new leadership, the Pace athletic department no longer uses ImPACT testing. But Mayers' message has not reached far beyond his university. A 2002 paper he wrote on the prevalence of body piercing and tattoos among undergrads has garnered far more notice and citations in research journals than his work on concussions. The headlong rush toward more testing continues, as 41 percent of high school athletes with concussions got computerized neuropsychological tests in 2009-10, according to a paper published last year in the American Journal of Sports Medicine. And since the beginning of 2011, 28 states have enacted concussion legislation; most of the laws mandate that a healthcare professional trained in concussion evaluation clear all youth athletes before they return to play. That will surely lead to even more testing. Rhode Island's law even uses "impact" as a generic term for all baseline testing.

Of course, there are good intentions behind those mandates and behind much of the testing going on around the nation. It's easy to feel that doing something to fight concussions must be better than doing nothing. But American football is in the midst of replacing one giant, uncontrolled experiment (letting young men play a violent game) with another -- diagnosing them en masse and on the cheap with a test that many experts deem unreliable.

The bottom line is this: Neuropsychological testing in general, and ImPACT in particular, can be part of an overall exam. Any athlete suspected of having a concussion needs to see a healthcare professional trained extensively to deal with brain injury and not just trained to administer a test. Those professionals should examine athletes' symptoms, balance and medical history along with his or her cognitive function and should have the final say in return-to-play decisions in the interests of athletes' long-term health. "The confluence of symptom assessment, balance assessment, physical assessment, neurocognitive assessment and clinical interview is the 'best practice' approach," says Philip Schatz, professor of psychology at St. Joseph's University.

That might seem unrealistic or expensive. But when it comes to concussions, as Mayers puts it, "There are no simple answers."