Interview with Dr. Thomas K. Pedigo, ED.D., NCSP., MSCP., ABMP
Dr. Pedigo is a leading innovator in the field of internet based attention and memory testing with over 20 years of experience in psychology and psychometrics as a clinician, author, speaker and published test developer. He has applied this broad background into developing scientifically validated, internet based brain training and testing applications in the areas of executive functioning, attention and memory testing.
Users of Dr. Pedigo’s Attention and Memory Tests application (called BrainCheck) can compare their performance scores on various tasks of attention and working memory to that of a large national sample of over 6,000 people.
Attention, Memory and Executive Functioning Tests
What is cognitive testing? How does it differ from Brain Training and what is the value to individuals, organizations and society?
The term "cognition" describes the mental processes that allow us to perform day-to-day functions, such as the ability to pay attention, to remember and to solve problems. Cognitive tests are used to measure a person's cognition. Other terms for cognitive tests include neuropsychological tests, psychometric tests, psychological tests, intelligence tests or neurocognitive tests.
Cognitive tests can measure any aspect of human mental performance that can be recorded in numerical form. Specifically of interest here are attention/concentration, short-term memory, working memory, response inhibition and speed of processing. These areas are used every day and often vary among individuals as a function of ability and objective success.
It is important to note that in stark contrast to the development of “Brain Training Games”, there are universal standards for the development of cognitive tests. The American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME) have collaborated on the joint development of tests since 1966. These guidelines are known as the Standards for Educational and Psychological Testing. There have been four revisions of these joint standards since they were first issued. The most recent edition of the Standards was published in 1999 and is currently being revised with an expected publication date of late 2012 or early 2013.
Cognitive tests are meant to be helpful by providing a picture or map of an individual’s important skills in key areas of functioning. This can inform decisions to determine if interventions are needed and/or working? Should they be modified or discontinued? Should we hire this potential employee? Or has a decline in functioning occurred? From the above discussion it can be easy to understand that Cognitive testing, and Cognitive tools in general, while designed to be helpful, can also be harmful if not developed properly. Standards at minimum require appropriate selection of test items i.e. do the items make sense? and research evidence to support their proposed use.
Tests must show repeatability or reliability in several key areas. Validity must be established to show the tests measure what they state they measure. Variability in scores must be investigated with standard error or the amount of change you can expect in an individual score with repeated administration and standard deviations or the amount of change you can expect from an individual’s score from the normative samples average score.
Cognitive Tests will have large national samples from all geographic locations and population demographics. For example, the BrainCheckTests normative sample includes over 6,000 subjects from all 50 states including all demographics and has been subjected to numerous studies of reliability, validity and variability. This way an individual taking the measures can be sure they are comparing to the national standard. Frequently, Brain Training companies will report the science behind the brain’s potential for change and then infer that if you improve on their games you are getting better. They will even tell you they have thousands of others with whom you are comparing. Research has shown that comparison to others playing the same non standardized games does not show meaningful improvement when subjected to external standards like cognitive testing and comparisons to control studies. This limitation in proving that Brain Games work outside of the game is a result of not studying the training games as if they were tests and from not subjecting them to the standards needed to prove they work. Basically the process for Brain Training is something like: ”We know the brain can improve, hey these activities look like they should work, and Wow people do get better at them, so we must be Brain Training”! This is the current state of Brain Training.
As a Medical Psychologist, I have seen cognitive rehabilitation work for patients with brain injury and attention/memory weaknesses. However, the rehab process is often done with tools that are exacting, boring and require much effort with consistent application. These tools create demands like those found at school or work and do not inherently reward the Brain for participating as found in most Brain Games. Brain Training Games are designed to be fun, non-stressful, and approached casually. However, school and work settings do not normally evoke the fun word and are inherently stressful and routinely demanding.
Finally, training requires life long effort, just like physical training and dietary improvements. If one stops the healthy behavior, they will revert back to the old self. Proper Cognitive Conditioning can create gains and Cognitive Testing can help define your skills and abilities to a highly standardized national sample. Cognitive testing can give you a mental map of your skills and help design and track improvement that translates to work and school settings. The benefits of using Cognitive Tests come from being able to use reliable and valid information to determine: How am I doing? Do I need help? Am I getting the help I need? Should I do something else? In the end, standardized cognitive testing will allow you to decide if your efforts are paying off and to provide detailed information that can help you decide how to train, and to evaluate if the games or activities have the challenge needed to promote change. A more thorough review of the current research with Brain training can be found on the Brainchecktests.com website.
What was your inspiration for developing internet based attention and memory testing tools?
Over the course of my career, I have completed or supervised over 10,000 cognitive assessments. I know the value of this information in decision making. However, cognitive testing with a licensed psychologist requires looking for a diagnosis, is very expensive, is time consuming, and in some areas of the country hard to obtain. Most individuals do not need this level of information as a routine function. With the changes in technology and the advent of awareness for Brain Health our test development section (Targeted Testing, Inc.) set out to see if we could design a web based service that could allow individuals to reliably and affordably self-screen important cognitive skills via an internet based delivery system. The results from the research trials for the BrainCheck Test batteries proved this could be done with over 6,000 participants from all 50 states and relevant demographics. Now people can perform a reliable and valid self- screening from the privacy of their own home for about the price of having a pizza delivered.
What does Brain Check measure?
The test batteries each consist of 13 tests covering Executive Skills: Working Memory (Holding and manipulating information in mind), Activating/Inhibiting (Starting and stopping when needed), Shifting/Flexibility (Changing rules and focus between tasks or jobs), Planning/Organizing (Deciding on solutions and following through), Metacognition (Using one’s internal speech/thoughts to guide action), Emotional Control and Sustained Effort (Suppressing emotions and negative feelings while enduring). Additionally, the tasks sample short term visual memory, visual motor coordination, cognitive flexibility, visual discrimination, speed of mental operation and the ability to stop responding when needed.
What is a meaningful sample size for these types of tests?
Typically, cognitive tests made by large test publishers like Pearson and Psychological Assessment Resources will seek normative samples from 1,000-2,000 subjects and will attempt to stratify the sample to match the most recent U.S. census. The BrainCheck Test Batteries include over 6,000 subjects from all 50 states and relevant demographics. A review of the research and development shows we have equaled or exceeded numbers of subjects and have stratification equal to those established by the large test companies.
What are the best foods to eat for ideal mental performance and testing?
For individuals interested in Brain health and function, start with a diet that reduces or eliminates saturated fats and Trans fats and replaces them with grains, legumes, nuts, fruits, vegetables, lean meat and lots of fish is essential. Proper exercise, fun, hobbies and creativity all are neuro-protective. I would suggest staying away from too many supplements unless a physician you trust is recommending them and it makes sense to you.
How can someone with ADD or ADHD improve their mental and/or emotional condition using Brain Check or some other brain training and testing application?
Anyone that thinks they have weak attention and concentration and clear impairment in their work or school performances should get a comprehensive evaluation by a licensed clinical psychologist with specialty training and experience with attention disorders. State Psychological Association referral services and physician referrals are best to ensure you get to someone with the proper training. However, concerns are frequently not that clear as in the case outlined above. In this instance self-testing can be a valuable tool to see if one should embark on formal testing that is time consuming and expensive.
If after self-testing it becomes evident significant weaknesses exist in several key areas as is likely with ADHD, then formal testing is indeed warranted. However, there will be a meaningful number of individuals that have low average ability and maybe a single area of significant weakness, and for these individuals self-testing and tracking to follow progress can be effective and efficient.
There are numerous approaches for personal improvement from personal training, diet and exercise to Brain Training with the right tools, or increasing the use of Neurofeedback. All of the above can promote improvement. Each of these approaches has some evidence and will be reviewed on the BrainCheck Test website. Additionally, a national referral service is in the works.
What special groups of people and organizations have unique needs for memory, attention, executive functioning and other brain testing and training? How these needs can be satisfied using your technology?
Screening and follow up tracking can be helpful for a wide variety of individuals: As mentioned above those concerned for their attention and memory skills, those completing specific training programs to promote better attention and memory, those involved in age management treatment and those individuals being treated with medications for attention/concentration, depression/anxiety, cognitive decline or substance abuse can benefit from ongoing assessment. Having a cost and time effective solution to self-assessment can be a major improvement to the efforts of those mentioned above tracking their progress.
Is there any research into whether the types of testing and training you offer can reduce the risk of Alzheimer’s and other related diseases?
Prevention and monitoring are key areas to help stave off the effects of brain degeneration. Early identification and routine tracking go hand in hand. Early identification means treatment starts sooner and routine assessment means improvements can be validated or changes can be made sooner. Sooner is always better than later. Often physicians are overworked and have limited time constraints and resources for frequent testing. A personal assessment site can repeatedly monitor progress and needs over time, in a manner that is not allowed by most insurance companies.
How can people who regularly use these types of brain testing and training batteries experience both a fresh and validated experience? Specifically, I assume your application and other related testing and training applications on the web have a limited number of tests, games and training routines. That said if someone wants to use these applications regularly for the long run, won't they be repeating to the extent that there is reduced interest and with that reduced scoring due to less motivation? Or are these programs such that you don't encounter the same or very material/sequences to the extend one would feel they have 'been there, done that'? Is there a way to get both rigorous measurements with large samples AND be fresh for the people who regularly use the apps?
Specifically, from a psychometric standpoint you are referring to practice effects. If subjects were to learn the answers to given test questions the test would quickly become a measure of achievement but not good for predictive ability. This is one of the essential concerns for standardized tests that they must demonstrate acceptable reliability while not becoming perfect. Similar arguments from the scientific community about Brain Training suggest the specificity of the games which will produce large practice effects and thus inflate actual improvement. To address this we have proposed that individuals completing such training embark on external assessment that is different from the games.
Sound psychometric tests will have demonstrated excellent Test-Retest stability or a strong tendency to accurately measure performance over time. Our measures have shown this at two weeks and at over 6 months. Additionally , we have multiple versions and recommend placing at least two weeks between a given version. Finally, because we use nondescript colors and shapes their is less cultural carryover or unique stimulus to remember. This approach makes over learning quite difficult.
Game training uses novelty and reward to encourage use and to aid in the overall experience which will keep folks thinking about the activity. Once you complete a Battery of well designed psychometric tests you usually are just glad it's over and won't have a keen sense of how you did without feedback. This dynamic unique to well standardized performance tests is best seen in professional license exams where it's just to much to keep up to learn the test. This is in stark contrast to Computer Games built to be fun and engaging with many identifiers or stimuli to pull from.
Humans are inherently driven to control and master their environment. Self- improvement is just another extension of this biological drive. Anyone that has spent time in a weight lifting gym can attest to observing well-meaning individuals breaking their back with the wrong approach or technique. These individuals get less progress and increased risk of injury. The same is true for Brain Training, Peak Performance Training and the booming supplement market. However, most have scant scientific evidence for their approach or supplement. What these forces promote is usually the science that suggests certain physical properties or skills may be amenable for improvement.
The Brain Training Boom is a prime example. This industry for the most part suggests extraordinary change is possible by playing a few games a few times per week in an enjoyable format. Behind this claim is the scientific understanding that nerve cells can adapt and strengthen with repeated pressure. Unfortunately controlled clinical studies show what common sense dictates, that simple effort will not create substantial improvement. Research shows it is possible to attain meaningful change with cognition if the right tools and work ethic are applied. Likewise, it is much harder than is often promoted through marketing and will need to be lifelong. Finally, the same approach will not work for everyone and some will have more potential for change than others.
External assessment can be a valuable adjunct to show what real effort feels like. If your Brain Training or other program does not push you like the assessment tests do, change them in favor of more effort and challenge. In this manner external assessment can track progress and show what is and is not working.
Beware of programs that make it seem effortless as there is no replacement for hard work. Solid external assessment will have proper norms, be developed through nationals standards, have acceptable psychometric evidence, and will not be a part of the actual training program. Only through external assessment can an individual be sure their time and money are well used.
