Performance variability has been linked with cognitive decline (e.g., Hultsch et al., 2000, 2008; Williams et al., 2005; Haynes et al., 2018). However, given the normal reductions in cognitive performance that occur with aging, it is unknown how much different aspects of cognitive performance should vary within an individual, and to what degree impairments are pathological. Cognitive performance on speeded, reaction-time-based tasks can be captured in three ways: first, in terms of accuracy for measures that include correct/incorrect response options; second, in terms of mean or median reaction time (RT); and third, in terms of RT variability (e.g., West et al., 2002; Bunce et al., 2004; Deary and Der, 2005; Williams et al., 2005; Der and Deary, 2006; Gooch et al., 2009). Although the vast majority of the literature in neuropsychiatry focuses on accuracy or mean RT, here we compare measures of accuracy and mean RT to measures of RT variability to see to what degree age-related differences in each of these metrics show distinct or convergent patterns.

Variability in cognition may be a particularly important metric, in that individuals with lower baseline performance levels may be able to normally compensate across cognitive domains, whereas when their performance becomes less stable (more variable), such compensatory processes may fail (e.g., MacDonald et al., 2003; Bielak et al., 2010). For decades, researchers have posited that RT variability is an important component of intelligence (Jensen, 1992, 1993; Schmiedek et al., 2007, 2009, 2013). RT variability has been found to be sensitive to the cognitive changes that occur in psychiatric and neurological disorders, normal development, and aging (MacDonald et al., 2006, 2009; Esterman et al., 2013).

Lower RT variability (more consistent performance) has been associated with better cognitive control (Vasquez et al., 2017) and higher RT variability (less consistent performance) has been associated with cognitive instability (Fjell et al., 2011), greater mental noise (Ode et al., 2011), and poorer cognitive control (Strauss et al., 2007; Papenberg et al., 2011; Vaughan et al., 2013). Previous studies have found that RT variability follows a u-shaped curve across development and aging (see Williams et al., 2005), with variability being highest in childhood and older adulthood. While most studies on RT variability within an individual use cross-sectional data, longitudinal studies of aging show that RT variability continues to increase linearly from early adulthood into late adulthood (MacDonald et al., 2003; Lövdén et al., 2007). Typically, variability within an individual is greater when the task requires response selection or cognitive control, as opposed to tasks that require minimal cognitive control, such as a simple RT task (Bielak et al., 2010).

There have been inconsistencies in the literature, however, in terms of how RT variability differs across the lifespan. Different measures of RT variability can produce different results (McAuley et al., 2006; Dykiert et al., 2012). One of the sources of divergence in results is the degree to which measures of RT variable are adjusted for differences in mean RT. Since RT variability (computed in terms of the standard deviation in RTs) tends to increase with mean RT, differences in RT variability could reflect differences in mean RT. In a systematic review and meta-analysis, Dykiert et al. (2012) found that effect sizes were larger for variability measures that did not adjust for differences in mean RT than those that did, with some studies that used mean-adjusted RT variability metrics showing little to no increase in RT variability in older age. Here, we focus on variability in RTs within a task, rather than variability across trials within the same session (dispersion) or variability in scores at different timepoints (e.g., longitudinal variability, measurement burst designs, or ecological momentary assessment). While the latter forms of variability are potentially important and informative (e.g., Bielak et al., 2010; Stawski et al., 2019), they are related to fluctuations along longer time scales (hours, days, years) rather than the moment-to-moment variability we focus on for the purposes of this study. Future work might clarify whether within task RT variability that we examined here produces similar findings to measures of variability examined along longer time scales for choice and simple RT data (e.g., in ecological momentary assessment designs).

Here, we sought to compare and contrast different measures of cognitive performance and variability in a large, diverse sample to better characterize patterns of age-related change, establish updated norms, and compare across standard performance metrics in a large, well-powered sample. Specifically, we wanted to better understand the potential effect of aging on mean RT, RT variability (raw and mean-adjusted), and accuracy in measures of simple and choice RT. Our large sample size allowed us to look at differences in performance year-by-year across the lifespan, addressing sample size limitations of prior studies, and allowing us to estimate potential trajectories of cognitive change for different indices. We also compared different approaches to capturing RT variability to see whether they produced similar or discrepant results (Hultsch et al., 2002; Stuss et al., 2003; Dykiert et al., 2012).

We hypothesized that older age would be associated with increased mean RT and RT variability, for both raw RT variability metrics and mean-adjusted RT variability metrics, consistent with the literature described above. Like Fortenbaugh et al. (2015), who used a similar type of sample, we expected to find a non-linear relationship between age and variability in RT, distinct from the relationship of mean RT with age. We were particularly interested in the age and slope of increases in RT variability in normal aging. Our study expands on the prior literature, as it is the largest evaluation of RT variability in measures of choice (Choice RT) and simple RT (Simple RT) across the lifespan to date.

This is the largest study to compare and contrast different indices of performance and performance variability in simple RT and choice RT tasks. Different measures of performance capture different characteristics of human behavior, and here we found that different measures, including measures that putatively measure the same constructs, exhibited different patterns of age-related performance. For example, we found that the peak age for cognitive performance based on median RT was 17, whereas based on coefficient of variability scores for choice RT, we found highest performance at age 36 years. Interestingly, different measures of RT variability also showed distinct patterns across age. While measures of raw variability (SD RT) and coefficient of variability (SD RT/mean RT) showed similar differences with age, for choice RT mean residualized variability (SD RT controlling for mean RT) showed a distinct relationship with age – with apparent decreases in variability across the lifespan, contrary to the literature. This contrasted with simple RT where mean-residualized variability produced similar (but attenuated) age effects when compared with other variability measures.

Our large sample size allowed us to finely characterize variations in RT and RT variability across the lifespan. Such results can provide normative models for cognitive performance on such tasks across the lifespan, which could provide a basis for revealing abnormal trajectories (Fortenbaugh et al., 2015). Here, we were able to distinguish between different measures of cognitive performance and variability across the lifespan, demonstrating that different measures capture different aspects of cognitive performance with respect to age. For instance, our data clearly demonstrated the classic speed accuracy trade-offs that emerge across the lifespan (Bruyer and Brysbaert, 2011; Heitz, 2014), with increasing median RT (even after adjusting for basic psychomotor speed) as well as increasing accuracy on our choice RT. IES, which adjusts for speed accuracy trade-offs, showed minimal change during early adulthood, with poorer performance not emerging until age 28 years. Finally, RT variability seemed to increase around middle age for raw variability, and mean adjusted variability (coefficient of variability), although these effects were eliminated and reversed after controlling for mean RT.

Our findings of different ages of peak performance based on mean RT and RT variability suggest that different measures of variability might yield different information about lifespan-related processes. What is most striking about our data here was how remarkably consistent the linear changes in residualized variability were across the lifespan for choice RT, with an almost entirely linear decrease across age that could not be explained by differences in accuracy. We are not certain why the unusual residualized SD RT result appears only in the choice RT task but not simple RT; however, we suspect that it is due to the fact that responses in the choice RT reflect a more complex cognitive process than simple RT that contributes to increases in the mean with age but not SD. Although mean residualized standard deviations have become the primary method of quantifying RT variability, it may be that adjusting for the mean can sometimes obscure true differences in variability. If mean RTs increase with older age due to multiple additive processes, not all of which are associated with changes in variability of RTs, then removing variance associated with the mean could result in an overcorrection that might explain the results observed here. For example, a tendency to respond less impulsively with age would tend to offset greater variability in RTs in a way that is due to shifts in strategy that increase response time without concomitant increases in RT variability. The fact that our finding contrasts sharply with the literature may be due to task differences, file drawer effects (due to violation of a priori expectations), or unmeasured confounds. Our findings of reduced mean residualized variability with age should therefore be interpreted with caution until it has been replicated in another sample.

We found a small effect for differences in variability (based on raw variability and coefficient of variability) between men and women, with men demonstrating slightly more consistent performance. We cannot assume that the differences are due to sex differences in processing speed alone [men are typically faster than women, see Roivainen (2011) for a review], as variability within an individual takes average processing speed into account. Some researchers have suggested that differences in cognitive performance processes are related to differences in symptom trajectories between men and women, namely the idea that women are more vulnerable to depression while men are more vulnerable to impulse control disorders (Li et al., 2009) that may be related to differences in variability and speed–accuracy trade-offs.

There are several limitations of the current study. First, our cross-sectional design unfortunately prohibits us from making any strong conclusions related to individual lifespan trajectories as findings could be due to cohort effects or ascertainment biases that vary by age. Interpretation of correlations (slopes in segmented regression) is problematic with such unequal age bins, and thus, we emphasize patterns of change in terms of changes in slope (breakpoints) rather than correlations as point estimates. Second, given that our sample was self-selected, it is possible that our results are biased toward higher functioning older adults with more expertise using computers. The expectation is that this would cause us to underestimate performance decrements with age and potentially overestimate any performance improvements (e.g., increased accuracy). Additionally, selection biases in older adults due to mental and physical fitness has been well-documented (Ganguli et al., 1998; Golomb et al., 2012). On the other hand, our self-selected sample allows for a larger outreach to individuals from communities that are typically harder to reach including working adults, people in rural areas, or those with limited mobility or resources to participate in research studies. Third, while prior TestMyBrain.org sample studies have been replicated compared to traditionally collected and nationally representative US samples (Germine et al., 2011; Hartshorne and Germine, 2015), further validation work is recommended. Finally, our very brief measures of simple and choice RT were sufficient to produce reliable measures of psychomotor response speed, but did not allow us to employ more sophisticated model fitting techniques to trial-by-trial data that have been used in other studies of RT variability, such as estimation of multiple components of variability using an ex-Gaussian distribution (see McAuley et al., 2006; Matzke and Wagenmakers, 2009) or estimation of variability over longer time intervals, such as in a measurement burst design (Stawski et al., 2019).

Despite these limitations, our study provides a potential foundation for future research on lifespan performance and performance variability, how best to conceptualize variability, as well as a richer characterization of how performance metrics differ even in relatively simple task designs. Findings from web-based samples such as this one have been shown to match traditional findings from the literature, and are being used more often to recruit larger, more diverse samples (i.e., Nosek et al., 2002; Soto et al., 2011; Fortenbaugh et al., 2015; Hartshorne and Germine, 2015). Additionally, our pattern of mean RT across the lifespan, which peaks in the early 20s, replicates gold standard longitudinal work (e.g., Deary and Der, 2005), as we have previously shown that lifespan patterns of change in processing speed replicate work from these gold standard studies (Hartshorne and Germine, 2015). Given the utility and ease of web-based cognitive tests [see Koo and Vizer (2019) for a recent review], an approach that integrates multiple metrics may be useful for clinicians and researchers to study cognitive performance and performance variation at many stages across the lifespan.

The primary results reported in this manuscript have been presented at local and national conferences between 2018 and 2019. The data are deposited to the Open Science Framework, which can be accessed with the following link: https://osf.io/w5nge/.

All datasets generated for this study are included in the article/Supplementary Material. The data is available through OSF: https://osf.io/w5nge/.

The studies involving human participants were reviewed and approved by Harvard University.

LR and LG contributed to the conception and design of the study, organized the database, and performed the statistical analysis. LR wrote the first draft of the manuscript. All authors contributed to the manuscript revision, and read and approved the submitted version of the manuscript.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.