On the stability of instability: Optimising the reliability of intra-subject variability of reaction times

While reaction times have been traditionally aggregated using measures of central tendency, interest in higher moments of the reaction time distribution, particularly intra-subject variability (ISV), has grown in recent years. However it is unclear to what extent individual differences in these higher moments are stable across time, reflecting trait-like features. The present study compares the reliability of a number of metrics for higher moments of the reaction time distribution on a battery of speeded tasks, and supplements these analyses with a Monte Carlo analysis of the go no-go task. The reliability of ISV is shown to be dependent on both the metric used and the number of trials used to calculate them. However, when using sufficient trials and appropriate metrics, ISV shows good test-retest and odd-even reliability. This study has important practical implications for the design and analysis of studies into ISV, as well as theoretical importance for the trait concept of ISV.