Random without replacement is not random: Caveat emptor

The vast majority of psychology labs rely on prepackaged software applications (e.g., E-Prime) for the programming of experiments. These programs are often used for stimulus selection, and many use a selection method referred to as random without replacement. We demonstrate how random without replacement deviates from random selection, and we detail selection biases that result. We also demonstrate, in a simple experiment, how these selection biases, if left unchecked, can influence behavior. Recommendations for reducing the impact of these biases on performance when random without replacement is used are discussed.