The Effect of Method Characteristics on Retest Score Gains and Criterion-Related Validity

Purpose

We sought to empirically assess the effect of predictor method characteristics (test form, item-type, and test-type) on retest score change associated with an invariant construct—general mental ability (GMA)—and to evaluate the effect of retesting on the criterion-related validity of assessments that vary in their susceptibility to retest effects.

Design

Three hundred seven individuals completed a battery of GMA assessments. After a 6-week interval, participants returned to the testing site to retest using both alternate and identical forms of the initial assessments.

Findings

Greater score gains were observed on assessments comprising heterogeneous item-types than homogeneous item-types, and on performance-based assessments than self-report assessments. However, despite variations in score gains, the relationships between the initial test scores and criterion scores were no different than the relationships between retest scores and criterion scores for all assessments.

Implications

Tests and procedures that reduce reliance on test- or item-specific knowledge and skill may help minimize score changes due to retesting across multiple administrations. Moreover, under the boundary conditions present in this study, the criterion-related validity of ability assessments may not be affected by increases in test-specific knowledge and skills.

Originality/Value

Despite the prevalence and industry support of retesting, a comprehensive understanding of retest score change still eludes researchers and practitioners. This ambiguity may be due in part to neglecting the method-construct distinctions in the retest literature. This is the first report to explicitly utilize the method-construct distinction in an effort to examine the causes and consequences of retest effects.

     

Instruction-driven processing in human perceptual learning

We conducted a series of experiments to determine the relative contribution of unsupervised versus controlled mechanisms to the intermixed-blocked effect. In Experiment 1, participants received pre-exposure instructions prompting a search for differences between stimuli, in keeping with past studies, and the intermixed-blocked effect was observed. In the remaining experiments, participants did not receive the aforementioned instructions, but instead were instructed either to simply observe the stimuli (Experiment 2) or in relation to a masking task (Experiment 3). None of the latter experiments produced an intermixed-blocked effect, suggesting that the effect found in Experiment 1 was driven by the instructions to search for differences, consistent with a controlled processing account of the effect. Moreover, we tested a prediction assuming the operation of a search strategy against one assuming the operation of a short-term habituation mechanism and found evidence more consistent with the search strategy hypothesis. We formulate a new account of the intermixed-blocked effect in humans based on an instruction-driven search and discuss how the account could explain many findings in the human literature.     

Optimal assertions,and what they implicate. A uniform game theoretic approach

To determine what the speaker in a cooperative dialog meant with his assertion, on top of what he explicitly said, it is crucial that we assume that the assertion he gave was optimal. In determining optimal assertions we assume that dialogs are embedded in decision problems (van Rooij 2003) and use backwards induction for calculating them (Benz 2006). In this paper, we show that in terms of our framework we can account for several types of implicatures in a uniform way, suggesting that there is no need for an independent linguistic theory of generalized implicatures. In the final section, we show how we can embed our theory in the framework of signaling games, and how it relates with other game theoretic analyses of implicatures.

Equivalence of paper and pencil vs Internet forms of the ZKPQ-50-CC in Spanish and French samples

This study was designed to check for the equivalence of the ZKPQ-50-CC (Spanish and French versions) through Internet on-line (OL) and paper and pencil (PP) answer format. Differences in means and deviations were significant in some scales, but effect sizes are minimal except for Sociability in the Spanish sample. Alpha reliabilities are also very similar in both versions with no significant differences between formats. A robust factorial structure was found for the two formats and the average congruency coefficients were 0.98. The goodness-of-fit indexes obtained by confirmatory factorial analysis are very similar to those obtained in the ZKPQ-50-CC validation study and they do not differ between the two formats. The multi-group analysis confirms the equivalence among the OL–PP formats in both countries. These results in general support the validity and reliability of the Internet as a method in investigations using the ZKPQ-50-CC.     

The sunk‐time effect: An exploration

We explored the potential for a sunk‐cost effect in the realm of time. Questionnaire studies (Experiments 1–4) obtained a sunk‐time effect that was robust to manipulations of prospective value, individual versus group consequences, and the effort or enjoyment inherent in the time. Behavioral experiments (Experiments 5–7) also suggested a sunk‐time effect and found support for a personal responsibility by sunk‐cost interaction on choice behavior. We discuss theoretical implications and a potential connection to animal sunk‐cost phenomena. Copyright © 2008 John Wiley & Sons, Ltd.     

The MCMI-III personality disorders scores predicted by the NEO-FFI-R and the ZKPQ-50-CC: a comparative study

This study was designed to compare the NEO-FFI-R versus the ZKPQ-50-CC in their relationships with the fourteen MCMI-III personality disorder scales in a Spanish non-clinical sample (N = 674). Previous studies showed consistent relationships between the Five Factor Model and the DSM-IV personality disorders (PD), but there is no comparative study between both Five Factor and Zuckerman's personality models. The aim was to replicate previous results about relationships between the Big-Five and PDs using the revised short version of the NEO-PI-R, and to compare the NEO-FFI-R versus the ZKPQ-50-CC regarding the relationships with MCMI-III personality disorder scales. Results showed no sharp differences between the NEO-FFI-R and ZKPQ-50-CC scales. Each instrument explained around 30% of the PDs MCMI-III scales variance. Using conjointly the 10 personality scales from the NEO-FFI-R and ZKPQ-50-CC, the PDs accounted variance rose to 38%. Differences and similarities between both short personality questionnaires are discussed.     

Slower eating rate reduces the food intake of men, but not women: implications for behavioral weight control

This study tested a central tenet of behavioral weight control: does slower eating rate reduce food intake and positively affect appetite ratings. This study used direct behavioral observation of participants' (n=48; 25BMI35) eating behavior using Universal Eating Monitors. Food intake and ratings of "desire to eat" (collected each minute during meals) were represented as a function of time. Based on an acclimation meal, participants' eating rate was modified by instructing them to eat a bite of food when prompted by a computer, which generated three types of meals: (1) baseline (eating rate was the same as the acclimation meal), (2) reduced-rate (eating rate was reduced by 50%), and (3) combined-rate (eating rate at the beginning of the meal was identical to that of the acclimation meal, subsequently eating rate was reduced by 50%). All meals were ad libitum. Slower eating rate during the reduced-rate and combined-rate meals resulted in less food intake compared to the baseline meal for men, but not women. Ratings of desire to eat, normalized for the amount of food consumed, were lower during the combined-rate meal for men and women. Thus, this basic premise of behavioral weight control was supported for men, but not for women.     

Ingroup and outgroup stereotypes and selective processing

The main concern of the two studies presented here is to investigate whether the different nature of ingroup and outgroup stereotypes is reflected in different selective processing of ingroup and outgroup information. It was predicted that when processing ingroup information people will preferentially encode stereotype-inconsistent information as compared to stereotype-consistent information, whereas the reverse pattern will hold when people process outgroup information. In addition to selective processing, response bias due to stereotyping was studied. To measure selective processing and response bias, recognition memory measures derived from the theory of signal detection were used. Results of the two studies confirmed our main prediction. Also, response bias was demonstrated. ©1997 John Wiley & Sons, Ltd.     

Screening for cognitive impairment in an Australian aged care assessment team as part of comprehensive geriatric assessment

Accurate detection of mild cognitive impairment (MCI) is important to stratify and address risk. Yet, few short cognitive screening instruments are validated for this. . In Australia, all clients referred to an Aged Care Assessment Team (ACAT) receive comprehensive geriatric assessment (CGA) including the Standardized Mini-Mental State Examination (SMMSE). We compared the accuracy of the quick mild cognitive impairment (Qmci) screen to the SMMSE in 283 participants: 195 with dementia, 47 with MCI, and 41 with subjective cognitive decline (SCD) in an Australian community-based ACAT. Both had similar accuracy in identifying dementia, AUC of 0.86 for the Qmci versus 0.93 for the SMMSE (p = 0.10), but the Qmci was more accurate than the SMMSE in differentiating MCI from SCD, AUC of 0.84 versus 0.71, respectively, p = 0.046. These suggest that the new, short (3–5 min) Qmci screenis appropriate for use in an ACAT or other units conducting CGA.