Meta-interpretive reliability of computer-based test interpretations: the Karson Clinical Report.

Meta-interpretive reliability is a new method to evaluate the accuracy with which personality trait scores are communicated via interpretive statements in a computer-based test interpretation (CBTI). The prototypic experimental design is based on a two-way repeated measures analysis of variance (ANOVA); the two effects are personality traits and randomly chosen CBTI protocols. In this application, 101 psychologists read four examples of the Karson Clinical Report (KCR, Karson & O'Dell, 1975) and estimated the original trait scores from the Sixteen Personality Factor Questionnaire (16PF; Cattell, Eber, & Tatsuoka, 1970) on which the KCR is based. Estimated trait score variance was significantly related to the Trait x Protocol interaction and the main effects for personality trait and differences among protocols (omega 2 = .55). The total effect size corresponded to a multiple correlation of .74, suggesting that the KCR had acceptable meta-interpretive reliability. The protocol effect denoted a context effect created by the juxtaposition of several interpretive statements. Additional analyses showed that individual differences among raters contributed to less than 1% of the estimated standard ten (sten) score variance. Meta-interpretive reliability is proposed as an index of the upper limit of validity for CBTIs.     

Relative user ratings of MMPI-2 computer-based test interpretations

There are eight commercially available computer-based test interpretations (CBTIs) for the Minnesota Multiphasic Personality Inventory-2 (MMPI-2), of which few have been empirically evaluated. Prospective users of these programs have little scientific data to guide choice of a program. This study compared ratings of these eight CBTIs. Test users were randomly assigned to rate either a single authentic CBTI report on one of their clients or a single CBTI report generated from a modal MMPI-2 profile for their clinical setting. In all, 257 authentic and modal CBTI reports were rated by 41 clinicians on 10 dimensions. Each of the authentic reports received substantially higher ratings than the modal reports, with ratings of perceived accuracy and opinion confirmation best differentiating between authentic and modal reports. Automated Assessment Associates' report received the highest overall ratings; reports published by Western Psychological Services, Pearson Assessments, and the Caldwell Report were also distinguished on one or more ratings dimensions.     

The Role of Computer-Based Test Interpretation (CBTI) in Occupational Assessment

The paper explores a range of issues relating to the design and use of computer-based test interpretation (CBTI) systems. It describes a number of dimensions along which CBTI systems vary and considers both the positive and negative implications of these for test users, for their clients and for test candidates. A distinction is drawn between systems that generate finished reports and those that generate questions or guidance for the user. The question of how the validity of narrative reports can be assessed is considered. A study on the discriminative validity of reports generated by the PREVUE ICES CBTI system is reported which shows clear differences between judgements made about true and bogus reports. It is concluded that there are two main issues to consider when evaluating a CBTI system: first, the validity of the information generated by it and, second, the ways in which that information is likely to be used.     

Assessment: what role the assessor?

The advent of automated assessment contains the seeds of both threat and promise for professional assessors. Placing the problem in the historical and sociological contexts, this article examines computer-based test interpretation (CBTI) and the frequently misunderstood role of actuarial data in relation to it. I argue that sound clinicianship is central not only to the development of adequate CBTI systems, but also to the systems' appropriate and skillful use in practice. When properly utilized by otherwise well-qualified clinicians, CBTI can substantially enhance practice.     

Assessment: What Role the Assessor?

The advent of automated assessment contains the seeds of both threat and promise for professional assessors. Placing the problem in the historical and sociological contexts, this article examines computer-based test interpretation (CBTI) and the frequently misunderstood role of actuarial data in relation to it. I argue that sound clinicianship is central not only to the development of adequate CBTI systems, but also to the systems' appropriate and skillful use in practice. When properly utilized by otherwise well-qualified clinicians, CBTI can substantially enhance practice.     

Brief Behavioral Treatment for Insomnia vs. Cognitive Behavioral Therapy for Insomnia: Results of a Randomized Noninferiority Clinical Trial Among Veterans

The goal of this study was to compare a brief behavioral treatment for insomnia (BBTI), which has fewer sessions (4), shorter duration (< 30–45 minutes), and delivers treatment in-person plus phone calls to cognitive behavioral therapy for insomnia (CBTI), which has 5 in-person sessions. The hypothesis was BBTI would be noninferior to CBTI. The Reliable Change Index was used to establish a noninferiority margin (NIM) of 3.43, representing the maximum allowable difference between groups on the pre-post Insomnia Severity Index change (ΔISI). Sixty-three veterans with chronic insomnia were randomized to either BBTI or CBTI and veterans in both groups had significant reductions of their insomnia severity per the ISI and improved their sleep onset latency, total wake time, sleep efficiency, and sleep quality per sleep diaries. While CBTI had a larger pre-post ΔISI, this was not significantly different than ΔISI BBTI and was less than the NIM. However, the 95% confidence interval of the between group pre-post ΔISI extended beyond the NIM, and thus BBTI was inconclusively noninferior to CBTI. Limitations, such as small sample size and high rate of dropout, indicate further study is needed to compare brief, alternative yet complementary behavioral insomnia interventions to CBTI. Still, evidence-based brief and flexible treatment options will help to further enhance access to care for veterans with chronic insomnia, especially in non-mental-health settings like primary care.     

Assessment of the Barnum Effect in Computer-Based Test Interpretations

A prototypic experiment for validating computer-based test interpretations (CBTIs) was conducted. Undergraduates (N = 63) completed the Comprehensive Personality Profile Compatibility Questionnaire (CPPCQ; Craft, 1987). One treatment group rated real CBTIs for relative accuracy, and another group rated bogus CBTIs. A significant main effect for differences in ratings indicated that the real CPPCQ profiles were rated as 74.5% accurate whereas the bogus CBTIs were rated as 57.9% accurate. Several covariate effects were tested, but none were significant.     

Brief Cognitive Behavioral Therapy for Insomnia in Parkinson's Disease: A Case Series Study1

Parkinson's disease (PD) patients frequently suffer from insomnia and insomnia can result in reduction of quality of life in PD. Although pharmacotherapy is most applicable for insomnia, it may cause side-effects in PD. The purpose of the study was to investigate the efficacy of brief cognitive behavioral therapy for insomnia (CBTI) in PD. A total of 11 PD patients aged 43–84 years with chronic insomnia received two sessions of CBTI. Patients reported a significant decrease in total score for the Insomnia Severity Index (ISI). The total score for the Parkinson's Disease Sleep Scale (PDSS) improved. Although objective sleep measured by actigraph did not improve, subjective sleep measured by sleep diary improved. Functional impairment measured by the Sheehan Disability Scale (SDS) significantly decreased. These results revealed that brief CBTI was effective in improving insomnia in PD, with improvements extending to functional impairments that had been affected by insomnia. Additionally, this non-pharmacotherapy treatment could be easily applied to PD patients who may have difficulty coming to the clinic frequently due to physical symptoms.     

Assessing the validity of computer-based test interpretations: rating reliability and individual differences among raters

Three studies were conducted to assess the role of individual differences among raters participating in validation studies for computer-based test interpretations (CBTIs) and to assess the reliability of ratings. Studies centered on two CBTIs for the Sixteen Personality Factor Questionnaire (16PF). The first study involved 54 students who rated the accuracy of the Human Resource Development Report (HRDR) in an experimental context. The second study involved 73 students who rated the HRDR in a nonexperimental context. The third study involved 28 students who rated the Narrative Score Report (NSR) in a nonexperimental context. Results taken together indicate that (a) shrewdness may influence ratings of CBTI accuracy in an experimental context; (b) self-sufficiency, emotional stability, dominance, or abstract thought may influence ratings in a nonexperimental context, depending on the specific report involved: (c) test-retest reliabilities for accuracy ratings were .74 for HRDR and .81 for NSR; and (d) test-retest reliabilities for usefulness ratings were .75 for HRDR and .54 for NSR.     

A sociolinguistic approach to the belief-similarity model of racial attitudes

This study examined the belief-similarity model of prejudice from a sociolinguistic perspective. It was hypothesized that normatively regulated speech styles strongly affect observers' assumptions about the cultural background of the speaker. These linguistically based cultural assumptions were expected to override racial characteristics in controlling intergroup attitudes. Stimulus speech styles were Black English Vernacular (BEV) and Standard English (SE). Speech style was expected to strongly affect prejudicial attitudes, with such effects mediated by assumed cultural similarity. Racial label and speech style were expected to be most salient to ratings of “intimate” behavior and among more ethnocentric subjects. Subjects heard taped statements in either BEV or SE, ostensibly delivered by a White or a Black speaker. Subjects rated the speaker on perceived cultural similarity, general evaluation, perceived aggressiveness, and social distance. Speech style had a substantial main effect on each of these variables. Racial label had a marginally significant effect on evaluation, and interacted with ethnocentrism for perceived similarity and social distance. All effects of speech, race, and ethnocentrism were substantially attenuated or eliminated when similarity was used as a covariate. Thus, speech style had substantial effects on prejudice, as did race within more ethnocentric subjects. Both effects were largely mediated by assumed cultural similarity.     

The relationship between menopausal status, phase of menstrual cycle, and replacement estrogen on cognition in healthy women without dementia

The authors examined the effect of menopausal status on several aspects of cognition in 4 groups of women (young premenopausal women, middle-aged premenopausal women, naturally postmenopausal women not using hormone therapy, and postmenopausal women using hormone replacement therapy). Participants (N = 48) completed questionnaires designed to assess psychological and physical health. The authors administered a test battery consisting of 10 neuropsychological tests to assess cognitive functioning. Using multivariate analyses of covariance with age as the covariate, the authors found a significant main effect of menopausal status on attention and complex processing abilities. Postmenopausal women using hormone replacement therapy significantly outperformed postmenopausal women not using hormone therapy on the Trail Making Test, Part B of the Halstead-Reitan (R. M. Reitan, 1958). This effect was significant even when the authors controlled for the effects of age, vocabulary levels, and education. Results are consistent with previous findings and may provide further evidence for an ameliorative effect of estrogen replacement therapy on specific cognitive functions.     

Framing experts' (dis)agreements about uncertain environmental events

Agreements and disagreements between expert statements influence lay people's beliefs. But few studies have examined what is perceived as a disagreement. We report six experiments where people rated agreement between pairs of probabilistic statements about environmental events, attributed to two different experts or to the same expert at two different points in time. The statements differed in frame, by focusing on complementary outcomes (45% probability that smog will have negative health effects vs. 55% probability that it will not have such effects), in probability level (45% vs. 55% probability of negative effects), or in both respects. Opposite frames strengthened disagreement when combined with different probability levels. Approximate probabilities can be “framed” in yet another way by indicating reference values they are “over” or “under”. Statements that use different directional verbal terms (over vs. under 50%) indicated greater disagreement than statements with the same directional term but different probability levels (over 50% vs. over 70%). Framing and directional terms similarly affected consistency judgments when both statements were issued by the same expert at different occasions. The effect of framing on perceived agreement was significant for medium (10 and 20 percentage points) differences between probabilities, whereas the effect of directional term was stable for numerical differences up to 40 percentage points. To emphasize agreement between different estimates, they should be framed in the same way. To accentuate disagreements or changes of opinion, opposite framings should be used.     

Lyin' eyes: Ocular-motor measures of reading reveal deception

Our goal was to evaluate an alternative to current methods for detecting deception in security screening contexts. We evaluated a new cognitive-based test of deception that measured participants' ocular-motor responses (pupil responses and reading behaviors) while they read and responded to statements on a computerized questionnaire. In Experiment 1, participants from a university community were randomly assigned to either a "guilty" group that committed one of two mock crimes or an "innocent" group that only learned about the crime. Participants then reported for testing, where they completed the computer-administered questionnaire that addressed their possible involvement in the crimes. Experiment 2 also manipulated participants' incentive to pass the test and difficulty of statements on the test. In both experiments, guilty participants had increased pupil responses to statements answered deceptively; however, they spent less time fixating on, reading, and rereading those statements than statements answered truthfully. These ocular-motor measures were optimally weighted in a discrimination function that correctly classified 85% of participants as either guilty or innocent. Findings from Experiment 2 indicated that group discrimination was improved with greater incentives to pass the test and the use of statements with simple syntax. The present findings suggest that two cognitive processes are involved in deception-vigilance and strategy-and that these processes are reflected in different ocular-motor measures. The ocular-motor test reported here represents a new approach to detecting deception that may fill an important need in security screening contexts. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

Personal importance, motivation, and performance of older adults.

This investigation was conducted to examine whether older adults would improve their performance when given statements intended to make them feel important to a research study as well as statements concerning their over-all well-being. Women (n = 14) and men (n = 15) with mean ages of 76.0 yr. and 75.1 yr., respectively, were randomly assigned into a control group, a positive reinforcement group, and a group given statements of personal importance. Dependent measures on a rotary pursuit task, screw-and-nut assembly task, and an interest questionnaire were obtained during three separate visits. Data were analyzed with a 3 x 3 (Groups x Days) analysis of variance. While no significant group effects were found on any measure, a significant main effect was obtained for days, indicative of learning.     

Covariation and temporal contiguity as principles of causal inference in young children

Four- to seven-year-old children observed a simple physical effect which could be attributed to either a consistent but noncontiguous covariate or a contiguous but inconsistent covariate. When there was a physical rationale for the temporal delay between covariate and effect, children attributed the effect to the consistent but noncontiguous covariate. In the absence of such a rationale, they attributed the effect to the contiguous but inconsistent covariate. It was concluded that neither consistent covariation nor strict temporal contiguity were essential aspects of causal inference for these children.     

Virtual conversation partner for adults with autism

Autistic Spectrum Disorder (ASD) is notable for severely impaired reciprocal social interaction skills relative to language and intellectual abilities, presenting a major barrier to social integration and vocational success. Evidence-based interventions to address these needs are lacking. We report on the development of a small, prototype conversation simulation to teach conversational skills to adolescents and adults with ASD and average to superior intellectual abilities. We also report on a test of the feasibility and acceptability of the simulation approach with a sample of the target population. The simulation engages the user in a virtual conversation with an on-screen partner whose reactions provide naturalistic feedback geared to the appropriateness of the learner's response choices. The prototype simulation, which provides for up to 12 potentially unique multi-turn conversations, was used over a period of 2 weeks by 16 adolescents and adults who then rated statements about the system on a linear scale of 1 (disagreement) to 5 (high agreement). The participants highly endorsed the majority of positive statements about the quality and credibility of the interaction and the virtual conversation partner. In contrast, agreement with positive statements about instructional features external to the conversation was moderate. Unexpectedly, most participants strongly agreed that using the simulation had been helpful to them. Further development and testing in the context of a controlled study with randomized assignment to control and experimental groups are needed to determine whether this approach is effective in improving real-world pragmatic language behavior of high-functioning adults with ASD.     

Robustness of parameter and standard error estimates against ignoring a contextual effect of a subject-level covariate in cluster-randomized trials

In experimental research, it is not uncommon to assign clusters to conditions. When analysing the data of such cluster-randomized trials, a multilevel analysis should be applied in order to take into account the dependency of first-level units (i.e., subjects) within a second-level unit (i.e., a cluster). Moreover, the multilevel analysis can handle covariates on both levels. If a first-level covariate is involved, usually the within-cluster effect of this covariate will be estimated, implicitly assuming the contextual effect to be equal. However, this assumption may be violated. The focus of the present simulation study is the effects of ignoring the inequality of the within-cluster and contextual covariate effects on parameter and standard error estimates of the treatment effect, which is the parameter of main interest in experimental research. We found that ignoring the inequality of the within-cluster and contextual effects does not affect the estimation of the treatment effect or its standard errors. However, estimates of the variance components, as well as standard errors of the constant, were found to be biased.     

Multilevel multidimensional item response model with a multilevel latent covariate

In a pre‐test–post‐test cluster randomized trial, one of the methods commonly used to detect an intervention effect involves controlling pre‐test scores and other related covariates while estimating an intervention effect at post‐test. In many applications in education, the total post‐test and pre‐test scores, ignoring measurement error, are used as response variable and covariate, respectively, to estimate the intervention effect. However, these test scores are frequently subject to measurement error, and statistical inferences based on the model ignoring measurement error can yield a biased estimate of the intervention effect. When multiple domains exist in test data, it is sometimes more informative to detect the intervention effect for each domain than for the entire test. This paper presents applications of the multilevel multidimensional item response model with measurement error adjustments in a response variable and a covariate to estimate the intervention effect for each domain.     

Skulking around the dinosaur: Eliciting cues to children's deception via strategic disclosure of evidence

Research has shown that cues to deception are more salient as an effect of strategic use of evidence (SUE) during interviews. This study examined the feasibility of the SUE‐technique for eliciting cues to children's deception. Experiment 1 investigated verbal cues to deception as a function of early vs. late disclosure of evidence. Eighty‐four children (12–14 years) either guilty or innocent of a mock crime were interviewed. As predicted, deceptive statements were significantly more inconsistent with the evidence than truthful statements, and this was more pronounced as a function of late compared to early disclosure of evidence. In Experiment 2, adult observers (N = 168) made veracity assessments of the videotaped statements. Observers in the late disclosure condition achieved an accuracy rate higher than chance (63.1%), whereas accuracy rates in the early disclosure condition were at chance level (56%). Accuracy rates were significantly higher for truthful (70.2%), than deceptive statements (48.8%). Copyright © 2009 John Wiley & Sons, Ltd.