Revising Test Items to Increase Validity

Eight female interviewers varied their interview styles in interviewing 207 adults who had been involved in automobile accidents during a four-year period. A “professional” and a rapport or “interpersonal” style was used. Dependent variables consisted of reporting of events related to an automobile accident in which the respondent was the driver, and for which independent criterion data were available. In addition, the amount of health information reported was also analyzed. In spite of the respondents' more favorable attitudes towards the interpersonal interviews, accuracy was not significantly greater in these rapport interviews. In fact, significant interaction effects between interviewer style and recency of the accident were obtained. Respondents, however, reported more completely under rapport conditions. The results are interpreted within a cognitive, role-learning framework.     

邪教团体检测项目:量表编制与信效度研究

本研究从提高社会公众认知度出发,通过对中外相关文献研究,结合帮教专职干部、宗教学专家、心理学者、大学生访谈结果,编制了邪教团体检测项目,通过3次条目筛查后制定了62项初始项目;又通过214名大学生探索性因素分析,形成了由5个维度、32个条目组成的正式项目;再通过497名大学生的测查,考察各项测量学指标,结果显示量表区分度、信度、效度指标均较高,结论是邪教团体检测项目具有较高的可靠性和有效性,可作为邪教团体检测的评估工具。     

The Apartment Test: Validity of a Memory Measure

ABSTRACT

The Memory in Reality (MIR) test, or Apartment Test, was designed as an ecologically valid alternative to word list recall measures. Participants name 10 everyday objects, place them in a scale model of an apartment and after a delay with interference, recall the objects and their placement. Two studies are reported. Study 1 included 51 older adult volunteers. Study 2 was a population-based sample of 633 older twins. The Apartment Test Recall and Placement tasks were highly correlated with word list delayed recall. Ecological validity was supported by correlations with informant reports of participants' functioning. Apartment Test Recall was not affected by education. Analysis of Receiver Operating Characteristics (ROC) showed Apartment Test Recall performed as well as word list delayed recall in distinguishing participants with and without dementia. We conclude that the Apartment Test is a valid measure of verbal memory with evidence of ecological and concurrent validity.     

Longitudinal Predictive Validity of a Learning Potential Test

The aim of this article is to provide empirical psychometric evidence of the (longitudinal) predictive validity of a learning potential measure—the Learning Potential Computerised Adaptive Test (LPCAT)—in comparison with standard static tests with school aggregate results as the criterion measure. Participants were 79 boys (mean age 12.44, SD = 0.44) and 72 girls (mean age 11.18, SD = 0.42) attending two private schools. Correlation and regression analyses were used to evaluate the predictive validity of the learning potential and standard test scores for school aggregate academic results as criterion measure. Results indicate that learning potential scores were statistically significant predictors of aggregate academic results and provided results that were comparable to those of the standard test results—providing empirical support for the use of learning potential tests in mainstream educational settings.     

Changes in Liberalism-Conservatism of College Students since the Depression

Summary

One hundred and twenty female subjects took part in an experiment to explore the necessity of deceiving subjects about the purpose of a conformity experiment. The subjects were randomly assigned to one of three groups: no information, partial information, or complete information about the purpose of the experiment. In addition, group pressure to conform was given to half of the subjects on trials on which the correct response was fairly obvious and to the other half on trials on which the correct response was somewhat ambiguous. Anova revealed a significant effect only for item difficulty. The statistic omega squared indicated that none of the variance in the conformity scores could be attributed to amount of information about the purpose of the experiment, either alone or in interaction with item difficulty. It was suggested that these results raise severe doubts about methodological assumptions commonly employed in social Psychology     

Application of a Multidimensional Nested Logit Model to Multiple-Choice Test Items

Nested logit models have been presented as an alternative to multinomial logistic models for multiple-choice test items (Suh and Bolt in Psychometrika 75:454–473, 2010) and possess a mathematical structure that naturally lends itself to evaluating the incremental information provided by attending to distractor selection in scoring. One potential concern in attending to distractors is the possibility that distractor selection reflects a different trait/ability than that underlying the correct response. This paper illustrates a multidimensional extension of a nested logit item response model that can be used to evaluate such distinctions and also defines a new framework for incorporating collateral information from distractor selection when differences exist. The approach is demonstrated in application to questions faced by a university testing center over whether to incorporate distractor selection into the scoring of its multiple-choice tests. Several empirical examples are presented.     

The Validity of the Role Construct Repertory Test as a Measure of Sexual Identification

Although the majority of research on Therapeutic Assessment (TA) discusses the application of TA in research or private practice settings, we found that the model could be applied in a community mental health setting. We argue that when implementing this model with racially diverse, low socioeconomic status children, it is essential to integrate issues of class, privilege, and race into the assessment process. A case is presented that illuminates the specific concerns and struggles of adapting this model to a community psychology population. This analysis includes the interface with systems, placement stability, and consideration of culturally responsible treatment. We also demonstrate how the support of a treatment team helps the individual clinician process and integrate the levels of trauma and pain with which these families present.     

Reliability and Validity of a Paper-and-Pencil Test Measuring Hemisphere Preference

The present studies examined the reliability and validity of the Preference Test (PT), a widely used self-report index of preferred hemisphere thinking styles. The PT consists of items that intend to tap left-hemisphere and right-hemisphere cognitions. In the first study (N=47), PT scores were found to have reasonable test–retest stability. In the second study (N=334), a factor analysis of PT scores showed that the PT has a two-factor solution that can best be interpreted in terms of putative left- and right-hemisphere items. In the third study (N=29), background EEG activity (F3, F4, P3 and P4) of subjects was recorded. PT scores were found to be related to EEG asymmetry patterns for frontal, but not for parietal, recording sites. More specifically, subjects with a left-hemisphere preference displayed relatively stronger left frontal activity than subjects with a right-hemisphere preference. By and large, the present findings provide strong support for the reliability and some suggestive evidence for the validity of the PT.     

Negro Responses to Certain Intelligence Test Items

Fifty-eight female undergraduate students were administered the Torrance Tests of Creative Thinking (TTCT), Figural form B, and the Washington Sentence Completion Test in order to investigate the hypothesis of a positive relationship between creativity and ego development. A Spearman rank order correlation coefficient was computed for ego development and each TTCT creativity factor (fluency, flexibility, originality, and elaboration), as well as a total composite creativity score. All creativity scores correlated with ego development at the .01 level, except for originality, which correlated at the .05 level. Results were discussed in terms of Maslow's theories concerning highly developed persons, and possible preconscious aspects of the creative process. Conclusions were drawn, and recommendations for further research were made.     

论心理测验项目编写的科学性

本文以作为心理测验科学性最基本保证的项目编写为核心,论证了心理测验编制的前提、方法,心理测验项目编写的依据、项目的来源以及项目的分析指标等方面的科学性问题。     

MCMI-III Diagnostic Validity: Bad Test or Bad Validity Study

Operating characteristics describe the validity of tests that attempt to dichotomously predict a diagnosis. These statistics are not fully published in the Millon Clinical Multiaxial Inventory-III Manual (MCMI-III manual Millon, 1994) When calculated from available statistics, the positive predictive powers of the MCMI-III scales are poor both in absolute terms and relative to the MCMI-II (Millon, 1987). There were a number of problems, however, with the initial MCMI-III validity study both inherently and in execution. Although it is doubtful that the MCMI-III is weaker than the MCMI-II, a new validity is needed.     

No Benefit of Eye-Closure Rehearsal in a Unimodal Recognition Memory Test for Word Items

Recent studies have suggested that closing the eyes helps memory retrieval in recall tests for audiovisual clips that contain multimodal information. In two experiments, we examined whether eye-closure improves recognition memory performance for word lists presented unimodally (i.e., visually or aurally). In the encoding phase, participants saw or heard a list of unrelated, meaningful word items. After a fixed retention interval of 1 week (Experiment 1, n = 110) and 5 min (Experiment 2, n = 44), the participants were asked to mentally rehearse the items with their eyes open or closed, and then they performed a recognition test. The results revealed no effect of eye-closure rehearsal on recognition performance. We discuss the possible reasons why no eye-closure benefit was found in recognition memory tests for unrelated word items.     

Item-focussed Trees for the Identification of Items in Differential Item Functioning

A novel method for the identification of differential item functioning (DIF) by means of recursive partitioning techniques is proposed. We assume an extension of the Rasch model that allows for DIF being induced by an arbitrary number of covariates for each item. Recursive partitioning on the item level results in one tree for each item and leads to simultaneous selection of items and variables that induce DIF. For each item, it is possible to detect groups of subjects with different item difficulties, defined by combinations of characteristics that are not pre-specified. The way a DIF item is determined by covariates is visualized in a small tree and therefore easily accessible. An algorithm is proposed that is based on permutation tests. Various simulation studies, including the comparison with traditional approaches to identify items with DIF, show the applicability and the competitive performance of the method. Two applications illustrate the usefulness and the advantages of the new method.     

Validity of Stimulus and Response Components in a Video Test of Social Competence

Various stimulus components (video, orally-presented questions) and response components (multiple-choice, written, orally-given replies) of situational judgement tests of occupational social competency were investigated as to their impact upon the validity for a behavior-oriented role playing criterion while keeping test content constant. The stimulus component video alone had no impact upon validity. The response components contributed to validity. Validity increased with improved fidelity of response components. Concerning stimulus-response-combinations, the validity of two video tests (r = 0.17 and r = 0.36) was not higher than the validity of similar oral questioning (r = 0.13 and r = 0.37) but was significantly lower than a situational interview (r = 0.59). Response fidelity proved to be a bottleneck regarding validity of video tests. As a result, it is recommended that, in order to maximize validity of video and multimedia efforts, the developers of video- and multimedia tests focus special attention on response fidelity.     

The Abbreviated Character Strengths Test (ACST): A Preliminary Assessment of Test Validity

The 24-item Abbreviated Character Strengths Test (ACST) was developed to efficiently measure character strengths (Peterson, Park, &; Castro, 2011 Peterson, C., Park, N., &; Castro, C. A. (2011). Assessment for the US Army Comprehensive Soldier Fitness program: The Global Assessment Tool. American Psychologist, 66, 10–18.[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]). However, its validity for this purpose has not yet been sufficiently established. Using confirmatory factor analysis to test a series of structural models, only a modified bifactor model showed reasonably acceptable fit. Further analyses of this model failed to demonstrate measurement invariance between male and female respondents. Relationships between ACST dimension and Big Five personality trait scores were generally weak-to-moderate, and support for hypotheses regarding each ACST virtue's expected correspondence with specific Big Five dimensions was mixed. Finally, scores on ACST dimensions accounted for a combined 12% of the variance in satisfaction with life scores, after controlling for socially desirability. Although an abbreviated measure of character strengths represents a practical need, considerable improvements to the ACST are needed for it to adequately meet this purpose.     

The Levels Hypothesis and the Study of Stimulus Value of Projective Test Items

Summary

Anagrams and semantic differential (S-D) test responses of Ss for whom words were “loaded” on a free association test were compared with those of Ss for whom the words were neutral. No differences in rate of word building or mean productivity of the two groups on the anagrams test were found. Mean S-D scale values similarly failed to differentiate between the two groups. However, when numbers of neutral (0) and extreme (+3 and ?3) responses were compared, “loaded” Ss were found on the whole to rate words more neutrally or more positively. The conclusion was that the stimulus value of the words in the associative condition could not be directly determined from responses in other conditions. The strategy of studying stimulus value of projective test items by using those items as stimuli on tests eliciting different levels of response was, therefore, questioned. Several other related experimental strategies which may mix response levels were also discussed and more appropriate techniques were suggested.     

Validity of a happiness Implicit Association Test as a measure of subjective well-being

The current study examined the validity of a happiness Implicit Association Test (IAT) as a measure of SWB. One hundred and fifty participants (75 pairs of friends) completed a newly developed happiness IAT and a standard self-esteem IAT. Participants also made self and informant ratings of life satisfaction and happy and sad affect. The results revealed convergent validity among the explicit measures, and among the implicit measures. Explicit–implicit correlations were lower. Self-ratings (.88) and informant ratings (.78) were more highly related with a common SWB factor than the implicit measures (.31). This finding suggests that implicit measures of well-being and happiness assess a different construct that overlaps only modestly with SWB. As a result, these implicit measures have limited utility to control for response styles in self-ratings of SWB.