Assessing the Reliability of Situational Judgment Tests Used in High‐Stakes Situations

Assessing reliability of situational judgment tests (SJTs) in high‐stakes situations is problematic with reliability inappropriately measured by Cronbach's alpha when test items are heterogeneous. We computed the corrected, weighted mean alpha from 56 alpha coefficients, which produced a value of α = .46 and reviewed appropriate types of reliability to use with SJTs. In the current longitudinal study, SJT test–retest reliability was r = .82, compared with internal consistency, α = .46, and stratified alpha, α = .45 at Time 1 and α = .52 and stratified α = .51 at Time 2. We used a student sample (Time 1: n = 185; Time 2: n = 132) with items from a credentialing exam with ‘should do’ instructions. The SJT correlated significantly with cognitive ability, r = .30, and agreeableness, r = .24. In Study 2, we assessed test–retest reliability with Human Resource professionals (Time 1: n = 94; Time 2: n = 32) who had been recently credentialed and who participated in a pilot test of new SJT items with ‘most likely/least likely do’ response options. The SJT test–retest reliability was r = .66 compared with internal consistency, α = .43 and stratified α = .47 at Time 1 and α = .61 and stratified α = .67 at Time 2. We discuss the theoretical implications of the Study 1 results as well as the practical implications for use of SJTs in credentialing examinations.     

The reward probability index: design and validation of a scale measuring access to environmental reward

Behavioral models of depression implicate decreased response-contingent positive reinforcement (RCPR) as critical toward the development and maintenance of depression (Lewinsohn, 1974). Given the absence of a psychometrically sound self-report measure of RCPR, the Reward Probability Index (RPI) was developed to measure access to environmental reward and to approximate actual RCPR. In Study 1 (n = 269), exploratory factor analysis supported a 20-item two-factor model (Reward Probability, Environmental Suppressors) with strong internal consistency (α = .90). In Study 2 (n = 281), confirmatory factor analysis supported this two-factor structure and convergent validity was established through strong correlations between the RPI and measures of activity, avoidance, reinforcement, and depression (r = .65 to .81). Discriminant validity was supported via smaller correlations between the RPI and measures of social support and somatic anxiety (r = – .29 to – .40). Two-week test–retest reliability was strong (r = .69). In Study 3 (n = 33), controlling for depression symptoms, hierarchical regression supported the incremental validity of the RPI in predicting daily diary reports of environmental reward. The RPI represents a parsimonious, reliable, and valid measure that may facilitate understanding of the etiology of depression and its relationship to overt behaviors.     

Dissociation between Categorical and Coordinate Spatial Computations: Modulation by Cerebral Hemispheres, Task Properties, Mode of Response, and Age

Kosslyn suggested a dissociation of two kinds of spatial representations: categorical and coordinate, the former computed by the left hemisphere and the latter by the right hemisphere. In addition, with practice, a “categorization” of the coordinate computation would appear. These suggestions resulted largely from an experiment, replicated by others, in which the subjects had to estimate the relative position of a dot and a line by giving an oral response, and feedback was provided. The present series of five experiments was an attempt to test whether this finding could be generalized, under several methodological manipulations, some of which have already been used by researchers in separate studies. In the five experiments, accuracy was a more contributive dependent variable than correct latencies, and practice effects on the task × field interaction were not verified. Experiments 1–3 concerned the kind of response. When a manual instead of an oral response was required (Experiment 1;n= 32 Ss), the expected dissociation was observed (as well as when the latency of responses made by the right hand were studied). When the number of oral responses was increased from two (a binary choice) to eight, the dissociation was still observed for accuracy but disappeared when a more liberal criterion of accuracy was used to reduce the considerable task effect (Experiment 2;n= 32 Ss), or when response requirements were equated for both tasks (Experiment 3;n= 32 Ss). In Experiment 4 (n= 32 Ss), a manual response was called but the feedback was removed and the task dissociation disappeared. Finally, the task dissociation observed in Experiment 1 was not verified when a sample of elderly subjects was enrolled (Experiment 5;n= 32 Ss). However, age per se was the source of an interesting additional dissociation since only the coordinate computation was age sensitive. Our results suggest that the dissociation proposed by Kosslyn between the computation of categorical vs coordinate spatial relationships is highly unstable and sensitive to subtle methodological factors (vocal vs manual response, presence vs absence of feedback, binary vs “continous” response, age) which could preclude its general application.     

The Experimental Measurement of a Social Hierarchy in Gallus Domesticus: VI. Preliminary Identification of Social Law

Bousfield and Sedgewick showed that the cumulative number of associations produced to a category as a function of time could be described by an exponential function having the parameters m (rate of association) and C (total number of associates produced). In two experiments using male and female college students (N = 16 and N = 14, respectively), the present study investigated the hypothesis suggested by previous research, that the values of m and C for different categories are negatively correlated. S s produced examples of different categories in four-minute periods per category on mimeographed sheets. The predicted negative correlation across categories was found (Experiment I, r = .54, p < .05, one-tailed test; Experiment 2, r = .75, p < .01). Categories with a low m value had a high C value and vice versa. The effect can be explained by a model which assumes exemplars are stored more or less densely in semantic memory.     

Effects of an Auditory Model on the Learning of Relative and Absolute Timing

The effects of an auditory model on the learning of relative and absolute timing were examined. In 2 experiments, participants attempted to learn to produce a 1,000- or 1,600-ms sequence of 5 key presses with a specific relative-timing pattern. In each experiment, participants were, or were not, provided an auditory model that consisted of a series of tones that were temporally spaced according to the criterion relative-timing pattern. In Experiment 1, participants (n = 14) given the auditory template exhibited better relative- and absolute-timing performance than participants (n = 14) not given the auditory template. In Experiment 2, auditory and no-auditory template groups again were tested, but in that experiment each physical practice participant (n = 16) was paired during acquisition with an observer (n = 16). The observer was privy to all instructions as well as auditory and visual information that was provided the physical practice participant. The results replicated the results of Experiment 1: Relative-timing information was enhanced by the auditory template for both the physical and observation practice participants. Absolute timing was improved only when the auditory model was coupled with physical practice. Consistent with the proposal of D. M. Scully and K. M. Newell (1985), modeled timing information in physical and observational practice benefited the learning of the relative-timing features of the task, but physical practice was required to enhance absolute timing.     

Reliability and Validity of the Current Anxiety Level Measure

Professionals interested in assessing state anxiety (in the moment) levels before and after brief activity-based treatments would find limited evidence-based assessments. The purpose of this study was to develop and test the current anxiety level measure, a free, valid, and reliable state anxiety assessment. To establish concurrent validity, the current anxiety level measure was administered (n?=?98) alongside the State-Trait Anxiety Inventory. Results found a strong relationship (r?=?.904, p?r?=?.702, p?n?=?102). Both samples showed the current anxiety level measure to be reliable and valid.     

Effect of Sex and Joystick Experience on Pursuit Tracking in Adults

Using a joystick, adults (n = 39 males, 40 females in Experiment 1; n = 35 males, 40 females in Experiment 2; and n = 18 males, 18 females in Experiment 3) performed a computerized pursuit tracking task. Contrary to previously reported findings, the males were not more accurate than the females when performance was adjusted for prior perceptual-motor experience. Although no sex differences were found in a speeded tracking task, in an inverted tracking task the males exhibited a significant performance advantage; that advantage remained after several blocks of practice. Because participants' performance was adjusted statistically for prior perceptual-motor experience, the male advantage in inverted tracking was not related to experience. Rather, more proficient inverted tracking performance was associated with higher 3-dimensional mental rotations scores. In sum, sex differences in normal pursuit tracking may be better explained by differences in perceptual-motor experience. Inverted tracking, however, may depend on proficiency with spatial transformations.     

Reliability of the sliding scale for collecting affective responses to words

Warriner, Shore, Schmidt, Imbault, and Kuperman, Canadian Journal of Experimental Psychology, 71; 71–88 (2017) have recently proposed a slider task in which participants move a manikin on a computer screen toward or further away from a word, and the distance (in pixels) is a measure of the word’s valence. Warriner, Shore, Schmidt, Imbault, and Kuperman, Canadian Journal of Experimental Psychology, 71; 71–88 (2017) showed this task to be more valid than the widely used rating task, but they did not examine the reliability of the new methodology. In this study we investigated multiple aspects of this task’s reliability. In Experiment 1 (Exps. 1.1–1.6), we showed that the sliding scale has high split-half reliability (r = .868 to .931). In Experiment 2, we also showed that the slider task elicits consistent repeated responses both within a single session (Exp. 2: r = .804) and across two sessions separated by one week (Exp. 3: r = .754). Overall, the slider task, in addition to having high validity, is highly reliable.     

Children's Causal Explanations of Psychogenic Bodily Reactions

In Experiment 1, 4‐, 5‐, 8‐ and 11‐year‐old Japanese children (n = 69) and adults (n = 21) explained their reasons for bodily induced reactions (e.g. overeating leads to vomiting) and psychogenic bodily reactions (bodily outcomes originating in the mind, e.g. frustration leads to vomiting). Children gave vitalistic explanations, that is, explaining causal connections by referring to a vital force, in responses concerning bodily induced reactions, whereas adults typically gave these explanations in responses concerning psychogenic bodily reactions. In Experiment 2, 5‐, 8‐ and 11‐year‐old children (n = 96) and adults (n = 24) explained bodily induced and psychogenic bodily reactions, and psychological behaviour, for example, that frustration leads to nail biting. As in Experiment 1, vitalistic explanations tended to be found for psychogenic tasks but were seldom found in either children's or adults' explanations of psychological behaviour. The findings suggest that with age vitalistic causality obtains cross mind‐body implications. Copyright © 2013 John Wiley & Sons, Ltd.     

Mindfulness and False-Memories: The Impact of Mindfulness Practice on the DRM Paradigm

Mindfulness practice is the cultivation of awareness to the present moment and has been shown in recent years to have beneficial effects on cognition. However, to date, the data regarding the impact of mindfulness on memory—and specifically on memory distortions—is scarce and incomplete. The present study was aimed to examine whether mindfulness practice would have an effect on true and false memories. To this end, the effect of mindfulness meditation practice on memory performance was examined in two experiments in which false memories were provoked using the Deese-Roediger-McDermott (DRM) paradigm (Roediger & McDermott, 1995). In Experiment 1, college students were randomly divided into either a 5-week mindfulness-practice group (n = 29) or a waitlist control group (n = 22). In Experiment 2, college students were randomly divided into either a brief mindfulness session (n = 21) or a mind-wandering control group (n = 19). The results indicated that mindfulness increased the recognition of true memories with no effect on spontaneous false-memories, yet increased the rate of provoked false-memories. These findings are discussed in terms of memory sensitivity and response bias, and it is argued that mindfulness may have a lesser effect on encoding processes than previously suggested.     

Measuring Prosociality: The Development of a Prosocial Behavioral Intentions Scale

Prosociality is a critical issue in behavioral research. In this investigation, we developed a measure of prosocial behavioral intentions. Qualitative responses from two surveys (n = 465) and items from existing measures were used to generate a list of prosocial behaviors in which people might intend to engage. We factor analyzed responses to these items (n = 319) and retained the most common and representative items. The new measure demonstrated adequate internal consistency (n = 247, 147; α = .81, .83); convergent validity with past prosocial behavior (r = .51, .43), moral identity (r = .50, .55), and materialism (r = –.30, –.20). The instrument also predicted prosocial behavior while controlling for a prior measure of prosocial intentions, Exp(B) = 1.99, Wald = 10.59, p = .001, thereby demonstrating incremental predictive validity. This 4-item scale could be used across contexts to advance the study of prosociality.     

Accuracy of respiratory symptom perception in persons with high and low negative affectivity

Accuracy of respiratory symptom perception was investigated in different contexts in participants (N = 56) scoring high or low for negative affectivity (NA). Within subject-correlations were calculated between minute ventilation (frequency per minute × tidal volume) and the subjective symptom ‘faster and/or deeper breathing’ across 10 subsequent breathing trials of 2?min with varying air mixtures, containing fresh or foul smelling odours and/or 5.5% CO₂. Half the participants were given a positive information frame for the sensations (‘some air mixtures may induce a pleasantly arousing feeling, like when being in love’), whereas the other half was given a negative frame (‘some air mixtures may induce feelings of being anxious and distressed’). Interoceptive accuracy was overall fairly high (r = 0.56–0.74), but it dropped considerably (r = 0.27), when bodily sensations were induced in high NA persons in a negative information frame (interaction, p < 0.005). Interoceptive accuracy appears low when persons with high NA are in situations characterised by negative affective cues.     

Children's memory for television advertising: effects of programme–advertisement congruency

The effects of the surrounding programme environment and level of involvement upon children's memory (year 6 primary school, mean age 11.1, SD = 0.4, 48 boys and 69 girls) for television advertisements was examined by manipulating the degree of congruency between the production genre of the programme and the advertisements embedded within it. In Experiment One (n = 56) the ad‐break comprised either all cartoons (n = 4) or all non‐cartoons (n = 4). In Experiment Two (n = 61) cartoon and non‐cartoon advertisements were equally represented in each ad‐break. After viewing, participants provided evaluative ratings of the programme and completed memory tests for advertisements, using prompted recall and brand‐recognition tasks. Memory performance overall was better for cartoon advertisements than for non‐cartoon advertisements, with the latter being poorly remembered when presented in the non‐cartoon programme. In Experiment Two, the programme had no effect on overall recall of advertising, but was related to brand recognition, which was better for advertisements placed in the cartoon programme. Free recall of non‐cartoon advertisements was best when placed in the cartoon programme, while free recall only of cartoon advertisements was best when placed in the non‐cartoon programme. The measures of audience involvement (programme ratings) did not show any significant correlation with memory for advertising. Copyright © 2002 John Wiley & Sons, Ltd.     

The Socioeconomic Status,Intelligence, and Psychometric Pattern of Native-Born White Homeless Men

Build-up of proactive interference (PI) with visual-picture and auditory-verbal input modalities and the subsequent release from PI following a change in modality was investigated in three experiments with boys and girls, as follows: Experiment I (n = 64) at two mean age levels, 7–6 and 10–5; Experiment II (n = 64) at mean age 7–6; and Experiment III (n = 48) at age 11–4. PI build-up occurred in both modalities for all ages tested. Release from PI occurred following a change from auditory to visual input but not following a visual to auditory shift. In the final experiment, this asymmetrical improvement in performance was dependent upon an interaction between the modality of the input and distractor task on the final or release trial; changing to visual input produced a release effect regardless of the distractor task modality, while auditory input was associated with improvement in recall if a visual distractor task was employed whether or not a shift in input modality had occurred. This improvement was hypothesized to represent a decrease in retroactive interference rather than a release from proactive interference.     

Olfactory working memory: exploring the differences in n-back memory for high and low verbalisable odorants

ABSTRACT

We describe four experiments each examining n-back performance for high and low verbalisable odorants. Participants were presented with a sequence of odorants and were required to state if the current odorant was the same or different to the odorant presented two items earlier. Experiment 1 reported superior performance for high, relative to low, verbalisable odorants and was evident despite above-chance memory performance for the low verbalisable odorants. Experiment 2 showed that such superiority persisted with a concurrent articulation condition, suggesting that the memory benefit was not a consequence of verbal recording and rehearsal. Experiment 3 employed metacognitive judgments and showed that correct 2-back responses for high verbalisable odorants received more recollection responses compared to low verbalisable odorants. Experiment 4 compared n-back performance across different stimulus types and showed that, for high verbalisable odorants, performance correlated with both letters and abstract shapes, but such correlations were absent for low verbalisable odorants. Taken together, these findings show differences in n-back performance between high and low verbalisable odorants, and show that high verbalisable odorants exhibit performance similarities with both verbal and visual stimuli. We further argue that n-back performance for low verbalisable odorants operates differently to that of high verbalisable odorants.     

The Effects of Warnings and Speeding on Scale Scores and Convergent Validity of Conscientiousness

We examined the effects of warnings and speeding on scale scores and convergent validity of a measure of Conscientiousness in a faking context (N = 329). A completely crossed 2 × 2 experimental design was used in which instructions (no warning or a warning) and speeding (with or without a time limit) were manipulated. No statistically significant effects on scale scores or convergent validity were evidenced for speeding. Warning participants did decrease Conscientiousness scores by almost 1 standard deviation (d = .91). Warnings also moderated the relationship between self- and observer-ratings of Conscientiousness such that the relationship between self- and observer-ratings was statistically significant and positive in the warned conditions (r _partial = .29, p < .01, n = 136), but this same relationship was statistically nonsignificant (r _partial = ?.14, p > .05, n = 148) in the unwarned conditions.     

The Simple Copy Task: Detecting Higher Order Visual Processing Deficits in Schizophrenia,Dementia, and Movement Disorder Groups

The Simple Copy Task (SCT) is a figure copying test with inherent appeal due to its short administration time, graded task difficulty, varied stimuli, and potential to eliminate floor effects. Despite this, there is a lack of data regarding its construct validity and diagnostic utility. The present study compared SCT performance of schizophrenia (n = 29), dementia (n = 64), and movement disorder (n = 12) groups with that of unmatched healthy control participants (n = 49). Movement disorder patients committed a high degree of misplacement errors, whereas dementia patients tended to omit items, add extraneous detail, and perseverate. The schizophrenia group was most similar to the dementia group in their performance on the SCT, committing primarily omission and perseveration errors. The SCT was most closely related to the Rey Complex Figure Task (r = 0.68, p < .01) and the Block Design Task (r = 0.62, p < .01). Age (r = ?0.14, p < .01) and education (r = 0.35, p < .01) effects were present; however, there was no impact of gender and handedness. Taken together, these findings provide support for the utility of the task and directions for clinical interpretation.     

Sex differences in information processing

Sex differences in cognitive activity were investigated. Using a list containing words with masculine, feminine, and neutral connotations, Experiment 1 revealed that masculine words were better recalled by male college students (n=16) while feminine words were better recalled by female college students (n=16). This difference was reduced on a recognition test. Experiment 2 demonstrated that the phenomenon was not due to a difference in the way males (n=20) and females (n=20) interpreted the meaning of the words. Finally, in Experiment 3, the memorial selectivity was shown to exist in the same manner for White middle-class elementary school (n=32) and high school (n=32) students. Learning and repression interpretations of the effect were proposed. Some practical implications of the data were suggested.These investigations were supported by a seed grant to the first author. This grant was jointly funded by Southern Methodist University and the National Science Foundation's Institutional Grants for Science Program (GU-3752). The authors would like to express their appreciation to the pupils and teachers on Fain Elementary School and Rider High School, both in Wichita Falls, Texas.     

The Feasibility of Cognitive Screening of Patients with Ischaemic Stroke Using the Preliminary Neuropsychological Battery

In the present study, we assessed the reliability, validity and diagnostic accuracy of the non-verbal Preliminary Neuropsychological Battery (PNB) in a sample of stroke patients (n = 149) without significant aphasia and a sample of similarly aged healthy elderly controls (n = 122). The results indicate that as a brief bedside test of global cognitive function the PNB had satisfactory internal, external reliability and validity in this population and a cut-off point of ≤? 55 on the PNB had a sensitivity of 71% and specificity of 94% to detect significant cognitive impairments. Furthermore, better performance on the non-symbolic section of the PNB was associated improved with executive function. In common with other brief neuropsychological tests, the PNB was susceptible to ceiling, educational, functional capability and depression effects. The results here indicate that further research is needed to determine the utility of the PNB in a longitudinal cohort of aphasic and non-aphasic stroke survivors.     

Improving eyewitness recall for licence plates

Two methods for maximizing the completeness and accuracy of eyewitness recall for licence plates were evaluated in this research: (1) asking questions derived from generally accepted principles of memory retrieval enhancement (Geiselman, Fisher, Firstenberg, Hutton, Sullivan, Avetissian and Prosk, 1984; Geiselman, Fisher, MacKinnon and Holland, 1985, 1986) and (2) providing subjects with a licence plate simulation device to view and interchange alphanumeric characters. Undergraduate subjects viewed a series of slides portraying a young man, placing a television set in a car and driving down the street. With a seven-character California plate in Experiment 1 (n = 151), subjects in the condition with the simulation device and cognitive interviewing had a significant 22 per cent increase in correct licence plate recall. The results were replicated in Experiment 2 (n = 108) with an 18 per cent increase in correct information using a six-character plate. Errors appeared to be those letters adjacent in the alphabet to the licence plate letters.