A Two-Stage Approach to Differentiating Normal and Aberrant Behavior in Computer Based Testing

Statistical methods for identifying aberrances on psychological and educational tests are pivotal to detect flaws in the design of a test or irregular behavior of test takers. Two approaches have been taken in the past to address the challenge of aberrant behavior detection, which are (1) modeling aberrant behavior via mixture modeling methods, and (2) flagging aberrant behavior via residual based outlier detection methods. In this paper, we propose a two-stage method that is conceived of as a combination of both approaches. In the first stage, a mixture hierarchical model is fitted to the response and response time data to distinguish normal and aberrant behaviors using Markov chain Monte Carlo (MCMC) algorithm. In the second stage, a further distinction between rapid guessing and cheating behavior is made at a person level using a Bayesian residual index. Simulation results show that the two-stage method yields accurate item and person parameter estimates, as well as high true detection rate and low false detection rate, under different manipulated conditions mimicking NAEP parameters. A real data example is given in the end to illustrate the potential application of the proposed method.     

Naming and categorization in young children: II. Listener behavior training

Following pretraining with everyday objects, 1- to 4-year-old children received listener training with three pairs of arbitrary stimuli of differing shapes. For each pair, 9 children were trained to select one stimulus in response to the spoken word /zog/ and the other to the spoken word /vek/. Next, in the look-at-sample category match-to-sample test, none categorized the six stimuli correctly when asked to look at the sample before selecting from five comparisons. Seven of these children failed a subsequent test of corresponding speaker behavior (tact test); following tact training, 5 of them passed either a repeat of the look-at-sample category test (2 subjects) or an alternative category test (3 subjects) in which they were required to tact the sample before selecting comparisons. The remaining 2 failed both category tests. Of the 2 who passed the tact test, 1 passed the tact-sample category test; the other failed to complete category testing. Two children were next given a second stimulus set. One passed the look-at-sample category test and the tact test; the other failed both tests but passed the tact-sample category test after tact training. The results show that 1- to 4-year-old children may learn listener behavior without corresponding speaker behavior. The results also show that common listener behavior is not sufficient to establish arbitrary stimulus classes, and they are consistent with the proposition that naming may be necessary for categorization of such stimuli.     

A New Concurrent Calibration Method for Nonequivalent Group Design under Nonrandom Assignment

This study proposes a new item parameter linking method for the common-item nonequivalent groups design in item response theory (IRT). Previous studies assumed that examinees are randomly assigned to either test form. However, examinees can frequently select their own test forms and tests often differ according to examinees’ abilities. In such cases, concurrent calibration or multiple group IRT modeling without modeling test form selection behavior can yield severely biased results. We proposed a model wherein test form selection behavior depends on test scores and used a Monte Carlo expectation maximization (MCEM) algorithm. This method provided adequate estimates of testing parameters.     

Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing

In order to identify aberrant response-time patterns on educational and psychological tests, it is important to be able to separate the speed at which the test taker operates from the time the items require. A lognormal model for response times with this feature was used to derive a Bayesian procedure for detecting aberrant response times. Besides, a combination of the response-time model with a regular response model in an hierarchical framework was used in an alternative procedure for the detection of aberrant response times, in which collateral information on the test takers’ speed is derived from their response vectors. The procedures are illustrated using a data set for the Graduate Management Admission Test^® (GMAT^®). In addition, a power study was conducted using simulated cheating behavior on an adaptive test.     

Test publishers' perspective on "an updated meta-analysis": comment on Van Iddekinge, Roth, Raymark, and Odle-Dusseau (2012)

Van Iddekinge, Roth, Raymark, and Odle-Dusseau's (2012) meta-analysis of pre-employment integrity test results confirmed that such tests are meaningfully related to counterproductive work behavior. The article also offered some cautionary conclusions, which appear to stem from the limited scope of the authors' focus and the specific research procedures used. Issues discussed in this commentary include the following: (a) test publishers' provision of studies for meta-analytic consideration; (b) errors and questions in the coding of statistics from past studies; (c) debatable corrections for unreliable criterion measures; (d) exclusion of laboratory, contrasted-groups, unit-level, and time-series studies of counterproductive behavior; (e) under-emphasis on the prediction of counterproductive workplace behaviors compared with job performance, training outcomes, and turnover; (f) overlooking the industry practice of deploying integrity scales with other valid predictors of employee outcomes; (g) implication that integrity test publishers produce biased research results; (h) incomplete presentation of integrity tests' resistance to faking; and (i) omission of data indicating applicants' favorable response to integrity tests, the tests' lack of adverse impact, and the positive business impact of integrity testing. This commentary, therefore, offers an alternate perspective, addresses omissions and apparent inaccuracies, and urges a return to the use of diverse methodologies to evaluate the validity of integrity tests and other psychometric instruments.     

Conditioned inhibitory effects of discriminated Pavlovian training with food in rats depend on interactions of search modes,related repertoires,and response measures

Like other accounts of conditioned inhibition, behavior systems predicts (and Experiment 1 shows) that during summation and retardation tests, presentation of a negative conditioned stimulus (a CS-) created by discriminative Pavlovian food conditioning will interfere with a focal search response, such as nosing in the feeder. Unlike most other views, behavior systems predicts (and Experiment 2 shows) that the same CS- can potentiate a general search response, like attending to a moving artificial prey stimulus. Contacting the prey stimulus in extinction increased over baseline when a CS- but not a CS Novel preceded it. Experiment 3 showed this effect was not due to unconditioned qualities of the CS-. It appears that the effects of a discriminative CS- depend on the interaction of the training contingency with search modes related to the unconditioned stimulus (US), their perceptual-motor repertoires and environmental support, and the choice of response measure.     

The criterion-related validity of integrity tests: an updated meta-analysis

Integrity tests have become a prominent predictor within the selection literature over the past few decades. However, some researchers have expressed concerns about the criterion-related validity evidence for such tests because of a perceived lack of methodological rigor within this literature, as well as a heavy reliance on unpublished data from test publishers. In response to these concerns, we meta-analyzed 104 studies (representing 134 independent samples), which were authored by a similar proportion of test publishers and non-publishers, whose conduct was consistent with professional standards for test validation, and whose results were relevant to the validity of integrity-specific scales for predicting individual work behavior. Overall mean observed validity estimates and validity estimates corrected for unreliability in the criterion (respectively) were .12 and .15 for job performance, .13 and .16 for training performance, .26 and .32 for counterproductive work behavior, and .07 and .09 for turnover. Although data on restriction of range were sparse, illustrative corrections for indirect range restriction did increase validities slightly (e.g., from .15 to .18 for job performance). Several variables appeared to moderate relations between integrity tests and the criteria. For example, corrected validities for job performance criteria were larger when based on studies authored by integrity test publishers (.27) than when based on studies from non-publishers (.12). In addition, corrected validities for counterproductive work behavior criteria were larger when based on self-reports (.42) than when based on other-reports (.11) or employee records (.15).     

Valid representation of a highly dynamic collision avoidance scenario in a driving simulator

In Germany the second-most frequent accidents in road traffic are rear-end collisions. For this reason rear-end collisions are quite important for accident research and the development of driving safety systems. To examine the functionality and to design the human–machine-interface of new driving safety systems, especially in the early development phase, subject tests are necessary. Because of the great hazard potential of such safety critical scenarios for the test persons, they are often conducted in a driving simulator (DS). Accordingly, validity is an important qualification to ensure that the findings collected in a simulated test environment can be directly transferred to the real world.This paper regards the question of driving behavior validity of DS in critical situations. There are hardly any validation studies which analyze the driving behavior in a specific collision avoidance situation.The validation study described in this paper aims to evaluate the behavioral validity of a fixed-base simulator in a collision avoidance situation. For this reason a field study from 2007 was replicated in a fixed-base simulator environment.The main questions of this validation study were if the driver can notice an active hazard braking system and if the driving behavior in a static simulator can be valid in such a critical situation.The key finding of the study states that there is no driving behavior validity in a static driving simulator for the tested dynamic scenario. The missing vestibular feedback causes a different behavior of the participants in field and simulator. The resulting absence of comparability leads to non-valid performance indicators. But these indicators are key parameters for analyzing the function and acceptance of active braking systems. So the question arises, which motion performance does a motion base have to provide in order to achieve valid acceleration simulation of such a highly dynamic collision avoidance scenario. The DS’s performance is measured in workspace, velocity and acceleration.     

Development of clinical tests of vision: Initial data on two hyperacuity paradigms

Many basic psychophysical functions offer promise as clinical tests of vision. Here, we discuss problems that one encounters in the clinical setting, how one identifies a psychophysical test for potential clinical development, and an orderly approach to development of suitable test paradigms. Parameters are selected which are relatively insensitive to variables encountered in the field (clinic) in a normal population, but which are sensitive to changes in the response system being studied. Initial data on two hyperacuity tests are presented. These tests are adaptations of hyperacuity paradigms (Westheimer, 1979) to a clinical environment. This particular test set offers promise because it exhibits a unique threshold which is dependent upon neural data processing and is relatively independent of retinal image degradation.     

Involvement of 5-HT1A receptors in animal tests of anxiety and depression: evidence from genetic models

Clinical studies have suggested the involvement of 5-HT1A receptors in anxiety and depressive disorders because partial 5-HT1A receptor agonists such as buspirone are therapeutic. The present review considers evidence from genetic animal models that support a role for 5-HT1A receptors in anxiety-like and depressed-like behavior in animals. Selective breeding for differential hypothermic responses to a selective 5-HT1A receptor agonist led to the development of the high DPAT sensitive (HDS) and low DPAT sensitive (LDS) lines of rats. The HDS rats differ from the LDS rats on several behavioral measures reflective of anxiety or depression, including reduced social interaction, reduced responding in a conflict task and exaggerated immobility in the forced swim test. However, they do not differ from the LDS rats in the elevated plus maze task, which is a commonly used test of anxiety. Nor do the HDS rats exhibit a typical anxiogenic response to the hippocampal administration of the 5-HT1A agonist. Although the HDS rats do exhibit elevations in 5-HT1A receptors in regions of the limbic cortex, it is not clear whether these increases account for the behavioral differences. Paradoxically, 5-HT1A receptor knockout mice also exhibit anxiety-like behavior in the plus maze, open field and conflict tests compared to wild type mice. However, the knockouts exhibited less immobility in the forced swim test than wild type control mice. Recent studies using selective regional reinstatement of the receptor have implicated the postsynaptic 5-HT1A receptors in these changes in anxiety-like behavior. Thus, preliminary evidence from two different types of genetic animal models suggests that anxiety-like behavior can arise if the 5-HT1A receptor function is eliminated or overexpressed. Further study with additional tests of anxiety are needed to confirm this intriguing relationship.     

Cholinergic system in the toad's (Bufo bufo L.) visual system

The brain structures mediating the prey-catching behavior of the toad have been described in earlier studies but none of these studies has identified the transmitter systems in the optic tectum responsible for this behavior. Behavioral tests with different test compounds provide evidence that the cells responsible for the orienting, jumping, and snapping behaviors associated with feeding in toads are normally inhibited by cholinergic synapses.     

Psychological and educational characteristics of instruction for people with severe disabilities: Interacting systems of responding

This paper presents a behavioral model which proposes that operants are organized and regulated into systems of responding. Multi-operant theory proposes that operants are organized into response systems that interact to adapt behavior to the complexities of the environment. The operant is the interaction between behavior and the environment which includes the conditions under which responses may occur, the class of behavior that is likely to be effective in producing outcomes, and the antecedent conditions that define the context of behavior. A central feature of this theory is that operants within a repertoire are organized into regulated systems of responding. The mechanisms of regulation are themselves operants that are learned and controlled by processes that are the same as those that govern more overt behavior. Operants stand in relationship to each other in coordinated response systems with some operants directly affecting and organizing the performance of other operants. An important implication of the systemic nature of behavioral repertoires is that bringing some aspect of a behavior class under control of new variables may demonstrate a spread throughout the entire operant system depending on past histories linking the classes.deceased     

Defining reinforcess — A problem in communication for consultation in behavior modification

This paper reviews the respondent (Hull-Spence) and operant (Skinnerian) conditioning definitions of reinforcers and reinforcement and demonstrates the need to keep the systems separate when consulting about behavior modification. The two systems are shown to lead to different modification procedures.One important distinction between the systems is whether a reinforcer is simply associated with a response (respondent) or whether it must follow the response (operant). A second important distinction is the definition of negative reinforcement. In respondent conditioning, negative reinforcement entails presenting an aversive stimulus in association with the response and results in a decrease in response rate. In operant conditioning, negative reinforcement entails the removal of an aversive stimulus following a correct response, which results in an increase in response rate.     

Stimulus control of habits: Evidence for both stimulus specificity and devaluation insensitivity in a dual-response task

Goal-directed and habitual actions are clearly defined by their associative relations. Whereas goal-directed control can be confirmed via tests of outcome devaluation and contingency-degradation sensitivity, a comparable criterion for positively detecting habits has not been established. To confirm habitual responding, a test of control by the stimulus–response association is required while also ruling out goal-directed control. Here we describe an approach to developing such a test in rats using two discriminative stimuli that set the occasion for two different responses that then earn the same outcome. Performance was insensitive to outcome devaluation and showed stimulus–response specificity, indicative of stimulus-controlled behavior. The reliance of stimulus–response associations was further supported by a lack of sensitivity during the single extinction test session used here. These results demonstrate that two concurrently trained responses can come under habitual control when they share a common outcome. By reducing the ability of one stimulus to signal its corresponding response–outcome association, we found evidence for goal-directed control that can be dissociated from habits. Overall, these experiments provide evidence that tests assessing specific stimulus–response associations can be used to investigate habits.     

Stimulus–reinforcer relations established during training determine resistance to extinction and relapse via reinstatement

The baseline rate of a reinforced target response decreases with the availability of response‐independent sources of alternative reinforcement; however, resistance to disruption and relapse increases. Because many behavioral treatments for problem behavior include response‐dependent reinforcement of alternative behavior, the present study assessed whether response‐dependent alternative reinforcement also decreases baseline response rates but increases resistance to extinction and relapse. We reinforced target responding at equal rates across two components of a multiple schedule with pigeons. We compared resistance to extinction and relapse via reinstatement of (1) a target response trained concurrently with a reinforced alternative response in one component with (2) a target response trained either concurrently or in separate components from the alternative response across conditions. Target response rates trained alone in baseline were higher but resistance to extinction and relapse via reinstatement tests were greater after training concurrently with the alternative response. In another assessment, training target and alternative responding together, but separating them during extinction and reinstatement tests, produced equal resistance to extinction and relapse. Together, these findings are consistent with behavioral momentum theory—operant response–reinforcer relations determined baseline response rates but Pavlovian stimulus–reinforcer relations established during training determined resistance to extinction and relapse. These findings imply that reinforcing alternative behavior to treat problem behavior could initially reduce rates but increase persistence.     

Individual differences in psychophysiological responsiveness in laboratory tests of deception

The study examined the interrelationship of a number of measures of electrodermal activity under varying degrees of task demand and their relationship to differential electrodermal responding to critical questions in either a card test (n=126) or a mock agent test (n=84) of deception. The data indicated two dimensions of electrodermal responsiveness: one, a reactivity dimension, related predominantly to differences in base level, and the other, a lability dimension, related to differences in response frequency. A third dimension reflecting differences in the extent to which the electrodermal system was the most responsive relative to other systems was not independent of the second. Both the reactivity and lability dimensions related to indices of differential responding in the laboratory tests of deception, with the second increasing the predictive power of the first. The results were interpreted in terms of two-factor theories of the orienting response relevant to the physiological detection of deception.     

Checking the Assumptions of Rasch's Model for Speed Tests

Two new tests for a model for the response times on pure speed tests by Rasch (1960) are proposed. The model is based on the assumption that the test response times are approximately gamma distributed, with known index parameters and unknown rate parameters. The rate parameters are decomposed in a subject ability parameter and a test difficulty parameter. By treating the ability as a gamma distributed random variable, maximum marginal likelihood (MML) estimators for the test difficulty parameters and the parameters of the ability distribution are easily derived. Also the model tests proposed here pertain to the framework of MML. Two tests or modification indices are proposed. The first one is focused on the assumption of local stochastic independence, the second one on the assumption of the test characteristic functions. The tests are based on Lagrange multiplier statistics, and can therefore be computed using the parameter estimates under the null model. Therefore, model violations for all items and pairs of items can be assessed as a by-product of one single estimation run. Power studies and applications to real data are included as numerical examples.     

Examining the Response Competition Hypothesis of Age Effects in Implicit Memory

ABSTRACT

Age reductions in priming have been explained by differences in processing demands across implicit memory tests. According to one hypothesis, older adults show reduced priming relative to younger adults on implicit tests that require production of a response because these tests typically allow for response competition. In contrast, older adults do not show reductions in priming on identification tests that contain little response competition. The following experiments tested the specific role of response competition in mediating age effects in implicit memory. In Experiment 1, younger and older adults studied a list of words and were then given an implicit test of word stem completion. They studied a second list of words and were given an implicit test of general knowledge. Each implicit test contained items with unique solutions (the low response competition condition) and items with multiple solutions (the high response competition condition). In Experiment 2, younger and older adults were given explicit versions of the word stem completion and the general knowledge tests. Results showed an effect of age on explicit memory (Experiment 2), but no effect of age or response competition on priming (Experiment 1). Results are inconsistent with the theory that response competition leads to age effects on production tests of implicit memory.     

POTENTIAL EFFECTS OF BANDING AS A FUNCTION OF TEST RELIABILITY

Cascio, Outtz, Zedeck, and Goldstein (1991) described the application of a number of test score banding procedures in personnel selection. Equations are developed illustrating the relationship between the width of test score bands and test reliability. When reliability is moderate to low, bands are likely to be larger than the standard deviation of the test, and are likely to include a large proportion of the applicant pool. The relationships between band widths and the differences between higher scoring and lower scoring groups are also examined. When the band is smaller than the differences between groups (which may happen when highly reliable tests are used), banding may not by itself prove effective as a means of reducing the adverse impact of tests, even when banding systems that maximize opportunities for members of the lower scoring group are used.     

More Power to the Executive? A Preliminary Test of CBT Plus Executive Skills Training for Treatment of Late-Life GAD

One hypothesized reason for the lower rates of cognitive behavior therapy (CBT) response among older as compared to younger anxiety patients is that they are more likely to show age-related deficits in executive skills, which are complex cognitive skills involved in the regulation of negative affect. Following an 8-week baseline period, this pilot study tested CBT augmented with an executive skills training program, Attention Process Training II (Sohlberg et al., 2001, Sohlberg and Mateer, 2001) against standard CBT in a small sample of 8 older adults with generalized anxiety disorder (comorbidity allowed) and low scores on executive skills tests. Those who received the augmented version (CBT/APT) evidenced more improvement on executive skills and a weekly process measure of worry than those who received CBT. All of the participants in CBT/APT, as compared to half the participants in CBT, met criteria for response, and more in CBT/APT met criteria for high endstate functioning at posttreatment and follow-up. It may be fruitful to test the intervention in a larger sample, and to continue to investigate the role of executive skills in CBT outcome and anxiety treatment.