Response style analysis with threshold and multi‐process IRT models: A review and tutorial

Two different item response theory model frameworks have been proposed for the assessment and control of response styles in rating data. According to one framework, response styles can be assessed by analysing threshold parameters in Rasch models for ordinal data and in mixture‐distribution extensions of such models. A different framework is provided by multi‐process item response tree models, which can be used to disentangle response processes that are related to the substantive traits and response tendencies elicited by the response scale. In this tutorial, the two approaches are reviewed, illustrated with an empirical data set of the two‐dimensional ‘Personal Need for Structure’ construct, and compared in terms of multiple criteria. Mplus is used as a software framework for (mixed) polytomous Rasch models and item response tree models as well as for demonstrating how parsimonious model variants can be specified to test assumptions on the structure of response styles and attitude strength. Although both frameworks are shown to account for response styles, they differ on the quantitative criteria of model selection, practical aspects of model estimation, and conceptual issues of representing response styles as continuous and multidimensional sources of individual differences in psychological assessment.     

Stochastic pairwise comparative judgements and direct ratings of alternatives in the REMBRANDT system

The REMBRANDT system for multicriteria decision analysis consists of both the multiplicative variant of the AHP (which employs a method of pairwise comparative judgements by a decision maker to arrive at final impact scores for the alternatives under consideration) and SMART, the simple multiattribute rating technique (which utilizes direct rating of alternatives to achieve final impact scores). This paper examines the effect of imprecision or uncertainty in the decision maker's pairwise judgements or ratings of alternatives by expressing each pairwise judgement or rating as a probability distribution, and the structure of REMBRANDT's component models is exploited to derive interval judgements or interval ratings of the alternatives’ final impact scores. These interval judgements or interval ratings can be used to determine the probability of rank reversal amongst alternatives, i.e. to assess the stability of the final impact score vector. © 1998 John Wiley & Sons, Ltd.     

Measuring Response Styles Across the Big Five: A Multiscale Extension of an Approach Using Multinomial Processing Trees

This study shows how to address the problem of trait-unrelated response styles (RS) in rating scales using multidimensional item response theory. The aim is to test and correct data for RS in order to provide fair assessments of personality. Expanding on an approach presented by Böckenholt (2012), observed rating data are decomposed into multiple response processes based on a multinomial processing tree. The data come from a questionnaire consisting of 50 items of the International Personality Item Pool measuring the Big Five dimensions administered to 2,026 U.S. students with a 5-point rating scale. It is shown that this approach can be used to test if RS exist in the data and that RS can be differentiated from trait-related responses. Although the extreme RS appear to be unidimensional after exclusion of only 1 item, a unidimensional measure for the midpoint RS is obtained only after exclusion of 10 items. Both RS measurements show high cross-scale correlations and item response theory-based (marginal) reliabilities. Cultural differences could be found in giving extreme responses. Moreover, it is shown how to score rating data to correct for RS after being proved to exist in the data.     

Modeling Intensive Polytomous Time-Series Eye-Tracking Data: A Dynamic Tree-Based Item Response Model

This paper presents a dynamic tree-based item response (IRTree) model as a novel extension of the autoregressive generalized linear mixed effect model (dynamic GLMM). We illustrate the unique utility of the dynamic IRTree model in its capability of modeling differentiated processes indicated by intensive polytomous time-series eye-tracking data. The dynamic IRTree was inspired by but is distinct from the dynamic GLMM which was previously presented by Cho, Brown-Schmidt, and Lee (Psychometrika 83(3):751–771, 2018). Unlike the dynamic IRTree, the dynamic GLMM is suitable for modeling intensive binary time-series eye-tracking data to identify visual attention to a single interest area over all other possible fixation locations. The dynamic IRTree model is a general modeling framework which can be used to model change processes (trend and autocorrelation) and which allows for decomposing data into various sources of heterogeneity. The dynamic IRTree model was illustrated using an experimental study that employed the visual-world eye-tracking technique. The results of a simulation study showed that parameter recovery of the model was satisfactory and that ignoring trend and autoregressive effects resulted in biased estimates of experimental condition effects in the same conditions found in the empirical study.

     

General mixture item response models with different item response structures: Exposition with an application to Likert scales

This article proposes a general mixture item response theory (IRT) framework that allows for classes of persons to differ with respect to the type of processes underlying the item responses. Through the use of mixture models, nonnested IRT models with different structures can be estimated for different classes, and class membership can be estimated for each person in the sample. If researchers are able to provide competing measurement models, this mixture IRT framework may help them deal with some violations of measurement invariance. To illustrate this approach, we consider a two-class mixture model, where a person’s responses to Likert-scale items containing a neutral middle category are either modeled using a generalized partial credit model, or through an IRTree model. In the first model, the middle category (“neither agree nor disagree”) is taken to be qualitatively similar to the other categories, and is taken to provide information about the person’s endorsement. In the second model, the middle category is taken to be qualitatively different and to reflect a nonresponse choice, which is modeled using an additional latent variable that captures a person’s willingness to respond. The mixture model is studied using simulation studies and is applied to an empirical example.     

The influence of decision criteria upon remembering and knowing in recognition memory.

One may recognise an item as 'old' on basis of a recollective experience or by a feeling of familiarity without specific recollection. The former is called a 'remember' judgement; the latter a 'know' judgement. It has been claimed that remember and know responses reflect qualitatively distinct components of recognition memory, and not just derive from gradual differences in perceived trace strength or subjective certainty (i.e. remember judgments include memories of which one is more confident). Nonetheless, the present study examined the possibility that the distinction does relate to decision criteria placed upon a single familiarity axis (see Donaldson, 1996; Hirshman & Master, 1997). To this purpose, two groups of subjects were compared: one, which was instructed to be very conservative in their old-new judgements, while the other group was stimulated to be very lenient instead. Remember hit rates increased with more lenient criteria, whereas know hit rates did not, but false alarm rates did. While remember sensitivity was equal in the two groups, know sensitivity was lower with liberal criteria. Also it correlated with overall response bias. This lends support to the possibility that subjects not only apply an old-new decision criterion, but also set a remember-know criterion, which is affected in a similar way by liberal versus conservative instructions.     

Evaluation on types of invariance in studying extreme response bias with an IRTree approach

In recent years, item response tree (IRTree) approaches have received increasing attention in the response style literature for their ability to partial out response style latent variables as well as associated item parameters. When an IRTree approach is adopted to measure extreme response styles, directional and content invariance could be assumed at the latent variable and item parameter levels. In this study, we propose to evaluate the empirical validity of these invariance assumptions by employing a general IRTree model with relaxed invariance assumptions. This would allow us to examine extreme response biases, beyond extreme response styles. With three empirical applications of the proposed evaluation, we find that relaxing some of the invariance assumptions improves the model fit, which suggests that not all assumed invariances are empirically supported. Specifically, at the latent variable level, we find reasonable evidence for directional invariance but mixed evidence for content invariance, although we also find that estimated correlations between content-specific extreme response latent variables are high, hinting at the potential presence of a general extreme response tendency. At the item parameter level, we find no directional or content invariance for thresholds and no content invariance for slopes. We discuss how the variant item parameter estimates obtained from a general IRTree model can offer useful insight to help us understand response bias related to extreme responding measured within the IRTree framework.     

反应风格的测量与统计控制

反应风格是共同方法偏差的主要来源之一。本文首先讨论反应风格的定义和类型,梳理其危害,认为反应风格能使测验分数出现偏差,影响测验信效度分析和变量关系分析,有必要控制其危害。然后介绍了常用的反应风格测量方法,包括计数法和模型法两大类,对测量方法的选择给出了建议,在此基础上,就如何结合反应风格的测量方法与残差回归法、偏相关法来控制反应风格危害给出建议。     

反应风格的测量与统计控制

反应风格是共同方法偏差的主要来源之一。本文首先讨论反应风格的定义和类型,梳理其危害,认为反应风格能使测验分数出现偏差,影响测验信效度分析和变量关系分析,有必要控制其危害。然后介绍了常用的反应风格测量方法,包括计数法和模型法两大类,对测量方法的选择给出了建议,在此基础上,就如何结合反应风格的测量方法与残差回归法、偏相关法来控制反应风格危害给出建议。     

Combining mixture distribution and multidimensional IRTree models for the measurement of extreme response styles

Personality constructs, attitudes and other non-cognitive variables are often measured using rating or Likert-type scales, which does not come without problems. Especially in low-stakes assessments, respondents may produce biased responses due to response styles (RS) that reduce the validity and comparability of the measurement. Detecting and correcting RS is not always straightforward because not all respondents show RS and the ones who do may not do so to the same extent or in the same direction. The present study proposes the combination of a multidimensional IRTree model with a mixture distribution item response theory model and illustrates the application of the approach using data from the Programme for the International Assessment of Adult Competencies (PIAAC). This joint approach allows for the differentiation between different latent classes of respondents who show different RS behaviours and respondents who show RS versus respondents who give (largely) unbiased responses. We illustrate the application of the approach by examining extreme RS and show how the resulting latent classes can be further examined using external variables and process data from computer-based assessments to develop a better understanding of response behaviour and RS.     

Investigating With IRT and MDS Approaches Translation and Adaptation of Rating Scales for Spanish-Speaking Populations

The goal of this study is to investigate how features of a rating scale developed for English-speaking populations interact with Spanish-speaking respondents' response styles and functional categories of judgment. A sample of 400 Spanish-speaking students took a translated scale and a scaling task developed to measure response sets and functional categories of judgment, respectively. Three response set models—extreme response, central tendency, and acquiescence—under two conditions—base and revised with respondents' functional categories—were studied with item response theory and multidimensional scaling methods. Revising the number of scale categories with the number of salient functional categories statistically improved fit of the base models. Multidimensional scaling results showed scale content features interacting with response styles and functional categories. Translation of rating scales requires adapting scale features to characteristics of target languages, such as salient response styles and respondents' functional categories of judgment.     

Effects of stimulus order on discrimination processes in comparative and equality judgements: Data and models

In typical discrimination experiments, participants are presented with a constant standard and a variable comparison stimulus and their task is to judge which of these two stimuli is larger (comparative judgement). In these experiments, discrimination sensitivity depends on the temporal order of these stimuli (Type B effect) and is usually higher when the standard precedes rather than follows the comparison. Here, we outline how two models of stimulus discrimination can account for the Type B effect, namely the weighted difference model (or basic Sensation Weighting model) and the Internal Reference Model. For both models, the predicted psychometric functions for comparative judgements as well as for equality judgements, in which participants indicate whether they perceived the two stimuli to be equal or not equal, are derived and it is shown that the models also predict a Type B effect for equality judgements. In the empirical part, the models' predictions are evaluated. To this end, participants performed a duration discrimination task with comparative judgements and with equality judgements. In line with the models' predictions, a Type B effect was observed for both judgement types. In addition, a time-order error, as indicated by shifts of the psychometric functions, and differences in response times were observed only for the equality judgement. Since both models entail distinct additional predictions, it seems worthwhile for future research to unite the two models into one conceptual framework.     

Covert judgements are sufficient to trigger subsequent task-switching costs

This research examines whether we have a tendency to repeat mental processes leading to decisions or judgements that are not accompanied by overt behaviours. We adapted the task-switching paradigm so that on selected trials task processing would be terminated prior to response execution. Switch costs were present subsequent to trials where task processing was terminated either at the stage of response selection or at the earlier stage of making a covert judgement (a mental decision) about the target stimulus. These costs were residual, as they occurred despite long preparation intervals, and they did not result from cue-switching or feature-repetition effects. We conclude that the same type of control mechanism may be recruited to select between potential alternative tasks whenever a stimulus needs to be processed in a task-specific way, regardless of whether or not an overt response is required.     

跨文化判断与决策(英文)

本文对判断和决策研究领域所发现的跨文化差异进行了回顾。鉴于大多数判断和决策的跨文化研究都集中于对亚洲和西方文化的比较,本文也主要关注这方面的研究发现。具体来说,本文回顾了在概率判断及信心、风险知觉、冒险行为、消费者行为以及经济判断和决策中所存在的跨文化差异。综述结果表明尽管亚洲人和西方人的判断和决策行为存在很大的跨文化差异,研究也发现了显著的文化内差异。目前关于判断和决策的跨文化差异的研究还相对匮乏,未来还需要更多的研究来进一步了解判断和决策行为的跨文化差别及机制。     

The dimensionality of the remember-know task: a state-trace analysis

This article addresses the issue of whether the remember-know (RK) task is best explained by a single-process or a dual-process model. All single-process models propose that remember and know responses reflect different levels of a single strength-of-evidence dimension. Thus, across conditions in which response criteria are held constant, these models predict that the RK task is unidimensional. Many dual-process models propose that remember and know responses reflect two qualitatively distinct processes underlying recognition memory, often characterized as recollection and familiarity. These models predict that the RK task is bidimensional. Using data from 37 studies, the author conducted a state-trace analysis to determine the dimensionality of the RK task. In those studies, non-memory-related differences between conditions were eliminated via decision criteria constrained to be constant across all levels of the independent variables. The results reveal little or no evidence of bidimensionality and lend additional support to the unequal-variance signal detection model. Other arguments supporting a bidimensional interpretation are examined, and the author concludes there is insufficient evidence for the RK task to be used to identify qualitatively different memory components.     

The Problem of Counterfactuals in Substituted Judgement Decision‐Making

The standard by which we apply decision‐making for those unable to do so for themselves is an important practical ethical issue with substantial implications for the treatment and welfare of such individuals. The approach to proxy or surrogate decision‐making based upon substituted judgement is often seen as the ideal standard to aim for but suffers from a need to provide a clear account of how to determine the validity of the proxy's judgements. Proponents have responded to this demand by providing the truth‐conditions for the substituted judgement in terms of counterfactual reasoning using a possible worlds semantics. In this paper, I show how these underpinnings fail to support the substituted judgement approach as a reasonable standard for decision‐making. Firstly, I show how this counterfactual element has been poorly interpreted. I then explain how various accounts have failed to reflect problems and limitations associated with providing an interpretation of their truth‐conditions using counterfactuals. Finally, I argue that, even when we attend to the initial problems of providing a counterfactual analysis, it still deeply problematic as a means of determining the validity of substituted judgements for two main reasons. Firstly, making determinate judgements as to the truth‐value of these judgements will often not be possible and, secondly, there is a strong requirement when interpreting many counterfactual claims to charitably accede to their being true. I conclude that substituted judgements, as interpreted through counterfactual reasoning and possible worlds semantics, do not therefore provide an adequate standard for surrogate decision‐making.     

Social Judgement Theory and Medical Judgement

Social judgement theory is particularly well suited to the study of medical judgements. Medical judgements characteristically involve decision making under uncertainty with inevitable error and an abundance of fallible cues. In medicine, as in other areas, SJT research has found wide variation among decision makers in their judgements and in the weighting of clinical information. Strategies inferred from case vignettes differ from physicians' self-described strategies and from the weights suggested by experts. These observations parallel recent findings of unexplained variation in diagnosis and management in clinical practice that have been the source of concern in the medical community. The lens model provides one of the few methods for quantitatively analysing physicians' judgements. Contrary to what one might expect from the variation in strategies on paper cases, several studies suggest that, in practice, physicians' diagnostic judgements are highly accurate. Cognitive feedback has been less successful as a practical teaching tool than originally hoped, but some aspects of this methodology show promise, particularly in conjunction with the increasing emphasis on statistical decision support. All things considered, SJT has provided insight into physicians' decisions and gives the medical research community important tools for studying judgements in actual practice.     

Predictions and causal estimations are not supported by the same associative structure

Studies performed by different researchers have shown that judgements about cue-outcome relationships are systematically influenced by the type of question used to request those judgements. It is now recognized that judgements about the strength of the causal link between a cue and an outcome are mostly determined by the cue-outcome contingency, whereas predictions of the outcome are more influenced by the probability of the outcome given the cue. Although these results make clear that those different types of judgement are mediated by some knowledge of the normative differences between causal estimations and outcome predictions, they do not speak to the underlying processes of these effects. The experiment presented here reveals an interaction between the type of question and the order of trials that challenges standard models of causal and predictive learning that are framed exclusively in associative terms or exclusively in higher order reasoning terms. However, this evidence could be easily explained by assuming the combined intervention of both types of process.     

Positive versus negative face recognition decisions: confidence,accuracy, and response latency

A large amount of eyewitness identification and face recognition research has investigated the confidence–accuracy (CA) relationship. One consistent finding is that positive recognition decisions (or choosers) demonstrate superior CA calibration to negative recognition decisions (or non‐choosers). This experiment tested whether an explanation of this difference, based on the information available for confidence judgements, accounted for the pattern of CA calibration in positive and negative face recognition decisions. CA calibration for positive and negative decisions was compared for both item and associative recognition judgements. Significantly greater resolution was observed for positive decisions in both the item and associative conditions. Similarly, for both judgement types, positive decisions evidenced a stronger response latency–accuracy relationship than negative decisions. Implications for diagnosing the accuracy of eyewitness identification are discussed. Copyright © 2005 John Wiley & Sons, Ltd.