An Empirical Investigation of Population Invariance in the Value of Subscores

Recently there has been an increasing level of interest in subtest scores, or subscores, for their potential diagnostic value. Haberman (2008 Haberman, S. J. 2008. When can subscores have value?. Journal of Educational and Behavioral Statistics, 33: 204–229. [Crossref], [Web of Science ®] , [Google Scholar]) suggested a method to determine if a subscore has added value over the total score. Researchers have often been interested in the performance of subgroups—for example, those based on gender or ethnicity—on subtests. Several researchers found that the difference in performance between the gender-based subgroups varied over the different subtests. In this article, we examine whether the added values of the subscores vary between subgroups using data from several operational tests, including an international English proficiency test. For these data sets, the added values of the subscores occasionally vary over the subgroups, but the added values of the augmented subscores are invariant over the subgroups.     

Conceptual and measurement issues of the complementarity dimension of the coach–athlete relationship across cultures

ObjectivesGrounded in the 3Cs model of the coach–athlete relationship, this multi-study outlined the development of two scales that measure coaches' dominant behaviors and athletes' submissive behaviors.Method & resultsIn study 1, a pool of items was generated based on relevant literatures and interviews. The item pool was then assessed by an expert panel including academics, athletes and coaches. In Study 2 and Study 3, construct and criterion validity, as wells as internal reliability of the refined items were tested with a sample of elite coaches and athletes. Analyses revealed that Coach Dominant Behavior Scale (CDB-S) and Athlete Submissive Behavior Scale (ASB-S) possessed sound psychometric properties. In Study 4, the measurement invariance of the 10-item ASB-S was assessed across gender and five countries: Britain, China, Greece, Spain, and Sweden. Results supported the full structural invariance of the ASB-S.ConclusionOverall, the items of both CDB-S and ASB-S were found to be psychometrically sound. The discussion highlights the contributions these findings make on both conceptual and measurement levels as well as the opportunities it opens up for research with practical relevance.     

A model for the time-order error in contrast discrimination

Trials in a temporal two-interval forced-choice discrimination experiment consist of two sequential intervals presenting stimuli that differ from one another as to magnitude along some continuum. The observer must report in which interval the stimulus had a larger magnitude. The standard difference model from signal detection theory analyses poses that order of presentation should not affect the results of the comparison, something known as the balance condition (J.-C. Falmagne, 1985, in Elements of Psychophysical Theory). But empirical data prove otherwise and consistently reveal what Fechner (1860/1966, in Elements of Psychophysics) called time-order errors, whereby the magnitude of the stimulus presented in one of the intervals is systematically underestimated relative to the other. Here we discuss sensory factors (temporary desensitization) and procedural glitches (short interstimulus or intertrial intervals and response bias) that might explain the time-order error, and we derive a formal model indicating how these factors make observed performance vary with presentation order despite a single underlying mechanism. Experimental results are also presented illustrating the conventional failure of the balance condition and testing the hypothesis that time-order errors result from contamination by the factors included in the model.     

Entitlement to Self‐Knowledge and Brute Error

Abstract

I discuss Burge’s argument that our entitlement to self‐knowledge consists in the constitutive relation between the second‐order review of thoughts and the thoughts reviewed, and defend it against Peacocke’s criticism. I then argue that though our entitlement to self‐knowledge is neutral to different environments, as Burge claims, the consideration of Burge’s own notion of brute error shows that Burge’s effort to reconcile externalism and self‐knowledge is not successful.     

The problem of empirical redundancy of constructs in organizational research: An empirical investigation

Construct empirical redundancy may be a major problem in organizational research today. In this paper, we explain and empirically illustrate a method for investigating this potential problem. We applied the method to examine the empirical redundancy of job satisfaction (JS) and organizational commitment (OC), two well-established organizational constructs. Analysis based on responses from a sample of 292 employees collected at two occasions showed that: (a) the construct-level correlation between JS and OC was very high (.91) and (b) both JS and OC are similarly related to positive affectivity and negative affectivity. These results suggest that the constructs may be empirically indistinguishable, despite their well-established conceptual distinction. These findings illustrate the problem of empirical redundancy of organizational constructs and provide a basis for a possible movement towards parsimony in the realm of constructs that could open the way to more rapid advances in knowledge in organizational research.     

Risk of bias in trial-based economic evaluations: Identification of sources and bias-reducing strategies

Objective: The objectives of this article are first to give an overview of the risks of bias in trial-based economic evaluations and, second, to identify how key sources for bias can be revealed and overcome (i.e. what bias-reducing strategies might be employed) in future trial-based economic evaluations in the field of health psychology.

Design: Narrative review discussing sources of bias in trial-based economic evaluations and bias-reducing strategies.

Results: We identified 11 biases and assigned them to a particular trial phase. A distinction is made between pre-trial biases, biases during the trial and biases that are relevant after the actual trial. All potential forms of bias are discussed in detail and strategies are shown to detect and overcome these biases.

Conclusion: In order to avoid bias in trial-based economic evaluations, one has to be aware of all the possible forms of bias. All stakeholders have to examine trial-based economic evaluations in a rigorous and stringent manner. This article can be helpful in this examination as it gives an overview of the possible biases which researchers should take into account.     

Multilevel multidimensional item response model with a multilevel latent covariate

In a pre‐test–post‐test cluster randomized trial, one of the methods commonly used to detect an intervention effect involves controlling pre‐test scores and other related covariates while estimating an intervention effect at post‐test. In many applications in education, the total post‐test and pre‐test scores, ignoring measurement error, are used as response variable and covariate, respectively, to estimate the intervention effect. However, these test scores are frequently subject to measurement error, and statistical inferences based on the model ignoring measurement error can yield a biased estimate of the intervention effect. When multiple domains exist in test data, it is sometimes more informative to detect the intervention effect for each domain than for the entire test. This paper presents applications of the multilevel multidimensional item response model with measurement error adjustments in a response variable and a covariate to estimate the intervention effect for each domain.     

Is reasoning from counterfactual antecedents evidence for counterfactual reasoning?

In most developmental studies the only error children could make on counterfactual tasks was to answer with the current state of affairs. It was concluded that children who did not show this error are able to reason counterfactually. However, children might have avoided this error by using basic conditional reasoning (Rafetseder, Cristi-Vargas, & Perner, 2010 Perner, J. and Rafetseder, E. 2010. “Counterfactual and other forms of conditional reasoning: Children lost in the nearest possible world”. In Understanding counterfactuals/Understanding causation, Edited by: Hoerl, C., McCormack, T. and Beck, S. R. New York: Oxford University Press.  [Google Scholar]). Basic conditional reasoning takes background assumptions represented as conditionals about how the world works. If an antecedent of one of these conditionals is provided by the task, then a likely conclusion can be inferred based only on background assumptions. A critical feature of counterfactual reasoning is that the selection of these additional assumptions is constrained by actual events to which the counterfactual is taken to be counterfactual. In contrast, in basic conditional reasoning one enriches the given antecedent with any plausible assumptions, unconstrained by actual events. In our tasks basic conditional reasoning leads to different answers from counterfactual reasoning. For instance, a doctor, sitting in the park with the intention of reading a paper, is called to an emergency at the swimming pool. The question, “If there had been no emergency, where would the doctor be?” should counterfactually be answered “in the park”. But by ignoring the doctor's intentions, and just reasoning from premises about the default location of a hospital doctor who has not been called out to an emergency, one might answer: “in the hospital”. Only by 6 years of age did children mostly give correct answers.     

Feelings of error in reasoning—in search of a phenomenon

Recent research shows that in reasoning tasks, subjects usually produce an initial intuitive answer, accompanied by a metacognitive experience, which has been called feeling of rightness. This paper is aimed at exploring the complimentary experience of feeling of error (FOE), that is, the spontaneous, subtle sensation of cognitive uneasiness arising from conflict detection during thinking. We investigate FOE in two studies with the “bat-and-ball” (B&B) reasoning task, in its standard and isomorphic control versions. Study 1 is a generation study, in which participants are asked to generate their own response. Study 2 is an evaluation study, in which participants are asked to choose between two conflicting answers (normative vs. intuitive). In each study, the FOE is measured by the FOE questionnaire. Results show that the FOE is significantly present in the standard B&B task when participants give a wrong answer, that our questionnaire can measure it, and furthermore, that it is diagnostic of genuine error.