Invisible Stimuli,Implicit Thresholds: Why Invisibility Judgments Cannot be Interpreted in Isolation

Some studies of unconscious cognition rely on judgments of participants stating that they have “not seen” the critical stimulus (e.g., in a masked-priming experiment). Trials in which participants gave invisibility judgments are then treated as those where the critical stimulus was “subliminal” or “unconscious,” as opposed to trials with higher visibility ratings. Sometimes, only these trials are further analyzed, for instance, for unconscious priming effects. Here I argue that this practice requires implicit assumptions about subjective measures of awareness incompatible with basic models of categorization under uncertainty (e.g., modern signal-detection and threshold theories). Most importantly, it ignores the potential effects of response bias. Instead of taking invisibility judgments literally, they would better be employed in parametric experiments where stimulus visibility is manipulated systematically, not accidentally. This would allow studying qualitative and double dissociations between measures of awareness and of stimulus processing per se.     

The test–retest reliability of the latent construct of executive function depends on whether tasks are represented as formative or reflective indicators

This study investigates the test–retest reliability of a battery of executive function (EF) tasks with a specific interest in testing whether the method that is used to create a battery-wide score would result in differences in the apparent test–retest reliability of children’s performance. A total of 188 4-year-olds completed a battery of computerized EF tasks twice across a period of approximately two weeks. Two different approaches were used to create a score that indexed children’s overall performance on the battery—i.e., (1) the mean score of all completed tasks and (2) a factor score estimate which used confirmatory factor analysis (CFA). Pearson and intra-class correlations were used to investigate the test–retest reliability of individual EF tasks, as well as an overall battery score. Consistent with previous studies, the test–retest reliability of individual tasks was modest (rs ≈ .60). The test–retest reliability of the overall battery scores differed depending on the scoring approach (r_mean = .72; r_{factor_score} = .99). It is concluded that the children’s performance on individual EF tasks exhibit modest levels of test–retest reliability. This underscores the importance of administering multiple tasks and aggregating performance across these tasks in order to improve precision of measurement. However, the specific strategy that is used has a large impact on the apparent test–retest reliability of the overall score. These results replicate our earlier findings and provide additional cautionary evidence against the routine use of factor analytic approaches for representing individual performance across a battery of EF tasks.     

Test–retest reliability of the soleus H-reflex excitability measured during human walking

The purpose of the study was to investigate with what accuracy the soleus H-reflex modulation and excitability could be measured during human walking on two occasions separated by days. The maximal M-wave (Mmax) was measured at rest in the standing position. During treadmill walking every stimulus elicited an M-wave of 25 ± 10% of Mmax in the soleus muscle and a supra-maximal stimulus elicited a maximal M-wave 60 ms after the first stimulus. Both Mmax during rest and during walking were later used for normalization. When normalized to resting Mmax, the peak reflex amplitude during walking was 5% lower on Day 2 than on Day 1 (p = .32). However, when the peak H-reflex was normalized to Mmax in every sweep, Day 2 showed a significant 15% lower amplitude (p = .037). The same pattern was found for the mean H-reflex. Spearman’s Rho was .92 when normalized to resting Mmax but .88 when normalized to Mmax in every sweep. The Pearson product was used to identify one participant at a time on Day 1 among all seven participants on Day 2. For both normalization procedures 5 of 7 participants were identified by this test. Since 5 of 7 participants were recognized between days, it must be recommended to use 10-15 participants for training or intervention studies as far as the H-reflex pattern of modulation during movement is concerned.     

Gender Differences in Self-Reports of Depression: The Response Bias Hypothesis Revisited

This study was designed to revisit the response bias hypothesis, which posits that gender differences in depression prevalence rates may reflect a tendency for men to underreport depressive symptoms. In this study, we examined aspects of gender role socialization (gender-related traits, socially desirable responding, beliefs about mental health and depression) that may contribute to a response bias in self-reports of depression. In addition, we investigated the impact of two contextual variables (i.e., cause of depression and level of intrusiveness of experimental follow-up) on self-reports of depressive symptoms. Results indicated that men, but not women, reported fewer depressive symptoms when consent forms indicated that a more involved follow-up might occur. Further, results indicated differential responding by men and women on measures of gender-related traits, mental health beliefs, and beliefs about depression and predictors of depressed mood. Together, our results support the assertion that, in specific contexts, a response bias explanation warrants further consideration in investigations of gender differences in rates of self-reported depression.     

Artifact, bias, and complexity of assessment: the ABCs of reliability

Interobserver agreement (also referred to here as "reliability") is influenced by diverse sources of artifact, bias, and complexity of the assessment procedures. The literature on reliability assessment frequently has focused on the different methods of computing reliability and the circumstances under which these methods are appropriate. Yet, the credence accorded estimates of interobserver agreement, computed by any method, presupposes eliminating sources of bias that can spuriously affect agreement. The present paper reviews evidence pertaining to various sources of artifact and bias, as well as characteristics of assessment that influence interpretation of interobserver agreement. These include reactivity of reliability assessment, observer drift, complexity of response codes and behavioral observations, observer expectancies and feedback, and others. Recommendations are provided for eliminating or minimizing the influence of these factors from interobserver agreement.     

Children’s social desirability: Effects of test assessment mode

This study examined a recently developed short version of the Children’s Social Desirability (CSD-S) scale with 157 fourth-grade children. Of interest was (a) whether one-month test–retest reliability would vary as a function of test assessment mode (interview or classroom), gender, race, SES, and BMI percentile, and (b) whether the degree of social desirability would vary as a function of these same variables. The CSD-S scale showed good test–retest reliability for both interview and classroom assessment modes (.85 and .83, respectively). Internal consistency also was good (first interview administration = .84; first classroom administration = .81). Reliability was good and did not vary significantly over assessment mode or any child subgroup variables, suggesting that the CSD-S scale is appropriate for general use. The interview mode elicited significantly more socially desirable answers than did the classroom mode. Social desirability did not differ across child subgroups. Some of these findings were examined, and replicated, on another sample. Thus, the CSD-S scale may be used with diverse groups of children to (a) reliably assess a social desirability bias that may systematically bias other self-reports of interest to researchers and (b) examine individual differences in degree of social desirability.     

Self‐esteem,stigma and community participation amongst people with learning difficulties living in the community

A cross‐sectional study examined the relationship between self‐esteem, community participation, age, perceived stigma and social support amongst a sample of 50 people with mild to moderate learning difficulties attending two day centres. Respondent‐paced, structured interviews following ethical and assessment guidelines were used to gather data. Retest interviews were conducted with 20% of the sample 3 months subsequently and only those measures with good test–retest reliability were used to test hypotheses. The results showed fairly high levels of community participation and self‐esteem. Self‐esteem was negatively correlated with stigma and a sub‐group with high concerns regarding meeting strangers was identified. In accordance with studies of non‐disabled samples, self‐esteem was positively associated with age. Social support was comparable to, or higher than, that observed in non‐disabled samples. High social support was observed for staff going out with respondents and for practical help from liked persons and friends. Community participation appeared to be related to higher self‐esteem in older participants but lower self‐esteem in younger participants. Implications for future research and practice are discussed. Copyright © 2002 John Wiley & Sons, Ltd.     

A tale from the pit – educational work at the university

Universities have their own particular ways of doing things. They have their own histories, their titles, their policy documents to attend to, and their own practices that have often evolved and taken form over centuries. The overall aim in this article is to illustrate an individual’s complex growth into becoming a proficient university teacher from being a novice. With a phenomenological point of departure I analyze the embodied working conditions that a newcomer to university teaching carries around and gradually becomes familiar with – and even relatively proficient in – over a period of about three years. The focus in the article is the newcomer’s reflective struggle with understanding her/himself and the social and cultural demands and possibilities that s/he is confronted with.     

A review and analysis of research on the test–retest reliability of professional judgment

This paper analyzes existing research on the test–retest reliability of human judgment, i.e. the extent to which a judge makes identical judgments when presented with identical stimuli on two occasions. Only research involving professional judges who make experimental judgments in a reasonable analog of their everyday experience is included. Studies of both internal consistency reliability and temporal stability reliability are analyzed (where the former refers to the inclusion of repeat stimuli in the same experimental session, and the latter refers to the repeating of the experimental task from a few days to several months later). It is found that (1) the test–retest reliability literature is concentrated in four substantive judgment areas (medicine/psychology, meteorology, human resources management, and business), (2) the literature is extremely variable in terms of research approach/design, the determinants or correlates of test–retest reliability that have been studied, and the quality of the execution and analysis, and (3) mean test–retest reliability differs across both substantive judgment areas and the internal consistency versus temporal stability distinction. An inescapable conclusion from the analysis is that our knowledge of this fundamental property of human judgment is quite meager. Therefore, the paper concludes with suggestions about future research that would address test–retest reliability more systematically. Copyright © 2000 John Wiley & Sons, Ltd.