Application of a Sensemaking Approach to Ethics Training in the Physical Sciences and Engineering

Integrity is a critical determinant of the effectiveness of research organizations in terms of producing high quality research and educating the new generation of scientists. A number of responsible conduct of research (RCR) training programs have been developed to address this growing organizational concern. However, in spite of a significant body of research in ethics training, it is still unknown which approach has the highest potential to enhance researchers' integrity. One of the approaches showing some promise in improving researchers' integrity has focused on the development of ethical decision-making skills. The current effort proposes a novel curriculum that focuses on broad metacognitive reasoning strategies researchers use when making sense of day-to-day social and professional practices that have ethical implications for the physical sciences and engineering. This sensemaking training has been implemented in a professional sample of scientists conducting research in electrical engineering, atmospheric and computer sciences at a large multi-cultural, multi-disciplinary, and multi-university research center. A pre-post design was used to assess training effectiveness using scenario-based ethical decision-making measures. The training resulted in enhanced ethical decision-making of researchers in relation to four ethical conduct areas, namely data management, study conduct, professional practices, and business practices. In addition, sensemaking training led to researchers' preference for decisions involving the application of the broad metacognitive reasoning strategies. Individual trainee and training characteristics were used to explain the study findings. Broad implications of the findings for ethics training development, implementation, and evaluation in the sciences are discussed.     

Retest reliability of the Rosenzweig Picture-Frustration Study and similar semiprojective techniques

Surveys the evidence for the reliability of the Adult, Adolescent and Children's Forms of the Rosenzweig Picture-Frustration (P-F) Study. Analysis of variance and split-half methods, which assume item homogeneity and internal consistency, may be appropriate to psychometric tests, but they are not apposite to projective and semi-projective techniques. The implications of applying such criteria are critically examined, with the P-F Study as an example to demonstrate their nonapplicability. Investigations on retest (total test) reliability from the laboratory of the senior author and from those of other investigators demonstrate statistically significant consistency for the main scoring categories of the P-F, especially those relating to directions of aggression. Implications for other semiprojective techniques are expounded.     

THE PERCEPTIONS AND USAGE OF STATISTICAL POWER IN APPLIED PSYCHOLOGY AND MANAGEMENT RESEARCH

We first assess the current level of statistical power across articles in seven leading journals that represent a broad sample of applied psychology and management research. We next survey the authors of these articles to examine their perceptions and usage of statistical power analysis. Finally, we examine the perceptions and usage of power analysis in a survey of authors of regression-based research appearing in leading journals. Findings from the assessment of power and surveys of researchers indicate that power analyses are not typically conducted, researchers perceive little need for statistical power, and power in published research is low. We conclude by discussing implications of low power for the field and recommending avenues for improving researchers' awareness and usage of statistical power.     

Tests of the relationship between children's temperament and asthma and of the reliability and validity of the Brief Scale of Temperament

Two studies are reported. One examined the reliability and validity of a brief scale to measure children's temperament; the other tested the relationship of early temperament and the development of asthma. In Study 1, principal caretakers of 46 4- to 7-year-olds, usually the mothers, filled out questionnaires containing the Brief Scale of Temperament (BST) and the Behavioral Style Questionnaire. The caretakers did this twice, about 1 week apart. The first time the children's recent temperament was assessed, and the second time past (> 1 year) temperament was assessed. Internal consistency and correlational analyses revealed substantial reliability and validity for BST assessments of recent and past temperament. Study 2 incorporated the BST in a large-scale survey of 325 families, with parents providing data on their children's asthma and temperament. BST assessments of early (past) temperament were made retrospectively regarding the child's first few years. Comparisons of early temperament revealed no differences between children who developed asthma and those who did not in their overall scores on the BST or for any of the temperament characteristics it measures. In addition, correlational analyses of data for asthmatics showed that early temperament was not related to ratings of the severity of the asthma condition the children developed or the impacts of any of 12 specific asthma triggers, including any involving emotional states, such as stress or worry.     

Looking back in time: self-concept change affects visual perspective in autobiographical memory

People who change often report that their old selves seem like "different people." Correlational (Study 1) and experimental (Studies 2 and 3) studies showed that participants tended to use a 3rd-person observer perspective when visualizing memories of actions that conflicted with their current self-concept. A similar pattern emerged when participants imagined performing actions that varied in self-concept compatibility (Study 4). The authors conclude that on-line judgments of an action's self-concept compatibility affect the perspective used for image construction. Study 5 shows applied implications. Use of the 3rd-person perspective when recalling past episodes of overindulgent eating was related to optimism about behaving differently at an upcoming Thanksgiving dinner. The authors discuss the effect of self-concept compatibility on cognitive and emotional reactions to past actions and consider the role of causal attributions in defining the self across time.     

Tests of the Relationship Between Children's Temperament and Asthma and of the Reliability and Validity of the Brief Scale of Temperament

Two studies are reported. One examined the reliability and validity of a brief scale to measure children's temperament; the other tested the relationship of early temperament and the development of asthma. In Study 1, principal caretakers of 46 4- to 7-year-olds, usually the mothers, filled out questionnaires containing the Brief Scale of Temperament (BST) and the Behavioral Style Questionnaire. The caretakers did this twice, about 1 week apart. The first time the children's recent temperament was assessed, and the second time past (> 1 year) temperament was assessed. Internal consistency and correlational analyses revealed substantial reliability and validity for BST assessments of recent and past temperament. Study 2 incorporated the BST in a large-scale survey of 325 families, with parents providing data on their children's asthma and temperament. BST assessments of early (past) temperament were made retrospectively regarding the child's first few years. Comparisons of early temperament revealed no differences between children who developed asthma and those who did not in their overall scores on the BST or for any of the temperament characteristics it measures. In addition, correlational analyses of data for asthmatics showed that early temperament was not related to ratings of the severity of the asthma condition the children developed or the impacts of any of 12 specific asthma triggers, including any involving emotional states, such as stress or worry.     

Issues of Reliability in Measuring Intimate Partner Violence during Courtship

The current paper focuses on problems in conceptualizing and establishing reliability when using self-administered measures of intimate partner violence to assess dating violence. Establishing reliability is an important step in the development of dating violence assessment instruments. However, the nature of dating violence can make it difficult to establish reliability. Most notably, measures of intimate partner violence in courtship yield data that are positively skewed, with almost no one reporting high levels of violence. This could have implications for the calculation of several forms of reliability that assume normality (e.g., Pearson correlations, intraclass correlations). In addition, there are other characteristics of dating violence that could impact reliability. For example, partner violence perpetrators do not necessarily use multiple acts (internal consistency reliability) or repeat specific acts (test-retest reliability). And, gender differences in the perception of partner violence may influence intra-couple reliability in heterosexual couples. Finally, statistical interdependence within couples makes current intra-couple reliability assessment suspect. Research is reviewed and recommendations are made concerning the establishment of test-retest reliability, intra-couple reliability, and internal consistency reliability for measures of dating violence.     

A Reliability Generalization Meta-Analysis of Self-Report Measures of Adult Attachment

This study is a reliability generalization meta-analysis that reviews 5 of the most frequently used continuous measures of adult attachment security: the Adult Attachment Scale, Revised Adult Attachment Scale, Adult Attachment Questionnaire, Experiences in Close Relationships, and Experiences in Close Relationships–Revised. A total of 313,462 individuals from 564 studies provided 1,629 internal consistency reliability estimates for this meta-analysis. We present the average internal consistency reliability of scores for each measure and test the consistency of score reliabilities across a wide variety of sample characteristics. In light of this, we highlight several issues in the measurement of adult attachment security and make concrete recommendations for researchers seeking to measure adult attachment.     

A New Interpretation of the Weighted Kappa Coefficients

Reliability and agreement studies are of paramount importance. They do contribute to the quality of studies by providing information about the amount of error inherent to any diagnosis, score or measurement. Guidelines for reporting reliability and agreement studies were recently provided. While the use of the kappa-like family is advised for categorical and ordinal scales, no further guideline in the choice of a weighting scheme is given. In the present paper, a new simple and practical interpretation of the linear- and quadratic-weighted kappa coefficients is given. This will help researchers in motivating their choice of a weighting scheme.     

Internal consistency reliability of the self-report antisocial process screening device

The self-report version of the Antisocial Process Screening Device (APSD) has become a popular measure for assessing psychopathic features in justice-involved adolescents. However, the internal consistency reliability of its component scales (Narcissism, Callous-Unemotional, and Impulsivity) has been questioned in several studies. This study evaluates the internal consistency reliability of the self-report APSD by examining various indices (e.g., Cronbach's alpha, mean interitem correlation) across 11 studies. Whereas the Narcissism and Impulsivity scales display moderate to good reliability, internal consistency indices for the Callous-Unemotional scale are consistently poor across studies. Suggestions are made for revisions to this scale that would substantially improve its internal consistency reliability.     

Why researchers don't study women: The responses of 62 researchers

Sixty-two researchers accounting for 64 single-sex studies that appeared in 1970 1971 issues of the Journal of Personality and Social Psychology reported why they had limited their studies to a single sex and why they had preferred either males or females for their studies. Also, articles and abstracts were read as a way of checking researchers' reporting practices. The replies were analyzed for thematic content and formed three major types: scientific (56%), practical (28%), and extrascientific (15%). The responses helped to explain why there has been an imbalance in psychology which led to the study of men rather than women instead of the exploration of both sexes. Sggestions are offered to help counteract this trend in future psychological research and reporting.     

Development and validation of a measure of children's automatic thoughts: the children's automatic thoughts scale

The aim of this study was to describe the history, factor structure and psychometric properties of the Children's Automatic Thoughts Scale (CATS). The CATS is a self-report measure designed to assess a wide range of negative self-statements in children and adolescents. The results of confirmatory factor analyses supported four distinct but strongly correlated factors relating to automatic thoughts on physical threat, social threat, personal failure and hostility. The internal consistency of the total score and subscales was high and test-retest reliability at 1 and 3 months was acceptable. The CATS effectively discriminated between control children and adolescents, and clinically anxious, depressed, or behavior disorder children and adolescents, and showed good discriminant validity across clinical subgroups on the subscales. Results suggest that the CATS is a promising instrument in the assessment of a range of negative automatic thoughts across both internalizing and externalizing problems in young people.     

Assessing individual differences in the degree to which people are committed to following their beliefs

The authors developed a new dispositional measure of Commitment to Beliefs (CTB)—the degree to which people feel it is important to follow their value-expressive beliefs—across three phases of research. In Phases 1 and 2 (Studies 1-4), the CTB scale demonstrated strong internal consistency, and convergent and discriminant validity. Phase 3 demonstrated the scale’s predictive validity: high-CTB individuals were more likely to report engaging in activities during the past month and year that followed directly from their beliefs (Study 5), act on their beliefs when given the opportunity (Study 6), and show more polarized attitudes toward ideologically-relevant groups (Study 7). These findings illustrate the utility of a dispositional approach for examining the extent to which people follow their beliefs.     

Dependability of personality, life satisfaction, and affect in short-term longitudinal data

The consistency of individual differences across time has implications for theory building and clinical applications. Indeed, personality psychologists have long worked to place constructs on the continuum of consistency of more trait-like to more state-like constructs. Recently, Chmielewski and Watson () highlighted the importance of dependability coefficients for interpreting the results of stability studies. These coefficients provide an estimate of how strongly short-term transient error affects retest correlations for a given measure. In this article, we use a modified version of Kenny and Zautra's (, ) STARTS model to estimate dependability of personality, life satisfaction, and affect in a 2-month longitudinal study of 8 waves. Results from 226 undergraduate students indicated that personality ratings were least influenced by transient state factors, whereas affect was most influenced. We discuss these findings in terms of their implications for the continuum of consistency and for the practical issue of selecting retest intervals for dependability analyses.     

The Revised Memory and Behavior Problems Checklist--Nursing Home: instrument development and measurement of burden among certified nursing assistants

Confirmatory factor analysis of the Revised Memory and Behavior Problems Checklist--Nursing Home (RMBPC) replicated the factor structure of the community-based RMBPC (L. Teri et al., 1992). The reliability of the total score was high as indexed by estimates of internal consistency (alpha = .95), test-retest reliability (r = .86), and interrater reliability between 2 interviewers (r = .88). Notably, the interrater reliability between 2 independent certified nursing assistants (CNAs) regarding residents' behavior problem frequency was more modest (r = .46), possibly reflecting the degree to which resident behaviors capture individual CNA's attention. This may have implications for the interpretation of data from the Minimum Data Set. CNAs reported moderately severe burden associated with behavior problems in 47% of residents under their care.     

Development and validation of the Delaying Gratification Inventory

Deficits in gratification delay are associated with a broad range of public health problems, such as obesity, risky sexual behavior, and substance abuse. However, 6 decades of research on the construct has progressed less quickly than might be hoped, largely because of measurement issues. Although past research has implicated 5 domains of delay behavior, involving food, physical pleasures, social interactions, money, and achievement, no published measure to date has tapped all 5 components of the content domain. Existing measures have been criticized for limitations related to efficiency, reliability, and construct validity. Using an innovative Internet-mediated approach to survey construction, we developed the 35-item 5-factor Delaying Gratification Inventory (DGI). Evidence from 4 studies and a large, diverse sample of respondents (N = 10,741) provided support for the psychometric properties of the measure. Specifically, scores on the DGI demonstrated strong internal consistency and test-retest reliability for the 35-item composite, each of the 5 domains, and a 10-item short form. The 5-factor structure fit the data well and had good measurement invariance across subgroups. Construct validity was supported by correlations with scores on closely related self-control measures, behavioral ratings, Big Five personality trait measures, and measures of adjustment and psychopathology, including those on the Minnesota Multiphasic Personality Inventory-2-Restructured Form. DGI scores also showed incremental validity in accounting for well-being and health-related variables. The present investigation holds implications for improving public health, accelerating future research on gratification delay, and facilitating survey construction research more generally by demonstrating the suitability of an Internet-mediated strategy.     

Unravelling Crime Series Patterns amongst Serial Sex Offenders: Duration,Frequency, and Environmental Consistency

Crime linkage and the investigation of behavioural consistency amongst serial offenders has been a flourishing field of research over the past decade or so, especially with respect to serial sex offenders. The emerging research in this field has often portrayed serial sex offenders as a single, distinct, and homogeneous group. Such an assumption, however, has never been empirically examined. Using a criminal career approach and a sample of 72 serial sex offenders who have committed a total of 361 sexual assaults on stranger victims, the current study aims to examine and describe subgroups of crime series patterns amongst serial sex offenders in terms of duration and frequency of offending. The level of environmental consistency display (i.e. offender's choice of crime location and characteristics of the crime site selected) across subgroups of crime series patterns is also examined. Study findings suggest the presence and heterogeneity of crime series patterns amongst serial sex offenders, advocating for the consideration of subgroups of crime series patterns when studying serial sex offenders. Moreover, the offenders' level of environmental consistency varies across the different crime series patterns identified, allowing for the identification of subgroups of offenders showing a higher or lower level of environmental consistency. Copyright © 2014 John Wiley & Sons, Ltd.     

Reliability in Content Analysis

In a recent article in this journal, Lombard, Snyder‐Duch, and Bracken (2002) surveyed 200 content analyses for their reporting of reliability tests, compared the virtues and drawbacks of five popular reliability measures, and proposed guidelines and standards for their use. Their discussion revealed that numerous misconceptions circulate in the content analysis literature regarding how these measures behave and can aid or deceive content analysts in their effort to ensure the reliability of their data. This article proposes three conditions for statistical measures to serve as indices of the reliability of data and examines the mathematical structure and the behavior of the five coefficients discussed by the authors, as well as two others. It compares common beliefs about these coefficients with what they actually do and concludes with alternative recommendations for testing reliability in content analysis and similar data‐making efforts.