Effects of validity screening items on adolescent survey data

In 2 studies, we examined the use of validity screening items in adolescent survey data. In each study, adolescent respondents were asked whether they were telling the truth and paying attention in answering survey questions. In Study 1 (N = 7,801), the prevalence rates of student risk behaviors were significantly lower after inappropriate (invalid) responders were screened out of the sample. In addition, confirmatory and multigroup factor analyses demonstrated significant differences between the factor structures of school climate scales with valid versus invalid responders. In Study 2, student perceptions of school climate were correlated with teacher perceptions in 291 schools. A bootstrap resampling procedure compared the correlations obtained with valid versus invalid responding students in each school and found that valid responders had more positive views of school conditions and produced higher correlations with teacher perceptions. These findings support the value of validity screening items in improving the quality of adolescent survey data.     

计划缺失设计——通过有意缺失让研究更高效

缺失值是社会科学研究中非常普遍的现象。全息极大似然估计和多重插补是目前处理缺失值最有效的方法。计划缺失设计利用特殊的实验设计有意产生缺失值, 再用现代的缺失值处理方法来完成统计分析, 获得无偏的统计结果。计划缺失设计可用于横断面调查减少(或增加)问卷长度和纵向调查减少测量次数, 也可用于提高测量有效性。常用的计划缺失设计有三式设计和两种方法测量。     

Construct Overlap Between Employee Engagement and Job Satisfaction: A Function of Semantic Equivalence?

The evidence for employee engagement as a distinct construct, separate from related work attitudes such as job satisfaction, has been contradictory. Despite a growing inventory of research, current statistical techniques fall short in providing additional answers beyond those previously documented through discriminant and incremental validity, among others. Understanding the semantic similarity between scales however could uncover the potential bounds of current evidence and highlight areas of further exploration. This study examined the semantic similarity between three measures of employee engagement frequently found in the research literature and a commonly used measure of job satisfaction. Examining a relatively new indicator of validity, manifest validity, semantic relatedness between items from employee engagement and job satisfaction instruments was assessed using latent semantic analysis. Semantic similarity between employee engagement and job satisfaction items predicted moderate to high amounts of variance (R ² = 25–69 %) in corresponding correlations of survey responses. Findings suggest that high correlations between certain employee engagement measures and job satisfaction scores may be a function of semantic equivalence rather than individual perceptions of separate theoretical constructs.     

多层次研究的数据聚合适当性检验:文献评价与关键问题试解

组织管理领域的多层次研究经常需要测量共享单位特性构念, 常用方法是将单位内若干个体成员的评分聚合到单位层次, 确保聚合后的分数具有充分代表性的统计前提是通过聚合适当性检验。聚合适当性检验的常用指标是组内一致性r_WG和组内信度ICC(1)、ICC(2), 但目前学界对于这两类指标何者更优、r_WG的原分布选择和数据清理、各指标的划界值等关键问题存在诸多争议。为此, 首先对国内9份管理学、心理学期刊2014年以来发表的166篇包含聚合适当性检验的论文进行内容分析, 并以Journal of Applied Psychology上的85篇论文为对比, 查明常规实践中的共性问题, 进而提出实践建议:(1)明确功能定位, 将r_WG作为聚合适当性指标, ICC(1)和ICC(2)分别作为效度、信度指标。(2)计算r_WG时审慎选择原分布, 排除组内一致性过低的组。(3)为各指标设定更加合理、有适度灵活性的划界值, 停止使用武断、粗糙的经验标准。最后, 强调研究者在模型构建和聚合决策中应加强理论考量, 避免片面依赖统计检验结果。     

Inventory of Interpersonal Problems Personality Disorder scales: operating characteristics and confirmatory factor analysis in nonclinical samples

Research involving clinical samples has demonstrated the utility of a 28-item personality disorder (PD) screening measure (Inventory of Interpersonal Problems--Personality Disorder scale [IIP-PD]) culled from the IIP in the prediction of the presence or absence of a PD (Pilkonis, Kim, Proietti, & Barkham, 1996). This article extends these diagnostic efficiency findings to nonclinical samples and presents additional data regarding the factor structure of the 28 IIP-PD items. Diagnostic efficiency statistics for the IIP-PD scale, calculated using both interview and self-report methods, support the utility of the IIP-PD scale as a screening tool for the presence or absence of a PD. High specificity estimates indicate that individuals who do not exceed Diagnostic and Statistical Manual of Mental Disorders (4th ed.; American Psychiatric Association, 1994) symptom thresholds rarely exceed the IIP-PD cutoff. Furthermore, a high negative predictive power (NPP) estimate derived using an interview-based diagnostic standard suggests that the IIP-PD scale accurately screens out individuals who do not have a PD. Finally, cross-validated confirmatory factor-analytic results involving items composing the 5 IIP PD subscales identified in previous research (Kim, Pilkonis, & Barkham, 1997) suggest that a measurement model with a single second-order factor (general PD) and 5 first-order factors (one representing each PD subscale) provided the best fit to the observed data compared to 2 other competing models.     

Detecting random responders with infrequency scales using an error-balancing threshold

Infrequency scales are becoming a popular mode of data screening, due to their availability and ease of implementation. Recent research has indicated that the interpretation and functioning of infrequency items may not be as straightforward as had previously been thought (Curran & Hauser, 2015), yet there are no empirically based guidelines for implementing cutoffs using these items. In the present study, we compared two methods of detecting random responding with infrequency items: a zero-tolerance threshold versus a threshold that balances classification error rates. The results showed that a traditional zero-tolerance approach, on average, screens data that are less indicative of careless responding than those screened by the error-balancing approach. Thus, the de facto standard of applying a “zero-tolerance” approach when screening participants with infrequency scales may be too stringent, so that meaningful responses may also be removed from analyses. Recommendations and future directions are discussed.     

Do you get what you pay for? Preventing insufficient effort responding in MTurk and student samples

Previous research has developed multiple indices to detect instances of insufficient effort responding (IER; Huang et al., 2015, 0.1037/a0038510). Although these techniques are largely successful, conflation between effort and insufficient effort leads to unnecessary data removal, thus warranting investigation into IER prevention strategies such as warning statements. For example, Brühlmann et al. (2020, 10.1016/j.metip.2020.100022) reported that close to 60% of participants were flagged for IER on at least one IER detection index. The current study used positive, negative, and no warning statements in two samples (MTurk and university participant pool) to examine their impact on two types of IER. Results do not support the use of warning statements to reduce IER covertly, overtly, or overall. Although slightly more university participants were flagged for IER for two of five IER indices, the remaining three indices did not differ between samples. This finding represents an important contribution to survey research literature by directly comparing samples on a variety of IER indices. Implications for survey research necessitate the examination of additional prevention strategies and sample differences in IER.     

Methodological considerations in developing measures of HIV risk-relevant behaviors and attitudes: An empirical illustration

Abstract

This paper describes the pilot testing of a survey designed to assess the prevalence of HIV exposure risk-relevant behaviors and associated attitudes, beliefs and knowledge in a heterogeneous, ethnically diverse, sexually active population (the US Army). The specific objectives for the pilot test were to: (1) define optimal survey question formats for behavioral and attitude items; (2) identify questions that were poorly worded or that would not contribute to the overall objectives of the survey; (3) assess the extent to which respondents could accurately follow instructions; (4) identify appropriate and culturally relevant terminology for sensitive items; (5) determine the appropriate timeframe for recall of simple and complex sexual behaviors; (6) assess the degree of respondent honesty and belief in the anonymity of the survey results.     

Validation of the Internet-Administered Quality of Life Inventory (QOLI) in Different Psychiatric Conditions

The Quality of Life Inventory (QOLI) is an established rating scale of self-perceived quality of life across 16 domains. Norms for different psychiatric conditions when rated via the Internet, responsiveness to change following treatment, and the clinical impact of importance-weighting items have yet to be examined. To investigate these unanswered questions, important for the continued and expanded use of the QOLI, we compiled archival screening and post-treatment data from 20 studies featuring Internet-delivered psychological interventions for seven different psychiatric conditions and an undergraduate sample, totalling over 4000 participants. Disorder-specific norms were indicated by between-group analyses and are reported here, item-by-item. The QOLI showed adequate responsiveness to change and construct validity. Discrepancies were found when conducting between-group analyses with and without weighted items (more significant differences when items were not weighted) on both the screening and post-treatment data, suggesting that weighting is a procedure that is likely to have an impact when analysing QOLI results. Limitations and the needs for future research are discussed.     

Identifying careless responses in survey data

When data are collected via anonymous Internet surveys, particularly under conditions of obligatory participation (such as with student samples), data quality can be a concern. However, little guidance exists in the published literature regarding techniques for detecting careless responses. Previously several potential approaches have been suggested for identifying careless respondents via indices computed from the data, yet almost no prior work has examined the relationships among these indicators or the types of data patterns identified by each. In 2 studies, we examined several methods for identifying careless responses, including (a) special items designed to detect careless response, (b) response consistency indices formed from responses to typical survey items, (c) multivariate outlier analysis, (d) response time, and (e) self-reported diligence. Results indicated that there are two distinct patterns of careless response (random and nonrandom) and that different indices are needed to identify these different response patterns. We also found that approximately 10%-12% of undergraduates completing a lengthy survey for course credit were identified as careless responders. In Study 2, we simulated data with known random response patterns to determine the efficacy of several indicators of careless response. We found that the nature of the data strongly influenced the efficacy of the indices to identify careless responses. Recommendations include using identified rather than anonymous responses, incorporating instructed response items before data collection, as well as computing consistency indices and multivariate outlier analysis to ensure high-quality data. (PsycINFO Database Record (c) 2012 APA, all rights reserved).     

The Analysis of Single-Case Research Data: Current Instructional Practices

Visual analysis is the predominant method of analysis in single-case research (SCR). However, most research suggests that agreement between visual analysts is poor, which may be due to a lack of clear guidelines and criteria for visual analysis, as well as variability in how individuals are trained. We developed a survey containing questions about the content and methods used to teach visual and statistical analysis of SCR data in verified course sequences (VCS) and distributed it via the VCS Coordinator Listserv. Thirty-seven instructors completed the survey. Results suggest that there is variability across instructors in some fundamental aspects of data analysis (e.g., number of effects required for a functional relation) but a great deal of consistency in others (e.g., emphasizing visual over statistical analysis). We discuss our results along with their implications both for teaching students to analyze SCR data and for conducting additional research on behavior-analytic training programs.

     

Identifying Variables Responsible for Data not Missing at Random

When data are not missing at random (NMAR), maximum likelihood (ML) procedure will not generate consistent parameter estimates unless the missing data mechanism is correctly modeled. Understanding NMAR mechanism in a data set would allow one to better use the ML methodology. A survey or questionnaire may contain many items; certain items may be responsible for NMAR values in other items. The paper develops statistical procedures to identify the responsible items. By comparing ML estimates (MLE), statistics are developed to test whether the MLEs are changed when excluding items. The items that cause a significant change of the MLEs are responsible for the NMAR mechanism. Normal distribution is used for obtaining the MLEs; a sandwich-type covariance matrix is used to account for distribution violations. The class of nonnormal distributions within which the procedure is valid is provided. Both saturated and structural models are considered. Effect sizes are also defined and studied. The results indicate that more missing data in a sample does not necessarily imply more significant test statistics due to smaller effect sizes. Knowing the true population means and covariances or the parameter values in structural equation models may not make things easier either. The research was supported by NSF grant DMS04-37167, the James McKeen Cattell Fund.     

追踪研究中缺失数据处理方法及应用现状分析

追踪研究中普遍存在缺失数据, 缺失数据处理方法的选择影响统计推断的精度及研究结果的有效性。首先, 阐述缺失机制及判断方法, 比较追踪研究中主要的缺失数据处理方法的特点、及实际应用中的缺失处理方法的选择和软件实现。其次, 对国内心理学中92篇追踪研究文献进行分析, 发现有59篇(64.13%)报告不同程度缺失, 其中仅39篇报告了处理方法且均为删除法。未来研究应深入探讨现有缺失数据处理方法的有效性, 进一步规范应用研究中缺失数据的处理。     

Psychometric Support for an Abbreviated Version of the Behavior Rating Inventory of Executive Function (BRIEF) Parent Form

The objectives of this study were to systematically develop and evaluate the psychometric properties of an abbreviated version of the Behavior Rating Inventory of Executive Function (BRIEF) Parent Report; a questionnaire widely used by pediatric neuropsychologists. A total of 24 items from the original BRIEF Parent Form were selected for the short-form, which was then evaluated in three complementary samples, according to six a priori psychometric criteria. The short-form generally demonstrated appropriate psychometric qualities, with convincing evidence for the reliability and validity of the three composite indices: Behavioral Regulation, Metacognition, and the Global Executive Composite. Potential clinical applications include screening at-risk children in medical clinics to facilitate appropriate referrals for further psychological consultation. In research settings, the short-form can be easily integrated into studies involving mass collection of data (e.g., large-scale epidemiological research), facilitating advancements in the scientific understanding of neuropsychological morbidity in medically involved populations.     

Incidence of Data Duplications in a Randomly Selected Pool of Life Science Publications

Since the solution to many public health problems depends on research, it is critical for the progress and well-being for the patients that we can trust the scientific literature. Misconduct and poor laboratory practice in science threatens the scientific progress, leads to loss of productivity and increased healthcare costs, and endangers lives of patients. Data duplication may represent one of challenges related to these problems. In order to estimate the frequency of data duplication in life science literature, a systematic screen through 120 original scientific articles published in three different cancer related journals [journal impact factor (IF) <5, 5–10 and >20] was completed. The study revealed a surprisingly high proportion of articles containing data duplication. For the IF < 5 and IF > 20 journals, 25 % of the articles were found to contain data duplications. The IF 5–10 journal showed a comparable proportion (22.5 %). The proportion of articles containing duplicated data was comparable between the three journals and no significant correlation to journal IF was found. The editorial offices representing the journals included in this study and the individual authors of the detected articles were contacted to clarify the individual cases. The editorial offices did not reply and only 1 out of 29 cases were apparently clarified by the authors, although no supporting data was supplied. This study questions the reliability of life science literature, it illustrates that data duplications are widespread and independent of journal impact factor and call for a reform of the current peer review and retraction process of scientific publishing.     

Identification of Developmentally Appropriate Screening Items for Disruptive Behavior Problems in Preschoolers

Screening preschool-aged children for disruptive behavior disorders is a key step in early intervention. The study goal was to identify screening items with excellent measurement properties at sub-clinical to clinical levels of disruptive behavior problems within the developmental context of preschool-aged children. Parents/caregivers of preschool-aged children (N?=?900) were recruited from four pediatric primary care settings. Participants (mean age?=?31, SD?=?8) were predominantly female (87 %), either white (55 %) or African-American (42 %), and biological parents (88 %) of the target children. In this cross-sectional survey, participants completed a sociodemographic questionnaire and two parent-report behavioral rating scales: the PSC-17 and the BPI. Item response theory analyses provided item parameter estimates and information functions for 18 externalizing subscale items, revealing their quality of measurement along the continuum of disruptive behaviors in preschool-aged children. Of 18 investigated items, 5 items measured only low levels of disruptive behaviors among preschool-aged children. The remaining 13 items measured sub-clinical to clinical levels of disruptive behavior problems (i.e., >1.5 SD); however, 5 of these items offered less information, suggesting unreliable measurement. The remaining 8 items had high discrimination and difficulty parameters, offering considerable measurement information at sub-clinical to clinical levels of disruptive behavior problems. Behaviors measured by the 8 selected parent-report items were consistent with those identified in recent efforts to distinguish developmentally typical misbehaviors from clinically concerning behaviors among preschool-aged children. These items may have clinical utility in screening young children for disruptive behavior disorders.     

A note on the use of directional statistics in weighted euclidean distances multidimensional scaling models

The weighted euclidean distances model in multidimensional scaling (WMDS) represents individual differences as dimension saliences which can be interpreted as the orientations of vectors in a subject space. It has recently been suggested that the statistics of directions would be appropriate for carrying out tests of location with such data. The nature of the directional representation in WMDS is reviewed and it is argued that since dimension saliences are almost always positive, the directional representations will usually be confined to the positive orthant. Conventional statistical techniques are appropriate to angular representations of the individual differences which will yield angles in the interval (0, 90) so long as dimension saliences are nonnegative, a restriction which can be imposed. Ordinary statistical methods are also appropriate with several linear indices which can be derived from WMDS results. Directional statistics may be applied more fruitfully to vector representations of preferences.     

A Procedure for Dimensionality Analyses of Response Data from Various Test Designs

In some popular test designs (including computerized adaptive testing and multistage testing), many item pairs are not administered to any test takers, which may result in some complications during dimensionality analyses. In this paper, a modified DETECT index is proposed in order to perform dimensionality analyses for response data from such designs. It is proven in this paper that under certain conditions, the modified DETECT can successfully find the dimensionality-based partition of items. Furthermore, the modified DETECT index is decomposed into two parts, which can serve as indices of the reliability of results from the DETECT procedure when response data are judged to be multidimensional. A simulation study shows that the modified DETECT can successfully recover the dimensional structure of response data under reasonable specifications. Finally, the modified DETECT procedure is applied to real response data from two-stage tests to demonstrate how to utilize these indices and interpret their values in dimensionality analyses.