What do these clinical trial results mean? How product efficacy judgments are affected by data partitioning,framing, and quantification

Organizations often present data related to clinical trials, and other product efficacy information, in partitioned or aggregated formats, as successes or failures, and as frequencies or percentages. We examine how such different data presentation formats might interact to influence product efficacy judgments. The results of five experiments indicate that partitioned (vs. aggregated) frequency data affect judgments regarding perceived product efficacy and these effects are moderated by data frames (success vs. failure) and quantification (frequencies vs. percentages). Specifically, success-framed, partitioned, frequency data enhance product efficacy judgments and choice, while failure-framed, partitioned, frequency data have the opposite effects. However, these effects get attenuated when data are aggregated or presented as percentages.     

Optimizing the validity of personality assessments: The importance of aggregation and item content

The predictive validity of a psychological measure can be improved by minimizing measurement errors through increases in the length of the assessment (aggregation) and, for an assessment of finite length, by making use of objective strategies for choosing from all available component measures. Two prominent considerations in selecting individual measures to be aggregated involve standards of (a) item content (construct approach) and (b) item/criterion association (empirical approach). Personality trait scales of different lengths were assembled for this study in order to represent features of the construct and empirical methods of selection. It was observed that (a) although reliability and validity generally increased with test length, aggregation beyond a certain point can fail to be expedient; and (b) although the prediction performance of empirically derived measures initially surpassed that of construct based assessments, the superiority of the empirical scales did not generalize to trait criteria that were not used as a basis for item selection. The data are interpreted as providing support for a theory-based program of test development where substantive considerations involving item content play a major role. The findings are also viewed as encouragement for conventional conceptualizations about organized dimensions of behavior.     

Aggregation and beyond: Some basic issues on the prediction of behavior

Failure to appreciate the role that aggregation plays in increasing reliability and validity and in establishing the range of generalization of findings has resulted in misunderstandings about the stability of behavior across time and situations, and in the conduct of experiments that produce results that tend to be neither generalizable nor replicable. Appropriate aggregation can reduce error variance associated with the unrepresentativeness of individual stimuli, situations, occasions, judges, items of behavior, and subjects. Inappropriate aggregation can result not only in a loss of information but also in a reduction in reliability as well as validity. Different approaches to prediction with single items of behavior are discussed, and it is concluded that single items tend to be too unreliable and too narrow in scope to measure broad dispositions such as traits. A major emphasis is that behavior is often so highly situationally specific that unless this is taken into account by procedures such as aggregation over situations and/or occasions, or by the investigation of events that are so highly ego-involving that experimental effects dominate situation-ally unique effects, results will tend to be unreplicable or ungeneralizable, no matter what their level of statistical significance.     

Aggregation,Validation, and Generalization of Qualitative Data - Methodological and Practical Research Strategies Illustrated by the Research Process of an empirically Based Typology

The article deals with the question of how aggregated data which allow for generalizable insights can be generated from single-case based qualitative investigations. Thereby, two central challenges of qualitative social research are outlined: First, researchers must ensure that the single-case data can be aggregated and condensed so that new collective structures can be detected. Second, they must apply methods and practices to allow for the generalization of the results beyond the specific study. In the following, we demonstrate how and under what conditions these challenges can be addressed in research practice. To this end, the research process of the construction of an empirically based typology is described. A qualitative study, conducted within the framework of the Luxembourg Youth Report, is used to illustrate this process. Specifically, strategies are presented which increase the likelihood of generalizability or transferability of the results, while also highlighting their limitations.     

多层次研究的数据聚合适当性检验:文献评价与关键问题试解

组织管理领域的多层次研究经常需要测量共享单位特性构念, 常用方法是将单位内若干个体成员的评分聚合到单位层次, 确保聚合后的分数具有充分代表性的统计前提是通过聚合适当性检验。聚合适当性检验的常用指标是组内一致性r_WG和组内信度ICC(1)、ICC(2), 但目前学界对于这两类指标何者更优、r_WG的原分布选择和数据清理、各指标的划界值等关键问题存在诸多争议。为此, 首先对国内9份管理学、心理学期刊2014年以来发表的166篇包含聚合适当性检验的论文进行内容分析, 并以Journal of Applied Psychology上的85篇论文为对比, 查明常规实践中的共性问题, 进而提出实践建议:(1)明确功能定位, 将r_WG作为聚合适当性指标, ICC(1)和ICC(2)分别作为效度、信度指标。(2)计算r_WG时审慎选择原分布, 排除组内一致性过低的组。(3)为各指标设定更加合理、有适度灵活性的划界值, 停止使用武断、粗糙的经验标准。最后, 强调研究者在模型构建和聚合决策中应加强理论考量, 避免片面依赖统计检验结果。     

Ignoring Clustering in Confirmatory Factor Analysis: Some Consequences for Model Fit and Standardized Parameter Estimates

In many situations, researchers collect multilevel (clustered or nested) data yet analyze the data either ignoring the clustering (disaggregation) or averaging the micro-level units within each cluster and analyzing the aggregated data at the macro level (aggregation). In this study we investigate the effects of ignoring the nested nature of data in confirmatory factor analysis (CFA). The bias incurred by ignoring clustering is examined in terms of model fit and standardized parameter estimates, which are usually of interest to researchers who use CFA. We find that the disaggregation approach increases model misfit, especially when the intraclass correlation (ICC) is high, whereas the aggregation approach results in accurate detection of model misfit in the macro level. Standardized parameter estimates from the disaggregation and aggregation approaches are deviated toward the values of the macro- and micro-level standardized parameter estimates, respectively. The degree of deviation depends on ICC and cluster size, particularly for the aggregation method. The standard errors of standardized parameter estimates from the disaggregation approach depend on the macro-level item communalities. Those from the aggregation approach underestimate the standard errors in multilevel CFA (MCFA), especially when ICC is low. Thus, we conclude that MCFA or an alternative approach should be used if possible.     

机器学习在解决过拟合现象中的作用

过拟合现象是心理学走向预测科学的重要阻碍。文章综述了机器学习在解决过拟合现象中的价值和实现途径：（1）介绍了过拟合的两种表现形式和现状;（2）分析过拟合的根因,即“高解释力≠高预测力”;（3）厘清机器学习的建模逻辑与核心技术在解决过拟合中的作用;（4）利用样例数据和代码说明机器学习统计思想在模型拟合中的具体应用过程。文章指出心理学应从解决实际问题的角度出发,借鉴机器学习的分析思想,避免过拟合,进而提供更准确更稳定的结论和预测模型。     

The consistency of presented personality: Additional evidence from aggregation

The present study investigated people's variability across situations by getting ratings of 66 subjects on 14 bipolar dimensions from at least eight interactants, chosen for their diversity. The intercorrelation of single ratings yielded a mean coefficient of .221. The correlation of single ratings with the aggregate of the other ratings for a dimension resulted in a mean coefficient of .388. The correlation of two sets of aggregated ratings gave a mean coefficient of .550, or .710 with application of the Spearman-Brown correction. Finally, computation of Cronbach's alpha gave a mean coefficient of .735. The results provide a further demonstration of the coherence that can be revealed by aggregation. Correlations of aggregated ratings on each of the 14 dimensions with extraversion ranged up to .668, and correlations with neuroticism ranged up to .410. The study suggests that there is a dispositionality in the characteristics people display, and that the emphasis on variability (e.g., Mischel, 1968; Mischel & Peake, 1982) should be tempered.     

A model for stochastic drift in memory strength to account for judgments of learning

Previous research has shown that judgments of learning (JOLs) made immediately after encoding have a low correlation with actual cued-recall performance, whereas the correlation is high for delayed judgments. In this article, the authors propose a formal theory describing the stochastic drift of memory strength over the retention interval to account for the delayed-JOL effect. This is done by first decomposing the aggregated memory strength into exponential functions with slow and fast memory traces. The mean aggregated memory strength shows power-function forgetting curves. The drift of the memory strength is large for immediate JOLs (causing a low predictability) and weak for delayed JOLs (causing a high predictability). Consistent with empirical data, the model makes a novel prediction of JOL asymmetry, or that immediate weak JOLs are more predictive of future performance than are immediate strong JOLs. The JOL distributions for immediate and delayed JOLs are also accounted for.     

A Research Tool to Increase Attention to Experimental Materials: Manipulating Presentation Format

Experimental approaches require researchers in a variety of applied business disciplines to design experimental materials. This research is often busy and difficult to secure high-level professionals; thus, researchers must ensure sufficient attention to experimental materials to provide usable data. The current study suggests an experimental design feature to encourage participants to thoroughly process information contained in experimental materials. Based on Reynolds' (1992) selective attention strategy (SAS) model, variations in presentation format were expected to influence the relative salience of information presented. Two aspects of presentation format were manipulated to increase salience: display (paragraph vs. bullet point) and typeface (standard vs. bold underlined). Results indicate display improved ability to retrieve information, but typeface did not. Implications for the design of experimental materials are discussed.     

Aggregation, acquiescence, and the assessment of trait affectivity

General affect ratings are subject to problems that lessen their validity. Some critics therefore have argued that aggregated on-line ratings provide better measures of trait affectivity. However, aggregated mood ratings suffer from two significant problems. First, correlations among same-valenced affects are substantially higher, indicating poorer discriminant validity. Second, cross-valence correlations are much weaker, reflecting a virtual absence of bipolarity. We show how both of these effects are due to an inflated acquiescence component in aggregated data. We also demonstrate that (a) these effects are a direct result of the aggregation process itself and (b) controlling for acquiescence counteracts these effects. These results indicate that the validity of aggregated mood ratings is lessened by an augmented acquiescence component.     

Actigraph data are reliable, with functional reliability increasing with aggregation

Motion sensor devices such as actigraphs are increasingly used in studies that seek to obtain an objective assessment of activity level. They have many advantages, and are useful additions to research in fields such as sleep assessment, drug efficacy, behavior genetics, and obesity. However, questions still remain over the reliability of data collected using actigraphic assessment. We aimed to apply generalizability theory to actigraph data collected on a large, general-population sample in middle childhood, during 8 cognitive tasks across two body loci, and to examine reliability coefficients on actigraph data aggregated across different numbers of tasks and different numbers of attachment loci. Our analyses show that aggregation greatly increases actigraph data reliability, with reliability coefficients on data collected at one body locus during 1 task (.29) being much lower than that aggregated across data collected on two body loci and during 8 tasks (.66). Further increases in reliability coefficients by aggregating across four loci and 12 tasks were estimated to be modest in prospective analyses, indicating an optimum trade-off between data collection and reliability estimates. We also examined possible instrumental effects on actigraph data and found these to be nonsignificant, further supporting the reliability and validity of actigraph data as a method of activity level assessment.     

Levels of aggregation and the generalized matching law.

Concurrent operant behavior is often described using molar response tendencies measured over many sessions. In contrast, other models relate the allocation of behavior to momentary or molecular processes that operate over shorter periods. In this article, a formal model of the statistical properties of molecular behavioral matching is developed. Molar matching is the result when many individual bouts of responding are aggregated. However, behavioral allocation at the molecular level will differ from the molar result. The model indicates that aggregation effects should be controlled in analyses of matching and that the comparison of molecular theories of concurrent operant behavior with molecular models of matching performance affords ideographic analyses of choice behavior.     

Judgment research: Implications for clinical practice and testimony in court

When treating a client or testifying in court, clinicians should be aware of how they make judgments and when their judgments are likely to be correct and when they are likely to be wrong. Research on the validity of judgments is reviewed along with research on the cognitive processes of clinicians. Some of the results are surprising. For example, recent research indicates that clinicians might be able to make moderately valid long-term predictions of violence. Finally, recommendations are made for improving clinical judgment, and comments are made about the appropriate use of statistical prediction rules and automated assessment test reports.     

Clinical versus mechanical prediction: a meta-analysis

The process of making judgments and decisions requires a method for combining data. To compare the accuracy of clinical and mechanical (formal, statistical) data-combination techniques, we performed a meta-analysis on studies of human health and behavior. On average, mechanical-prediction techniques were about 10% more accurate than clinical predictions. Depending on the specific analysis, mechanical prediction substantially outperformed clinical prediction in 33%-47% of studies examined. Although clinical predictions were often as accurate as mechanical predictions, in only a few studies (6%-16%) were they substantially more accurate. Superiority for mechanical-prediction techniques was consistent, regardless of the judgment task, type of judges, judges' amounts of experience, or the types of data being combined. Clinical predictions performed relatively less well when predictors included clinical interview data. These data indicate that mechanical predictions of human behaviors are equal or superior to clinical prediction methods for a wide range of circumstances.     

Lay theories and intergroup relations: A social-developmental perspective

Abundant research suggests that the study of lay theories helps to explain intergroup relations. Building on this work as well as “interactionist” theories for understanding social behaviour from developmental and social psychology, we propose an integrative social-developmental perspective examining how lay perceivers' characteristics (e.g., age, race, psychological motivations) interact with the environments in which they are nested to impact lay theory use over time and during life transitions. Using this perspective to guide our investigation of the Protestant work ethic (PWE) and colourblind theory, we show that a single lay theory can have a socially tolerant or intolerant meaning. We review work with US children and adults (Asians, Blacks, Latinos, Whites) as well as research with Colombian children and adults (Mestizos), showing similarities and differences in perceivers' uses of PWE and the colourblind theory. Even when both meanings are prevalent in a given culture, they are not necessarily equally emphasised in all environments or for all people living in those environments, nor are they responded to in the same way by all people. We discuss the implications of these results for theorising on lay theories and offer directions for future work in this area.     

Statistics and ethics in medical research

Ethical conduct is an essential component in research, especially in medical research. Statistical methods for design and analysis are powerful research tools if used properly. Abuse of these principles and methods are just as unethical as other laboratory or clinical misconduct. Inadequate research design can produce worthless results and thus wastes effort and valuable resources. For clinical research, patient resources are wasted. Inappropriate analysis of data can also produce misleading results and conclusions. For clinical research, inferior therapy might be given to patients as a consequence. These ethical concerns can have implications for and affect the individuals responsible for the statistical design and analysis. Examples are provided which illustrate some possible abuses and inappropriate pressures.     

Evaluating Single-Subject Treatment Research: Lessons Learned from the Aphasia Literature

The mandate for evidence-based practice has prompted careful consideration of the weight of the scientific evidence regarding the therapeutic value of various clinical treatments. In the field of aphasia, a large number of single-subject research studies have been conducted, providing clinical outcome data that are potentially useful for clinicians and researchers; however, it has been difficult to discern the relative potency of these treatments in a standardized manner. In this paper we describe an approach to quantify treatment outcomes for single-subject research studies using effect sizes. These values provide a means to compare treatment outcomes within and between individuals, as well as to compare the relative strength of various treatments. Effect sizes also can be aggregated in order to conduct meta-analyses of specific treatment approaches. Consideration is given to optimizing research designs and providing adequate data so that the value of treatment research is maximized.