Agreement between Parents and Children on Ratings of Post-Concussive Symptoms Following Mild Traumatic Brain Injury

The level of parent-child agreement on post-concussive symptoms (PCS) was examined in children following mild traumatic brain injuries (TBI). As part of a larger longitudinal study, 186 children with mild TBI and 99 with orthopedic injuries (OI), from 8 to 15 years of age, were recruited prospectively. Parents and children completed the PCS Interview (PCS-I) and the Health and Behavior Inventory (HBI) at 2 weeks, 1 month, 3 months, and 12 months postinjury. Item-level correlations between child and parent ratings on both measures of PCS were significant but modest in both groups. Parent-child correlations for composite scales on the HBI and the total score on the PCS-I were significant in both groups, but somewhat higher in the OI group than in the mild TBI group. Mean symptom ratings tended to be significantly higher for children as compared to parents, especially for somatic symptoms. Parents and children display modest agreement when reporting PCS; their ratings correlate significantly, but children report higher mean levels of symptoms than parents.     

Back to basics: Percentage agreement measures are adequate, but there are easier ways

Percentage agreement measures of interobserver agreement or "reliability" have traditionally been used to summarize observer agreement from studies using interval recording, time-sampling, and trial-scoring data collection procedures. Recent articles disagree on whether to continue using these percentage agreement measures, and on which ones to use, and what to do about chance agreements if their use is continued. Much of the disagreement derives from the need to be reasonably certain we do not accept as evidence of true interobserver agreement those agreement levels which are substantially probable as a result of chance observer agreement. The various percentage agreement measures are shown to be adequate to this task, but easier ways are discussed. Tables are given to permit checking to see if obtained disagreements are unlikely due to chance. Particularly important is the discovery of a simple rule that, when met, makes the tables unnecessary. If reliability checks using 50 or more observation occasions produce 10% or fewer disagreements, for behavior rates from 10% through 90%, the agreement achieved is quite improbably the result of chance agreement.     

Interobserver Agreement in Behavioral Research: Importance and Calculation

Behavioral researchers have developed a sophisticated methodology to evaluate behavioral change which is dependent upon accurate measurement of behavior. Direct observation of behavior has traditionally been the mainstay of behavioral measurement. Consequently, researchers must attend to the psychometric properties, such as interobserver agreement, of observational measures to ensure reliable and valid measurement. Of the many indices of interobserver agreement, percentage of agreement is the most popular. Its use persists despite repeated admonitions and empirical evidence indicating that it is not the most psychometrically sound statistic to determine interobserver agreement due to its inability to take chance into account. Cohen's (1960) kappa has long been proposed as the more psychometrically sound statistic for assessing interobserver agreement. Kappa is described and computational methods are presented.     

Reliability in the context of the experiment: A commentary on two articles by Birkimer and Brown

Two sources of variability must each be considered when examining change in level between two sets of data obtained by human observers; namely, variance within data sets (phases) and variability attributed to each data point (reliability). Birkimer and Brown (1979a, 1979b) have suggested that both chance levels and disagreement bands be considered in examining observer reliability and have made both methods more accessible to researchers. By clarifying and extending Birkimer and Brown's papers, a system is developed using observer agreement to determine the data point variability and thus to check the adequacy of obtained data within the experimental context.     

Developmental trends in children’s typicality judgments

Category typicality norms from 12 natural language categories are presented for kindergarten, third-grade, sixth-grade, and college students. Subjects first selected examples of familiar word concepts and rated them on a 3-point scale in terms of category typicality. Age differences in the percentage of items included as category members were found primarily for the less typical items, with inclusion rates varying as a function of both age and typicality level. The absolute level of typicality judgments increased with age, although correlations between the children’s and college students’ ratings were generally significant for all three children’s groups, with average correlations increasing somewhat with age. It was suggested that the rating data would be useful to developmental investigators interested in children’s processing of category information.     

Youth-parent consistencies on ratings of difficulties and prosocial behavior: exploration of an at-risk sample

Low informant agreement is of particular concern in clinical settings, since diagnoses and interventions are largely based on reports from parents and/or youths. This study explores youth-parent consistencies in an at-risk sample of youths (n = 42), several from immigrant and refugee families, all with stressful background experiences. Data from a community sample (n = 165) was used to contrast the clinical results. Results showed that correlations between clinical youths and parents were within the same range as those in the community sample. Consistencies concerning absolute level of behavior were higher in the clinical sample. However, interesting differences in informant consistencies were found on difficulties and resources between the two samples. This study adds to the picture that there is no simple relationship between severity of behavior deviance and informant agreement. Asking about behavioral resources as well as difficulties may provide valuable additional information regarding differences in the perspectives of at-risk youths and parents.     

Crossed categorization effects on the formation of illusory correlations

This study investigates how stereotypes are formed and whether stereotype formation is reduced by the prevalence of multiple categorizations. Illusory correlations between the desirability of behaviours and two dimensions of social categorization, both containing a majority and a minority category, were assessed in single categorization and crossed categorization conditions. In the single categorization conditions, the usual illusory correlation in favour of the majority category was obtained. In the crossed condition, the combination of the two majority categories was positively discriminated from the remaining three combinations, while no differences were found among the latter. A source‐monitoring analysis of assignment frequencies replicated earlier findings that illusory correlations are due to an evaluative guessing bias, rather than to enhanced memory for individual instances of behaviour. The results show inconsistencies with a distinctiveness‐based and a social categorization account of illusory correlations, but they can be explained in terms of information loss. Copyright © 2001 John Wiley & Sons, Ltd.     

Percentage agreement and phi: A conversion table

STUDIES IN APPLIED BEHAVIOR ANALYSIS HAVE USED TWO EXPRESSIONS OF RELIABILITY FOR HUMAN OBSERVATIONS: percentage agreement (including percentage occurrence and percentage nonoccurrence agreement) and correlational techniques (including the phi coefficient). The formal relationship between these two expressions is demonstrated, and a table for converting percentage agreement to phi, or vice-versa, is presented. It is suggested that both expressions be reported in order to communicate reliability unambiguously and to facilitate comparison of the reliabilities from different studies.     

Inferring correlations: From exemplars to categories

Research and theorizing suggest a processing advantage of category-level correlations over exemplar-level correlations. That research has also shown that category-level correlations serve as a proxy for inferring exemplar-level correlations. For example, an individual may learn that the demand for a product category, like cheese, in one store predicts the demand for this category in another. The individual could then draw the unwarranted conclusion that the demand for an exemplar, like cheddar, would also predict the demand for this exemplar in the other store. This notion is supported by previous experiments demonstrating that the subjective exemplar-level correlation follows the implication of the category-level correlation. However, in virtually all previous experiments suggesting a processing advantage for category-level over exemplar-level correlations, the stimulus correlation at the category level was substantial, whereas the correlation at the exemplar level was weak. Here, we tested the hypothesis that individuals process the level that is most informative, either the exemplar or the category level. We presented participants with a zero correlation at the category level, but varied the correlation at the exemplar level. Participants presented with a zero correlation across exemplar products correctly reproduced a zero correlation across product categories. When presented with a substantial correlation at the exemplar level, however, they erroneously reproduced a similar correlation at the category level. These findings therefore imply that there is no general processing advantage for correlations at higher aggregation levels. Instead, individuals seemingly attend to the level that holds the most regular information. Findings are discussed regarding the role of covariation strength in correlation detection and use.     

Teachers' and mothers' assessment of children's behaviors from kindergarten to grade two: Stability and change within and across informants

Consistency between teachers' and parents' ratings of 229 children's prosocial and problem behaviors was investigated from kindergarten through grade two. Intrarater stability was also analyzed. Results indicated moderate but decreasing interrater agreement from kindergarten to grade two for boys on externalizing behaviors. Interrater agreement for the same behavior dimension for girls was moderate and stable, whereas it was low on internalizing behavior problems and prosocial behaviors, for both genders throughout the 3-year period. Moreover, difference between teachers' and parents' ratings of externalizing behaviors increased from kindergarten to grade two. Decreasing level of agreement through time appeared to be related to a reduction of teacher-rated externalizing behaviors not paralleled by a reduction in parent ratings. Analyses were also performed at the item level of the externalization scale to identify which specific behaviors were responsible for the increasing divergence in interrater agreement from kindergarten through grade two. Alternative interpretations were considered in order to account for present and past results. Implications for screening and assessment purposes were discussed.This research was supported by grants from the Social Sciences and Humanities Research Council of Canada and from Québec's Health Research Fund (FRSQ). We wish to thank the authorities and directors of the schools from the Val d'Or School Board as well as the kindergarten, first and second grade teachers, children, and parents for their collaboration. We also thank Lyse Desmarais-Gervais, Daniel Corriveau, Jean Desrosiers and Hélène Boileau for their participation in the data collection or analysis.     

Picture Naming and Lexical Access in Italian Children and Adults

Italian-speaking adults and 5- to 6-year-old children were compared in a timed picture-naming task, measuring latency as well as name agreement, in relation to frequency, age of acquisition (AoA), picture complexity, word length, word complexity, semantic category, phonological characteristics, and grammatical gender. Although children were slower and less accurate than adults, correlations of dependent variables with lexical predictors were similar in both groups. However, word complexity had effects on adults that were not seen in children, and grammatical gender had effects on children that were not seen in adults. Adult ratings of AoA had strong effects on both groups, but an objective measure of AoA only affected children. Adult and child reaction times were also differentially affected by semantic category (especially animal names). Results have implications for developmental and cross-linguistic studies of lexical access, in and out of context.     

A multi-informant study of the influence of targets’ and perceivers’ social desirability on self-other agreement in ratings of the HEXACO personality dimensions

In the current research we evaluated the impact of socially desirable responding on the strength of self-other agreement in personality judgments. We used a multi-informant design with self- and other-reports of HEXACO personality dimensions as well as self-reported and other-reported social desirability. Analyses were based on 305 primary participants and 709 informants. As shown by the results, neither self-other agreement nor other-other agreement regarding the personality dimensions was affected by social desirability. Furthermore, significant correlations between self-rated and other-rated social desirability emerged. The results are discussed as further evidence that most social desirability scales seem to measure substantive personality traits instead of response styles.     

Single-attribute implicit association tests (SA-IAT) for the assessment of unipolar constructs. The case of sociosexuality

A major problem with Implicit Association Tests (IATs) is that they require bipolar attributes (e.g., good-bad). Thus, IAT effects for an attribute category can be interpreted only relative to an opposite category. Problems arise if there is no clear opposite category; in this case, a neutral category can be used, although it induces systematic error variance and thus reduces validity. The present study suggests that this problem can be solved using single-attribute IATs (SA-IATs). Sociosexuality (the tendency to engage in uncommitted sex) was expected to be related at the implicit level to stronger stranger-sex associations relative to partner-sex associations. An IAT was constructed that used conversation as a neutral attribute; it showed satisfactory reliability but only low correlations with explicit sociosexuality. An alternative SA-IAT with sex as the only attribute showed a similar reliability but higher correlations with explicit sociosexuality.     

汉语言语产生中先行关联词的隐含数倾向对代词选择的影响

使用句子补全任务, 通过3个实验, 考察了汉语言语产生中先行关联词的隐含数倾向对代词选择的影响, 证实在汉语言语产生中也存在着数一致和数吸引现象。实验1和实验2分别考察了汉语双字动词和双字名词的隐含数倾向对代词选择的影响。实验3进一步探查了在汉语言语产生中的数吸引现象。结果发现, 在汉语言语产生中, 不仅存在着“主语-动词数一致”和“名词-代词数一致”, 还存在着对称的数吸引。研究结果部分支持了特征渗透模型和激活框架理论。     

类别使用的特征诊断效应

运用预备实验评定过的鱼轮廓图,考察了类别判断和特征预测这两种类别使用方式下的特征诊断效应。被试是48名大学生,学习材料是两类鱼的类别成员,在测验阶段,实验1判断新项目类别标签,实验2预测新项目的缺失特征。结果表明,类别判断条件下,单个特征诊断力加强和诊断性特征数量增加都对类别判断有促进作用;特征预测条件下,只有单个特征诊断力加强可以促进特征预测,诊断性特征数量增加无助于特征预测     

The effects of instructions and calculation procedures on observers' accuracy, agreement, and calculation correctness

Although the quality of observational data is generally evaluated by observer agreement, measures of both observer agreement and accuracy were available in the present study. Videotapes with a criterion protocol were coded by 16 observers. All observers calculated agreement scores both on their own and their partner's data and on a contrived data set misrepresented as data collected by other observers. Compared with agreement scores calculated by the experimenter, observers erroneously inflated their own agreement scores and deflated the agreement scores on the contrived data. Half of the observers (n = 8) had been given instructions emphasizing the importance of accuracy during observation while the other half had been given instructions emphasizing interobserver agreement. Accuracy exceeded agreement for the former group, whereas agreement exceeded accuracy for the latter group. The implications are that agreement should be calculated by the experimenter and that the accuracy-agreement relationship can be altered by differential observer instructions.     

评分者内部一致性的研究和应用

在行为科学的研究和实践中,研究者常常需要将个体层次的评价,整合到群体层次的评价,对于这种自下而上整合模式的一致性问题,国内一些学者常常错误地使用评分者内部信度作为评分者内部一致性的指标。评分者内部一致性和评分者内部信度不仅在理论基础上存在差异,而且在实践中也存在前者很高(或很低),而后者却很低(或很高)的不一致情况。文章阐述了学术界对评分一致性这一问题的提出、争论和取得一致观点的发展脉络,以期学者们对这一问题能够有深入的思索,避免在今后的研究中出现类似的错误。     

Category inference as a function of correlational structure, category discriminability, and number of available cues

Two experiments investigated category inference when categories were composed of correlated or uncorrelated dimensions and the categories overlapped minimally or moderately. When the categories minimally overlapped, the dimensions were strongly correlated with the category label. Following a classification learning phase, subsequent transfer required the selection of either a category label or a feature when one, two, or three features were missing. Experiments 1 and 2 differed primarily in the number of learning blocks prior to transfer. In each experiment, the inference of the category label or category feature was influenced by both dimensional and category correlations, as well as their interaction. The number of cues available at test impacted performance more when the dimensional correlations were zero and category overlap was high. However, a minimal number of cues were sufficient to produce high levels of inference when the dimensions were highly correlated; additional cues had a positive but reduced impact, even when overlap was high. Subjects were generally more accurate in inferring the category label than a category feature regardless of dimensional correlation, category overlap, or number of cues available at test. Whether the category label functioned as a special feature or not was critically dependent upon these embedded correlations, with feature inference driven more strongly by dimensional correlations.     

Assessment of incarcerated juvenile delinquents: Agreement across raters and approaches to psychopathology

The purpose of the current study was to examine the agreement across three informants (youth, teacher, caretaker) when rating behaviors of incarcerated juvenile delinquents. Furthermore, the agreement between a dimensional approach (Child Behavior Checklist) and a categorical approach (DSM-III-R) of assessing behavior was examined. Fifty-two delinquent youth were assessed with a structured interview. In addition, the youth, teacher, and caretaker completed the Child Behavior Checklist (CBCL). The results indicated significant correlations across the teacher and caretaker for externalizing types of problems. The youth report form of the CBCL did not correspond to either adult informant. However, the ratings by the teacher and caretaker were related to externalizing types of problems, primarily hyperactivity/inattention, derived from the structured interview with the delinquent. The results suggest that, in the psychological assessment of incarcerated juvenile delinquents, agreement across informants is dependent on multiple factors, including type of behavior assessed and the approach utilized in assessment.This work was supported, in part, by the Juvenile Justice Coordinating Council of Georgia, the Office of Juvenile Justice and Delinquency Prevention, and the University of Georgia's Institute for Behavioral Research.     

Cross-cultural comparison of personality: Hungarian children and english children

1150 Hungarian boys and 1035 girls were given the Junior Eysenck Personality Questionnaire translated into Hungarian. Product-moment correlations were factored by principal components methods, rotated by Varimax and then obliquely by Promax. Thirteen extra items, added for substitution if necessary, were included in this analysis. Very close agreement between the responses of British and Hungarian children were found for the Lie or Social Desirability scale, good agreement for Extraversion but less satisfactory agreement for Neuroticism and Psychoticism items. It is argued that the strength of the Lie score could well have interfered with the true responses for the other dimensions. Means and standard deviations for British and Hungarian children are given on scales comprised of items both groups have in common on their respective scoring keys, this being the only way the means can be compared.