Generalizability theory: A practical guide to study design,implementation, and interpretation

Generalizability Theory (GT) offers increased utility for assessment research given the ability to concurrently examine multiple sources of variance, inform both relative and absolute decision making, and determine both the consistency and generalizability of results. Despite these strengths, assessment researchers within the fields of education and psychology have been slow to adopt and utilize a GT approach. This underutilization may be due to an incomplete understanding of the conceptual underpinnings of GT, the actual steps involved in designing and implementing generalizability studies, or some combination of both issues. The goal of the current article is therefore two-fold: (a) to provide readers with the conceptual background and terminology related to the use of GT and (b) to facilitate understanding of the range of issues that need to be considered in the design, implementation, and interpretation of generalizability and dependability studies. Given the relevance of this analytic approach to applied assessment contexts, there exists a need to ensure that GT is both accessible to, and understood by, researchers in education and psychology. Important methodological and analytical considerations are presented and implications for applied use are described.     

Belief-based action prediction in preverbal infants

Successful mindreading entails both the ability to think about what others know or believe, and to use this knowledge to generate predictions about how mental states will influence behavior. While previous studies have demonstrated that young infants are sensitive to others’ mental states, there continues to be much debate concerning how to characterize early theory of mind abilities. In the current study, we asked whether 6-month-old infants appreciate the causal role that beliefs play in action. Specifically, we tested whether infants generate action predictions that are appropriate given an agent’s current belief. We exploited a novel, neural indication of action prediction: motor cortex activation as measured by sensorimotor alpha suppression, to ask whether infants would generate differential predictions depending on an agent’s belief. After first verifying our paradigm and measure with a group of adult participants, we found that when an agent had a false belief that a ball was in the box, motor activity indicated that infants predicted she would reach for the box, but when the agent had a false belief that a ball was not in the box, infants did not predict that she would act. In both cases, infants based their predictions on what the agent, rather than the infant, believed to be the case, suggesting that by 6 months of age, infants can exploit their sensitivity to other minds for action prediction.     

The origins of belief representation: Monkeys fail to automatically represent others’ beliefs

Young infants’ successful performance on false belief tasks has led several researchers to argue that there may be a core knowledge system for representing the beliefs of other agents, emerging early in human development and constraining automatic belief processing into adulthood. One way to investigate this purported core belief representation system is to examine whether non-human primates share such a system. Although non-human primates have historically performed poorly on false belief tasks that require executive function capacities, little work has explored how primates perform on more automatic measures of belief processing. To get at this issue, we modified Kovács et al. (2010)’s test of automatic belief representation to examine whether one non-human primate species—the rhesus macaque (Macaca mulatta)—is automatically influenced by another agent’s beliefs when tracking an object’s location. Monkeys saw an event in which a human agent watched an apple move back and forth between two boxes and an outcome in which one box was revealed to be empty. By occluding segments of the apple’s movement from either the monkey or the agent, we manipulated both the monkeys’ belief (true or false) and agent’s belief (true or false) about the final location of the apple. We found that monkeys looked longer at events that violated their own beliefs than at events that were consistent with their beliefs. In contrast to human infants, however, monkeys’ expectations were not influenced by another agent’s beliefs, suggesting that belief representation may be an aspect of core knowledge unique to humans.     

My mother is sensitive,but I am too tired to know: Infant sleep as a moderator of prospective relations between maternal sensitivity and infant outcomes

This study investigated the moderating role of infant sleep in the connections between maternal sensitivity and three indicators of infant functioning: attachment security, theory of mind, and executive functioning (EF). Maternal sensitivity was assessed when infants (27 girls and 36 boys) were 1 year of age. Infant sleep was assessed with actigraphy at age 2; attachment security, theory of mind, and EF were also assessed at age 2. Results indicated that maternal sensitivity was positively related to attachment security only among infants who got more sleep at night, and to conflict-EF and theory of mind only for infants who got greater proportions of their sleep during the night. These results suggest that sleep may enhance the benefits of maternal sensitivity for some aspects of infants’ functioning, providing further support for the importance of sleep maturation as a salient developmental task of infancy.     

Collaborative mother–toddler communication and theory of mind development at age 4

Focusing on social pragmatics, this longitudinal study investigated the contribution of mother–toddler collaborative communication to theory of mind (ToM) development at age 4. At age 2½, 78 toddlers (42 boys) and their mothers were observed during pretend play. At age 4, children were tested using 4 false belief understanding tasks. Both mothers and toddlers engaged in more collaborative (inform, guide/request, and support/confirm) than non-collaborative communication acts. Other-focused collaborative acts of support/confirm by mothers and toddlers predicted children's false belief understanding, even after controlling for 5 covariates. In addition, as active agents in their own ToM development, the contribution of toddlers' collaborative acts to false belief understanding was independent of their mother's. Finally, the way toddlers and their mothers co-constructed their communication mattered. Only when toddlers engaged in high levels of collaborative acts, the mothers' high levels of collaborative acts demonstrated a positive effect on children's ToM development. The applied implications of these findings are discussed.     

Commentary on Ted Ruffman’s “Belief or not belief: …”

I applaud Ruffman for cautioning us against interpreting early sensitivity to others’ beliefs as evidence for an innate theory of mind and for making room for learning. In turn, however, I caution against his claim that all infants need is to understand that people act depending on what they perceive. Instead, infants may keep experiential records (Perner & Roessler, 2010) for other people or records of what they have registered (Apperly & Butterfill, 2009), which makes it less obvious that all required knowledge can be acquired by statistical learning. As a general criticism I remonstrate with current theory of mind research on its lack of concern that we understand people as acting for reasons which goes beyond detecting lawful regularities in behaviour.     

Hypothetical identities: Explanatory problems for the explanatory argument

Recently, several philosophers have defended an explanatory argument that supposedly provides novel empirical grounds for accepting the type identity theory of phenomenal consciousness. They claim that we are justified in believing that the type identity thesis is true because it provides the best explanation for the correlations between physical properties and phenomenal properties. In this paper, I examine the actual role identities play in science and point out crucial shortcomings in the explanatory argument. I show that the supporters of the argument have failed to show that the identity thesis provides a satisfactory explanation for the correlations between physical and phenomenal properties. Hence, the explanatory argument, as it stands, does not provide new grounds for accepting the type identity theory.     

研究生入学考试写作评分的概化理论研究与多面Rasch分析

为了评价和改进硕士研究生入学考试一般能力测试的写作评分,研究者采用概化理论和多面Rasch分析对113位考生的写作样本的评分误差来源、评分信度等进行了探讨.概化理论研究显示,评分者和题目对评分准确性影响不大,以两道写作题的考试设计而言,评分者为2人即可保证评分信度在0.75以上.多面Rasch分析显示,评分者宽严度的估计值及其误差均在可接受的范围内,评分者之间在宽严度上不存在显著差异,且评分者自身在评分时总体上比较稳定.但个别评分者在特定考生特定题目上表现出特殊偏向.概化理论和多面Rasch分析丰富了写作评分研究的量化指标,证实了硕士研究生入学考试一般能力测试的写作评分具有较高的信度.     

L’intention comportementale dans les pratiques ménagères : faire soi-même son produit ménager pour moins polluer

Partly due to everyday products discharge, the presence of micropollutants in the sewage system highly concerns the public authorities and water stakeholders. Our study is focused on the measurement of the intention to substitute usual household products by a “homemade” product more eco-friendly. 304 questionnaires were completed by the inhabitants of the Strasbourg Eurometropole and were analyzed by using the theory of planned behavior (TPB). The results show that TPB explains 35 % of intention to use the “homemade” product. It seems that the respondents have a positive attitude toward this “homemade” product. It also appears that there is a strong capacity constraint of this product in terms of disinfection although this parameter is essential when choosing a household product.