Inferring action structure and causal relationships in continuous sequences of human action

In the real world, causal variables do not come pre-identified or occur in isolation, but instead are embedded within a continuous temporal stream of events. A challenge faced by both human learners and machine learning algorithms is identifying subsequences that correspond to the appropriate variables for causal inference. A specific instance of this problem is action segmentation: dividing a sequence of observed behavior into meaningful actions, and determining which of those actions lead to effects in the world. Here we present a Bayesian analysis of how statistical and causal cues to segmentation should optimally be combined, as well as four experiments investigating human action segmentation and causal inference. We find that both people and our model are sensitive to statistical regularities and causal structure in continuous action, and are able to combine these sources of information in order to correctly infer both causal relationships and segmentation boundaries.     

Pulling out the intentional structure of action: the relation between action processing and action production in infancy

Adults and children readily construct action representations organized with respect to an ultimate goal. These representations allow one to predict the consequences of action, interpret and describe actions, and categorize action sequences. In this paper, we explore the ontogeny of hierarchically organized action representations, and its relation to infants' ability to produce similar sequences. To do so, we examine infants' perception and performance of a means-end sequence: pulling a cloth to retrieve a toy. Using a visual habituation paradigm, we demonstrate that 12-month-old infants understand that the initial step of the cloth-pulling sequence is directed toward the ultimate goal of attaining the toy, and use their knowledge of the causal constraints of the sequence to make this goal attribution. Ten-month-olds, however, appear transitional with respect to this understanding: their ability to identify the goal of the cloth-pulling sequence is related to their own ability to planfully solve a similar sequence. These findings are consistent with a burgeoning body of literature suggesting an intimate link between action production and perception, and suggest that this link is in place by at least 10 months of age.     

The link between statistical segmentation and word learning in adults

Many studies have shown that listeners can segment words from running speech based on conditional probabilities of syllable transitions, suggesting that this statistical learning could be a foundational component of language learning. However, few studies have shown a direct link between statistical segmentation and word learning. We examined this possible link in adults by following a statistical segmentation exposure phase with an artificial lexicon learning phase. Participants were able to learn all novel object-label pairings, but pairings were learned faster when labels contained high probability (word-like) or non-occurring syllable transitions from the statistical segmentation phase than when they contained low probability (boundary-straddling) syllable transitions. This suggests that, for adults, labels inconsistent with expectations based on statistical learning are harder to learn than consistent or neutral labels. In contrast, a previous study found that infants learn consistent labels, but not inconsistent or neutral labels.     

Linking sounds to meanings: Infant statistical learning in a natural language

The processes of infant word segmentation and infant word learning have largely been studied separately. However, the ease with which potential word forms are segmented from fluent speech seems likely to influence subsequent mappings between words and their referents. To explore this process, we tested the link between the statistical coherence of sequences presented in fluent speech and infants’ subsequent use of those sequences as labels for novel objects. Notably, the materials were drawn from a natural language unfamiliar to the infants (Italian). The results of three experiments suggest that there is a close relationship between the statistics of the speech stream and subsequent mapping of labels to referents. Mapping was facilitated when the labels contained high transitional probabilities in the forward and/or backward direction (Experiment 1). When no transitional probability information was available (Experiment 2), or when the internal transitional probabilities of the labels were low in both directions (Experiment 3), infants failed to link the labels to their referents. Word learning appears to be strongly influenced by infants’ prior experience with the distribution of sounds that make up words in natural languages.     

Effects of visual information on adults' and infants' auditory statistical learning

Infant and adult learners are able to identify word boundaries in fluent speech using statistical information. Similarly, learners are able to use statistical information to identify word-object associations. Successful language learning requires both feats. In this series of experiments, we presented adults and infants with audio-visual input from which it was possible to identify both word boundaries and word-object relations. Adult learners were able to identify both kinds of statistical relations from the same input. Moreover, their learning was actually facilitated by the presence of two simultaneously present relations. Eight-month-old infants, however, do not appear to benefit from the presence of regular relations between words and objects. Adults, like 8-month-olds, did not benefit from regular audio-visual correspondences when they were tested with tones, rather than linguistic input. These differences in learning outcomes across age and input suggest that both developmental and stimulus-based constraints affect statistical learning.     

Speech segmentation by statistical learning is supported by domain-general processes within working memory

The purpose of this study was to examine the extent to which working memory resources are recruited during statistical learning (SL). Participants were asked to identify novel words in an artificial speech stream where the transitional probabilities between syllables provided the only segmentation cue. Experiments 1 and 2 demonstrated that segmentation performance improved when the speech rate was slowed down, suggesting that SL is supported by some form of active processing or maintenance mechanism that operates more effectively under slower presentation rates. In Experiment 3 we investigated the nature of this mechanism by asking participants to perform a two-back task while listening to the speech stream. Half of the participants performed a two-back rhyme task designed to engage phonological processing, whereas the other half performed a comparable two-back task on un-nameable visual shapes. It was hypothesized that if SL is dependent only upon domain-specific processes (i.e., phonological rehearsal), the rhyme task should impair speech segmentation performance more than the shape task. However, the two loads were equally disruptive to learning, as they both eradicated the benefit provided by the slow rate. These results suggest that SL is supported by working-memory processes that rely on domain-general resources.     

Timing is everything: Changes in presentation rate have opposite effects on auditory and visual implicit statistical learning

Implicit statistical learning (ISL) is exclusive to neither a particular sensory modality nor a single domain of processing. Even so, differences in perceptual processing may substantially affect learning across modalities. In three experiments, statistically equivalent auditory and visual familiarizations were presented under different timing conditions that either facilitated or disrupted temporal processing (fast or slow presentation rates). We find an interaction of rate and modality of presentation: At fast rates, auditory ISL was superior to visual. However, at slow presentation rates, the opposite pattern of results was found: Visual ISL was superior to auditory. Thus, we find that changes to presentation rate differentially affect ISL across sensory modalities. Additional experiments confirmed that this modality-specific effect was not due to cross-modal interference or attentional manipulations. These findings suggest that ISL is rooted in modality-specific, perceptually based processes.     

Preschoolers,adolescents, and adults visually anticipate an agent's efficient action; but only after having observed it frequently

The present study examined the contribution of efficiency reasoning and statistical learning on visual action anticipation in preschool children, adolescents, and adults. To this end, Experiment 1 assessed proactive eye movements of 5-year-old children, 15-year-old adolescents, and adults, who observed an agent stating the intent to reach a goal as quickly as possible. Subsequently the agent could four times either take a short, hence efficient, or long, hence inefficient, path to get to the goal. The results showed that in the first trial participants in none of the age groups predicted above chance level that the agent would produce the efficient action. Instead, we observed an age-dependent increase in action predictions in the subsequent repeated presentation of the same action. Experiment 2 ruled out that participants’ nonconsideration of the efficient path was due to a lack of understanding of the agent's action goal. Moreover, it demonstrated that 5-year-old children do predict that the agent will act efficiently when verbally reasoning about his future action. Overall, the study supports the view that rapid learning from frequency information guides visual action anticipations.     

Dissociation of sensory facilitation and decision bias in statistical learning

ABSTRACT

The detection of regularities in the sensory environment, known as statistical learning, is an important brain function that has been observed in many experimental contexts. In these experiments, statistical learning of patterned sensory stimulation leads to improvements in the speed and/or accuracy with which subsequent stimuli are recognized. That is, statistical learning facilitates the transformation of sensory stimuli into motor responses, but the mechanism by which this occurs is unclear. Statistical learning could improve the efficiency of sensory processing, or it could bias responses toward particular outcomes. The distinction is important, as these different hypotheses imply different functions and different neural substrates for statistical learning. Here we address this problem by studying statistical learning as a decision-making process, which allows us to leverage the extensive computational literature on this topic. Specifically we describe a method for applying the Diffusion Decision Model (DDM) to isolate different sensory and cognitive processes associated with decision-making. The results indicate that statistical learning improves performance on a visual learning task in two distinct ways: by altering the efficiency of sensory processing and by introducing biases in the decision-making process. By fitting the parameters of the DDM to data from individual subjects, we find that the prominence of these two factors differed substantially across the population, and that these differences were predictive of individual performance on the psychophysical task. Overall, these results indicate that different cognitive processes can be recruited by statistical learning, and that the DDM is a powerful framework for detecting these influences.     

Visual statistical learning in the newborn infant

Statistical learning – implicit learning of statistical regularities within sensory input – is a way of acquiring structure within continuous sensory environments. Statistics computation, initially shown to be involved in word segmentation, has been demonstrated to be a general mechanism that operates across domains, across time and space, and across species. Recently, statistical leaning has been reported to be present even at birth when newborns were tested with a speech stream. The aim of the present study was to extend this finding, by investigating whether newborns’ ability to extract statistics operates in multiple modalities, as found for older infants and adults. Using the habituation procedure, two experiments were carried out in which visual sequences were presented. Results demonstrate that statistical learning is a general mechanism that extracts statistics across domain since the onset of sensory experience. Intriguingly, present data reveal that newborn learner’s limited cognitive resources constrain the functioning of statistical learning, narrowing the range of what can be learned.     

Conscious awareness of action potentiates sensorimotor learning

Many everyday skills are unconsciously learned through repetitions of the same behaviour by binding independent motor acts into unified sets of actions. However, our ability to be consciously aware of producing newly and highly trained motor skills raises the question of the role played by conscious awareness of action upon skill acquisition. In this study we strengthened conscious awareness of self-produced sequential finger movements by way of asking participants to judge their performance in terms of maximal fluency after each trial. Control conditions in which participants did not make any judgment or performance-unrelated judgments were also included. Findings indicate that conscious awareness of action, enhanced via subjective appraisal of motor efficiency, potentiates sensorimotor learning and skilful motor production in optimising the processing and sequencing of action units, as compared to the control groups. The current work lends support to the claim that the learning and skilful expression of sensorimotor behaviours might be grounded upon our ability to be consciously aware of our own motor capability and efficiency.     

Children's imitation of causal action sequences is influenced by statistical and pedagogical evidence

Children are ubiquitous imitators, but how do they decide which actions to imitate? One possibility is that children rationally combine multiple sources of information about which actions are necessary to cause a particular outcome. For instance, children might learn from contingencies between action sequences and outcomes across repeated demonstrations, and they might also use information about the actor’s knowledge state and pedagogical intentions. We define a Bayesian model that predicts children will decide whether to imitate part or all of an action sequence based on both the pattern of statistical evidence and the demonstrator’s pedagogical stance. To test this prediction, we conducted an experiment in which preschool children watched an experimenter repeatedly perform sequences of varying actions followed by an outcome. Children’s imitation of sequences that produced the outcome increased, in some cases resulting in production of shorter sequences of actions that the children had never seen performed in isolation. A second experiment established that children interpret the same statistical evidence differently when it comes from a knowledgeable teacher versus a naïve demonstrator. In particular, in the pedagogical case children are more likely to “overimitate” by reproducing the entire demonstrated sequence. This behavior is consistent with our model’s predictions, and suggests that children attend to both statistical and pedagogical evidence in deciding which actions to imitate, rather than obligately imitating successful action sequences.     

The origins of probabilistic inference in human infants

Reasoning under uncertainty is the bread and butter of everyday life. Many areas of psychology, from cognitive, developmental, social, to clinical, are interested in how individuals make inferences and decisions with incomplete information. The ability to reason under uncertainty necessarily involves probability computations, be they exact calculations or estimations. What are the developmental origins of probabilistic reasoning? Recent work has begun to examine whether infants and toddlers can compute probabilities; however, previous experiments have confounded quantity and probability—in most cases young human learners could have relied on simple comparisons of absolute quantities, as opposed to proportions, to succeed in these tasks. We present four experiments providing evidence that infants younger than 12 months show sensitivity to probabilities based on proportions. Furthermore, infants use this sensitivity to make predictions and fulfill their own desires, providing the first demonstration that even preverbal learners use probabilistic information to navigate the world. These results provide strong evidence for a rich quantitative and statistical reasoning system in infants.     

Experience and sentence processing: statistical learning and relative clause comprehension

Many explanations of the difficulties associated with interpreting object relative clauses appeal to the demands that object relatives make on working memory. MacDonald and Christiansen [MacDonald, M. C., & Christiansen, M. H. (2002). Reassessing working memory: Comment on Just and Carpenter (1992) and Waters and Caplan (1996). Psychological Review, 109, 35-54] pointed to variations in reading experience as a source of differences, arguing that the unique word order of object relatives makes their processing more difficult and more sensitive to the effects of previous experience than the processing of subject relatives. This hypothesis was tested in a large-scale study manipulating reading experiences of adults over several weeks. The group receiving relative clause experience increased reading speeds for object relatives more than for subject relatives, whereas a control experience group did not. The reading time data were compared to performance of a computational model given different amounts of experience. The results support claims for experience-based individual differences and an important role for statistical learning in sentence comprehension processes.     

Visual learning of statistical relations among nonadjacent features: Evidence for structural encoding

Recent results suggest that observers can learn, unsupervised, the co-occurrence of independent shape features in viewed patterns (e.g., Fiser & Aslin, 2001). A critical question with regard to these findings is whether learning is driven by a structural, rule-based encoding of spatial relations between distinct features or by a pictorial, template-like encoding, in which spatial configurations of features are embedded in a “holistic” fashion. In two experiments, we test whether observers can learn combinations of features when the paired features are separated by an intervening spatial “gap”, in which other, unrelated features can appear. This manipulation both increases task difficulty and makes it less likely that the feature combinations are encoded simply as larger unitary features. Observers exhibited learning consistent with earlier studies, suggesting that unsupervised learning of compositional structure is based on the explicit encoding of spatial relations between separable visual features. More generally, these results provide support for compositional structure in visual representation.     

The neural correlates of statistical learning in a word segmentation task: An fMRI study

Functional magnetic resonance imaging (fMRI) was used to assess neural activation as participants learned to segment continuous streams of speech containing syllable sequences varying in their transitional probabilities. Speech streams were presented in four runs, each followed by a behavioral test to measure the extent of learning over time. Behavioral performance indicated that participants could discriminate statistically coherent sequences (words) from less coherent sequences (partwords). Individual rates of learning, defined as the difference in ratings for words and partwords, were used as predictors of neural activation to ask which brain areas showed activity associated with these measures. Results showed significant activity in the pars opercularis and pars triangularis regions of the left inferior frontal gyrus (LIFG). The relationship between these findings and prior work on the neural basis of statistical learning is discussed, and parallels to the frontal/subcortical network involved in other forms of implicit sequence learning are considered.     

Studying action representation in children via motor imagery

The use of motor imagery is a widely used experimental paradigm for the study of cognitive aspects of action planning and control in adults. Furthermore, there are indications that motor imagery provides a window into the process of action representation. These notions complement internal model theory suggesting that such representations allow predictions (estimates) about the mapping of the self to parameters of the external world; processes that enable successful planning and execution of action. The ability to mentally represent action is important to the development of motor control. This paper presents a critical review of motor imagery research conducted with children (typically developing and special populations) with focus on its merits and possible shortcomings in studying action representation. Included in the review are age-related findings, possible brain structures involved, experimental paradigms, and recommendations for future work. The merits of this review are associated with the apparent increasing attraction for using and studying motor imagery to understand the developmental aspects of action processing in children.     

Fine-grained sensitivity to statistical information in adult word learning

A language learner trying to acquire a new word must often sift through many potential relations between particular words and their possible meanings. In principle, statistical information about the distribution of those mappings could serve as one important source of data, but little is known about whether learners can in fact track multiple word-referent mappings, and, if they do, the precision with which they can represent those statistics. To test this, two experiments contrasted a pair of possibilities: that learners encode the fine-grained statistics of mappings in the input - both high- and low-frequency mappings - or, alternatively, that only high frequency mappings are represented. Participants were briefly trained on novel word-novel object pairs combined with varying frequencies: some objects were paired with one word, other objects with multiple words with differing frequencies (ranging from 10% to 80%). Results showed that participants were exquisitely sensitive to very small statistical differences in mappings. The second experiment showed that word learners' representation of low frequency mappings is modulated as a function of the variability in the environment. Implications for Mutual Exclusivity and Bayesian accounts of word learning are discussed.     

A review of literature on the link between action observation and action language: advancing a shared semantic theory

Understanding each other is a core concept of social cohesion and, consequently, has immense value in human society. Importantly, shared information leading to cohesion can come from two main sources: observed action and/or language (word) processing. In this paper, we propose a theoretical framework for the link between action observation and action verb processing. Based on the activation of common semantic representations of actions through semantic resonance, this model can account for the neurophysiological, behavioral and neuropsychological domains in the link between action observation and language. Semantic resonance is hypothesized to play a role beyond that of the mere observation of others and can benefit future studies trying to connect action production and language.     

Dissociable behavioural outcomes of visual statistical learning

ABSTRACT

Statistical learning refers to the extraction of probabilistic relationships between stimuli and is increasingly used as a method to understand learning processes. However, numerous cognitive processes are sensitive to the statistical relationships between stimuli and any one measure of learning may conflate these processes; to date little research has focused on differentiating these processes. To understand how multiple processes underlie statistical learning, here we compared, within the same study, operational measures of learning from different tasks that may be differentially sensitive to these processes. In Experiment 1, participants were visually exposed to temporal regularities embedded in a stream of shapes. Their task was to periodically detect whether a shape, whose contrast was staircased to a threshold level, was present or absent. Afterwards, they completed a search task, where statistically predictable shapes were found more quickly. We used the search task to label shape pairs as “learned” or “non-learned”, and then used these labels to analyse the detection task. We found a dissociation between learning on the search task and the detection task where only non-learned pairs showed learning effects in the detection task. This finding was replicated in further experiments with recognition memory (Experiment 2) and associative learning tasks (Experiment 3). Taken together, these findings are consistent with the view that statistical learning may comprise a family of processes that can produce dissociable effects on different aspects of behaviour.