On the use of multilevel modeling as an alternative to items analysis in psycholinguistic research

The use of multilevel modeling is presented as an alternative to separate item and subject ANOVAs (F1 x F2) in psycholinguistic research. Multilevel modeling is commonly utilized to model variability arising from the nesting of lower level observations within higher level units (e.g., students within schools, repeated measures within individuals). However, multilevel models can also be used when two random factors are crossed at the same level, rather than nested. The current work illustrates the use of the multilevel model for crossed random effects within the context of a psycholinguistic experimental study, in which both subjects and items are modeled as random effects within the same analysis, thus avoiding some of the problems plaguing current approaches.     

Parameter estimation of multiple item response profile model

Multiple item response profile (MIRP) models are models with crossed fixed and random effects. At least one between-person factor is crossed with at least one within-person factor, and the persons nested within the levels of the between-person factor are crossed with the items within levels of the within-person factor. Maximum likelihood estimation (MLE) of models for binary data with crossed random effects is challenging. This is because the marginal likelihood does not have a closed form, so that MLE requires numerical or Monte Carlo integration. In addition, the multidimensional structure of MIRPs makes the estimation complex. In this paper, three different estimation methods to meet these challenges are described: the Laplace approximation to the integrand; hierarchical Bayesian analysis, a simulation-based method; and an alternating imputation posterior with adaptive quadrature as the approximation to the integral. In addition, this paper discusses the advantages and disadvantages of these three estimation methods for MIRPs. The three algorithms are compared in a real data application and a simulation study was also done to compare their behaviour.     

Conditional mixed models with crossed random effects

The analysis of continuous hierarchical data such as repeated measures or data from meta‐analyses can be carried out by means of the linear mixed‐effects model. However, in some situations this model, in its standard form, does pose computational problems. For example, when dealing with crossed random‐effects models, the estimation of the variance components becomes a non‐trivial task if only one observation is available for each cross‐classified level. Pseudolikelihood ideas have been used in the context of binary data with standard generalized linear multilevel models. However, even in this case the problem of the estimation of the variance remains non‐trivial. In this paper, we first propose a method to fit a crossed random‐effects model with two levels and continuous outcomes, borrowing ideas from conditional linear mixed‐effects model theory. We also propose a crossed random‐effects model for binary data combining ideas of conditional logistic regression with pseudolikelihood estimation. We apply this method to a case study with data coming from the field of psychometrics and study a series of items (responses) crossed with participants. A simulation study assesses the operational characteristics of the method.     

Modeling Intensive Polytomous Time-Series Eye-Tracking Data: A Dynamic Tree-Based Item Response Model

This paper presents a dynamic tree-based item response (IRTree) model as a novel extension of the autoregressive generalized linear mixed effect model (dynamic GLMM). We illustrate the unique utility of the dynamic IRTree model in its capability of modeling differentiated processes indicated by intensive polytomous time-series eye-tracking data. The dynamic IRTree was inspired by but is distinct from the dynamic GLMM which was previously presented by Cho, Brown-Schmidt, and Lee (Psychometrika 83(3):751–771, 2018). Unlike the dynamic IRTree, the dynamic GLMM is suitable for modeling intensive binary time-series eye-tracking data to identify visual attention to a single interest area over all other possible fixation locations. The dynamic IRTree model is a general modeling framework which can be used to model change processes (trend and autocorrelation) and which allows for decomposing data into various sources of heterogeneity. The dynamic IRTree model was illustrated using an experimental study that employed the visual-world eye-tracking technique. The results of a simulation study showed that parameter recovery of the model was satisfactory and that ignoring trend and autoregressive effects resulted in biased estimates of experimental condition effects in the same conditions found in the empirical study.

     

Random Item IRT Models

It is common practice in IRT to consider items as fixed and persons as random. Both, continuous and categorical person parameters are most often random variables, whereas for items only continuous parameters are used and they are commonly of the fixed type, although exceptions occur. It is shown in the present article that random item parameters make sense theoretically, and that in practice the random item approach is promising to handle several issues, such as the measurement of persons, the explanation of item difficulties, and trouble shooting with respect to DIF. In correspondence with these issues, three parts are included. All three rely on the Rasch model as the simplest model to study, and the same data set is used for all applications. First, it is shown that the Rasch model with fixed persons and random items is an interesting measurement model, both, in theory, and for its goodness of fit. Second, the linear logistic test model with an error term is introduced, so that the explanation of the item difficulties based on the item properties does not need to be perfect. Finally, two more models are presented: the random item profile model (RIP) and the random item mixture model (RIM). In the RIP, DIF is not considered a discrete phenomenon, and when a robust regression approach based on the RIP difficulties is applied, quite good DIF identification results are obtained. In the RIM, no prior anchor sets are defined, but instead a latent DIF class of items is used, so that posterior anchoring is realized (anchoring based on the item mixture). It is shown that both approaches are promising for the identification of DIF.     

Semi‐parametric proportional hazards models with crossed random effects for psychometric response times

The semi‐parametric proportional hazards model with crossed random effects has two important characteristics: it avoids explicit specification of the response time distribution by using semi‐parametric models, and it captures heterogeneity that is due to subjects and items. The proposed model has a proportionality parameter for the speed of each test taker, for the time intensity of each item, and for subject or item characteristics of interest. It is shown how all these parameters can be estimated by Markov chain Monte Carlo methods (Gibbs sampling). The performance of the estimation procedure is assessed with simulations and the model is further illustrated with the analysis of response times from a visual recognition task.     

Explanatory Multidimensional Multilevel Random Item Response Model: An Application to Simultaneous Investigation of Word and Person Contributions to Multidimensional Lexical Representations

This paper presents an explanatory multidimensional multilevel random item response model and its application to reading data with multilevel item structure. The model includes multilevel random item parameters that allow consideration of variability in item parameters at both item and item group levels. Item-level random item parameters were included to model unexplained variance remaining when item related covariates were used to explain variation in item difficulties. Item group-level random item parameters were included to model dependency in item responses among items having the same item stem. Using the model, this study examined the dimensionality of a person’s word knowledge, termed lexical representation, and how aspects of morphological knowledge contributed to lexical representations for different persons, items, and item groups.     

Response-time evidence for mixed memory states in a sequential-presentation change-detection task

Response-time (RT) and choice-probability data were obtained in a rapid visual sequential-presentation change-detection task in which memory set size, study-test lag, and objective change probabilities were manipulated. False “change” judgments increased dramatically with increasing lag, consistent with the idea that study items with long lags were ejected from a discrete-slots buffer. Error RTs were nearly invariant with set size and lag, consistent with the idea that the errors were produced by a stimulus-independent guessing process. The patterns of error and RT data could not be explained in terms of encoding limitations, but were consistent with the hypothesis that long retention lags produced a zero-stimulus-information state that required guessing. Formal modeling of the change-detection RT and error data pointed toward a hybrid model of visual working memory. The hybrid model assumed mixed states involving a combination of memory and guessing, but with higher memory resolution for items with shorter retention lags. The work raises new questions concerning the nature of the memory representations that are produced across the closely related tasks of change detection and visual memory search.     

Why does lag affect the durability of memory-based automaticity: Loss of memory strength or interference?

In Rickard, Lau, and Pashler's (2008) investigation of the lag effect on memory-based automaticity, response times were faster and proportion of trials retrieved was higher at the end of practice for short lag items than for long lag items. However, during testing after a delay, response times were slower and proportion of trials retrieved was lower for short lag items than for long lag items. The current study investigated the extent to which the lag effect on the durability of memory-based automaticity is due to interference or to the loss of memory strength with time. Participants repeatedly practiced alphabet subtraction items in short lag and long lag conditions. After practice, half of the participants were immediately tested and the other half were tested after a 7-day delay. Results indicate that the lag effect on the durability of memory-based automaticity is primarily due to interference. We discuss potential modification of current memory-based processing theories to account for these effects.     

Computation and application of generalized linear mixed model derivatives using lme4

Psychometrika - Maximum likelihood estimation of generalized linear mixed models (GLMMs) is difficult due to marginalization of the random effects. Derivative computations of a fitted GLMM’s...     

Twice random,once mixed: Applying mixed models to simultaneously analyze random effects of language and participants

Psychologists, psycholinguists, and other researchers using language stimuli have been struggling for more than 30 years with the problem of how to analyze experimental data that contain two crossed random effects (items and participants). The classical analysis of variance does not apply; alternatives have been proposed but have failed to catch on, and a statistically unsatisfactory procedure of using two approximations (known as F ₁ and F ₂) has become the standard. A simple and elegant solution using mixed model analysis has been available for 15 years, and recent improvements in statistical software have made mixed models analysis widely available. The aim of this article is to increase the use of mixed models by giving a concise practical introduction and by giving clear directions for undertaking the analysis in the most popular statistical packages. The article also introduces the djmixed add-on package for SPSS, which makes entering the models and reporting their results as straightforward as possible.     

Spacing and lag effects in free recall of pure lists

Repeating list items leads to better recall when the repetitions are separated by several unique items than when they are presented successively; thespacing effect refers to improved recall for spaced versus successive repetition (lag > 0 vs. lag = 0); thelag effect refers to improved recall for long lags versus short lags. Previous demonstrations of the lag effect have utilized lists containing a mixture of items with varying degrees of spacing. Because differential rehearsal of items in mixed lists may exaggerate any effects of spacing, it is important to demonstrate these effects in pure lists. As in Toppino and Schneider (1999), we found an overall advantage for recall of spaced lists. We further report the first demonstration of a lag effect in pure lists, with significantly better recall for lists with widely spaced repetitions than for those with moderately spaced repetitions.     

Lower level mediation in multilevel models

Multilevel models are increasingly used to estimate models for hierarchical and repeated measures data. The authors discuss a model in which there is mediation at the lower level and the mediational links vary randomly across upper level units. One repeated measures example is a case in which a person's daily stressors affect his or her coping efforts, which affect his or her mood, and both links vary randomly across persons. Where there is mediation at the lower level and the mediational links vary randomly across upper level units, the formulas for the indirect effect and its standard error must be modified to include the covariance between the random effects. Because no standard method can estimate such a model, the authors developed an ad hoc method that is illustrated with real and simulated data. Limitations of this method and characteristics of an ideal method are discussed.     

Analyzing temporal patterns of infant sleep and negative affective behavior: a comparison between different statistical models

Objective

Variability in infant sleep and negative affective behavior (NAB) is a developmental phenomenon that has long been of interest to researchers and clinicians. However, analyses and delineation of such temporal patterns were often limited to basic statistical approaches, which may prevent adequate identification of meaningful variation within these patterns. Modern statistical procedures such as additive models may detect specific patterns of temporal variation in infant behavior more effectively.

Method

Hundred and twenty-one mothers were asked to record different behaviors of their 4–44 weeks old healthy infants by diaries for three days consecutively. Circadian patterns as well as individual trajectories and day-to-day variability of infant sleep and NAB were modeled with generalized linear models (GLMs) including a linear and quadratic polynomial for time, a GLM with a polynomial of the 8th order, a GLM with a harmonic function, a generalized linear mixed model (GLMM) with a polynomial of the 8th order, a generalized additive model, and a generalized additive mixed model (GAMM).

Results

The semi-parametric model GAMM was found to fit the data of infant sleep better than any other parametric model used. GLMM with a polynomial of the 8th order and GAMM modeled temporal patterns of infant NAB equally well, although the GLMM exhibited a slightly better model fit while GAMM was easier to interpret. Besides the well-known evening clustering in infant NAB we found a significant second peak in NAB around midday that was not affected by the constant decline in the amounts of NAB across the 3-day study period.

Conclusion

Using advanced statistical procedures (GAMM and GLMM) even small variations and phenomena in infant behavior can be reliably detected. Future studies investigating variability and temporal patterns in infant variables may benefit from these statistical approaches.     

Examining the lag effect under incidental encoding: Contributions of semantic priming and reminding

Memory is better when repeated learning events are spaced than when they are massed (spacing effect), as well as when material is processed semantically than when it is processed graphemically (levels-of-processing effect). Examination of the relationship between levels of processing and spacing for both deeply and shallowly encoded items has shown a spacing effect for items processed deeply, but not shallowly. A semantic priming account of spacing was proposed to explain the interaction between levels of processing and spacing on memory. The current study manipulated levels of processing and the amount of spacing (lag) that occurred between repetitions of items that were incidentally encoded. Results from Experiments 1A and 1B revealed lag effects in test performance when items were deeply and shallowly encoded. Although these findings are inconsistent with a semantic priming account, they can be interpreted within a reminding account, which is explored in Experiment 2. Results from the second experiment indicate that bringing reminding under conscious control benefited items that were presented at a long lag but not at a shorter lag. Together, this study provides evidence that is difficult to accommodate with a semantic priming account of spacing and instead provides additional support for a reminding account suggesting that automatic and controlled processes may both underlie the reminding process.     

A crossed random effects diffusion model for speeded semantic categorization decisions

Choice reaction times (RTs) are often used as a proxy measure of typicality in semantic categorization studies. However, other item properties have been linked to choice RTs as well. We apply a tailored process model of choice RT to a speeded semantic categorization task in order to deconfound different sources of variability in RT. Our model is based on a diffusion model of choice RT, extended to include crossed random effects (of items and participants). This model retains the interesting process interpretation of the diffusion model’s parameters, but it can be applied to choice RTs even in the case where there are few or no repeated measurements of each participant-item combination. Different aspects of the response process are then linked to different types of item properties. A typicality measure turns out to predict the rate of information uptake, while a lexicographic measure predicts the stimulus encoding time. Accessibility measures cannot reliably predict any component of the decision process.     

Assessing individual differences in categorical data

In cognitive modeling, data are often categorical observations taken over participants and items. Usually subsets of these observations are pooled and analyzed by a cognitive model assuming the category counts come from a multinomial distribution with the same model parameters underlying all observations. It is well known that if there are individual differences in participants and/or items, a model analysis of the pooled data may be quite misleading, and in such cases it may be appropriate to augment the cognitive model with parametric random effects assumptions. On the other hand, if random effects are incorporated into a cognitive model that is not needed, the resulting model may be more flexible than the multinomial model that assumes no heterogeneity, and this may lead to overfitting. This article presents Monte Carlo statistical tests for directly detecting individual participant and/or item heterogeneity that depend only on the data structure itself. These tests are based on the fact that heterogeneity in participants and/or items results in overdispersion of certain category count statistics. It is argued that the methods developed in the article should be applied to any set of participant 3 item categorical data prior to cognitive model-based analyses.     

Random effects and extended generalized partial credit models

In this paper it is shown that under the random effects generalized partial credit model for the measurement of a single latent variable by a set of polytomously scored items, the joint marginal probability distribution of the item scores has a closed-form expression in terms of item category location parameters, parameters that characterize the distribution of the latent variable in the subpopulation of examinees with a zero score on all items, and item-scaling parameters. Due to this closed-form expression, all parameters of the random effects generalized partial credit model can be estimated using marginal maximum likelihood estimation without assuming a particular distribution of the latent variable in the population of examinees and without using numerical integration. Also due to this closed-form expression, new special cases of the random effects generalized partial credit model can be identified. In addition to these new special cases, a slightly more general model than the random effects generalized partial credit model is presented. This slightly more general model is called the extended generalized partial credit model. Attention is paid to maximum likelihood estimation of the parameters of the extended generalized partial credit model and to assessing the goodness of fit of the model using generalized likelihood ratio tests. Attention is also paid to person parameter estimation under the random effects generalized partial credit model. It is shown that expected a posteriori estimates can be obtained for all possible score patterns. A simulation study is carried out to show the usefulness of the proposed models compared to the standard models that assume normality of the latent variable in the population of examinees. In an empirical example, some of the procedures proposed are demonstrated.     

The recall of missing items

In two experiments, we studied the recall of missing items. Short lists of common words were presented once and were followed immediately by a random permutation of all but one of the presented items. The task of the subject was to recall the missing item--that is, the item present in the study set but missing from the probe set. Experiment 1 replicated the high accuracy with five-item lists originally reported by Yntema and Trask (1963) and showed that the latencies were quite short (about 750 msec). Experiment 2 varied list length unpredictably and showed that accuracy was a function of both list length (four, five, or six items) and serial position. Latency was again quite short but was essentially independent of list length and serial position. It was possible to simulate most of the effects with the power set model with no free parameters (i.e., parameters that varied with the experimental manipulations). The results seemed to be more consistent with a direct access model (the power set model of TODAM; Murdock, 1995) than with a simple search or serial-scanning model.