Incorporating Mobility in Growth Modeling for Multilevel and Longitudinal Item Response Data

Multilevel data often cannot be represented by the strict form of hierarchy typically assumed in multilevel modeling. A common example is the case in which subjects change their group membership in longitudinal studies (e.g., students transfer schools; employees transition between different departments). In this study, cross-classified and multiple membership models for multilevel and longitudinal item response data (CCMM-MLIRD) are developed to incorporate such mobility, focusing on students' school change in large-scale longitudinal studies. Furthermore, we investigate the effect of incorrectly modeling school membership in the analysis of multilevel and longitudinal item response data. Two types of school mobility are described, and corresponding models are specified. Results of the simulation studies suggested that appropriate modeling of the two types of school mobility using the CCMM-MLIRD yielded good recovery of the parameters and improvement over models that did not incorporate mobility properly. In addition, the consequences of incorrectly modeling the school effects on the variance estimates of the random effects and the standard errors of the fixed effects depended upon mobility patterns and model specifications. Two sets of large-scale longitudinal data are analyzed to illustrate applications of the CCMM-MLIRD for each type of school mobility.     

Latent growth curve analysis with dichotomous items: Comparing four approaches

A Monte Carlo study was used to compare four approaches to growth curve analysis of subjects assessed repeatedly with the same set of dichotomous items: A two‐step procedure first estimating latent trait measures using MULTILOG and then using a hierarchical linear model to examine the changing trajectories with the estimated abilities as the outcome variable; a structural equation model using modified weighted least squares (WLSMV) estimation; and two approaches in the framework of multilevel item response models, including a hierarchical generalized linear model using Laplace estimation, and Bayesian analysis using Markov chain Monte Carlo (MCMC). These four methods have similar power in detecting the average linear slope across time. MCMC and Laplace estimates perform relatively better on the bias of the average linear slope and corresponding standard error, as well as the item location parameters. For the variance of the random intercept, and the covariance between the random intercept and slope, all estimates are biased in most conditions. For the random slope variance, only Laplace estimates are unbiased when there are eight time points.     

A Comparison of Inverse-Wishart Prior Specifications for Covariance Matrices in Multilevel Autoregressive Models

Multilevel autoregressive models are especially suited for modeling between-person differences in within-person processes. Fitting these models with Bayesian techniques requires the specification of prior distributions for all parameters. Often it is desirable to specify prior distributions that have negligible effects on the resulting parameter estimates. However, the conjugate prior distribution for covariance matrices—the Inverse-Wishart distribution—tends to be informative when variances are close to zero. This is problematic for multilevel autoregressive models, because autoregressive parameters are usually small for each individual, so that the variance of these parameters will be small. We performed a simulation study to compare the performance of three Inverse-Wishart prior specifications suggested in the literature, when one or more variances for the random effects in the multilevel autoregressive model are small. Our results show that the prior specification that uses plug-in ML estimates of the variances performs best. We advise to always include a sensitivity analysis for the prior specification for covariance matrices of random parameters, especially in autoregressive models, and to include a data-based prior specification in this analysis. We illustrate such an analysis by means of an empirical application on repeated measures data on worrying and positive affect.     

On the use of multilevel modeling as an alternative to items analysis in psycholinguistic research

The use of multilevel modeling is presented as an alternative to separate item and subject ANOVAs (F1 x F2) in psycholinguistic research. Multilevel modeling is commonly utilized to model variability arising from the nesting of lower level observations within higher level units (e.g., students within schools, repeated measures within individuals). However, multilevel models can also be used when two random factors are crossed at the same level, rather than nested. The current work illustrates the use of the multilevel model for crossed random effects within the context of a psycholinguistic experimental study, in which both subjects and items are modeled as random effects within the same analysis, thus avoiding some of the problems plaguing current approaches.     

Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model

Numerous ways to meta-analyze single-case data have been proposed in the literature; however, consensus has not been reached on the most appropriate method. One method that has been proposed involves multilevel modeling. For this study, we used Monte Carlo methods to examine the appropriateness of Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach for the meta-analysis of single-case data. Specifically, we examined the fixed effects (e.g., the overall average treatment effect) and the variance components (e.g., the between-person within-study variance in the treatment effect) in a three-level multilevel model (repeated observations nested within individuals, nested within studies). More specifically, bias of the point estimates, confidence interval coverage rates, and interval widths were examined as a function of the number of primary studies per meta-analysis, the modal number of participants per primary study, the modal series length per primary study, the level of autocorrelation, and the variances of the error terms. The degree to which the findings of this study are supportive of using Van den Noortgate and Onghena's (2008) raw-data multilevel modeling approach to meta-analyzing single-case data depends on the particular parameter of interest. Estimates of the average treatment effect tended to be unbiased and produced confidence intervals that tended to overcover, but did come close to the nominal level as Level-3 sample size increased. Conversely, estimates of the variance in the treatment effect tended to be biased, and the confidence intervals for those estimates were inaccurate.     

Reliability and motor memory

The generalizability framework was employed to estimate variance components and reliability of performance in a positioning task. The subjects (n =70) performed 16 trials at each of three target lengths of 25 cm, 45 cm, and 65 cm. The data were analyzed using a subjects x targets x trials repeated measures ANOVA. Two reliability coefficients were estimated. The first (R) provided an estimate of performance reliability generalized over targets and trials. The second (R') treated targets as a source of true-score variance and hence was generalized over trials alone. Both reliability coefficients were higher for algebraic error than for absolute error, and R1 provided higher reliability estimates than R for both dependent variables. Increasing the number of positioning trials from 2 to 16 at each target length did not appreciably alter either reliability coefficient. The overall low reliability of R appears to compound the dependent variables and statistical power problems associated with short-term motor-memory studies.     

Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches

Multiple-baseline studies are prevalent in behavioral research, but questions remain about how to best analyze the resulting data. Monte Carlo methods were used to examine the utility of multilevel models for multiplebaseline data under conditions that varied in the number of participants, number of repeated observations per participant, variance in baseline levels, variance in treatment effects, and amount of autocorrelation in the Level 1 errors. Interval estimates of the average treatment effect were examined for two specifications of the Level 1 error structure (σ² I and first-order autoregressive) and for five different methods of estimating the degrees of freedom (containment, residual, between—within, Satterthwaite, and Kenward—Roger). When the Satterthwaite or Kenward—Roger method was used and an autoregressive Level 1 error structure was specified, the interval estimates of the average treatment effect were relatively accurate. Conversely, the interval estimates of the treatment effect variance were inaccurate, and the corresponding point estimates were biased.     

人机交互过程中认知负荷的综合测评方法

设计模拟网络引擎搜索和心算双任务实验,分析主观评定、绩效测量和生理测量三类评估指标对认知负荷变化的敏感性;采用因素分析、BP神经网络和自组织神经网络三种建模方法,探索人机交互过程中认知负荷的综合评估建模方法。结果显示：心理努力、任务主观难度、注视时间、注视次数、主任务反应时、主任务正确率6个指标对认知负荷变化敏感;采用多维综合评估模型对双任务作业认知负荷进行测量总体上比采用单一评估指标的测量更为有效。BP网络和自组织神经网络两种神经网络模型对认知负荷的测量结果优于传统的因素分析方法     

A general model for the analysis of multilevel data

A general model is developed for the analysis of multivariate multilevel data structures. Special cases of the model include repeated measures designs, multiple matrix samples, multilevel latent variable models, multiple time series, and variance and covariance component models.We would like to acknowledge the helpful comments of Ruth Silver. We also wish to thank the referees for helping to clarify the paper. This work was partly carried out with research funds provided by the Economic and Social Research Council (U.K.).     

The Impact of Inappropriate Modeling of Cross-Classified Data Structures

Cross-classified random effects modeling (CCREM) is used to model multilevel data from nonhierarchical contexts. These models are widely discussed but infrequently used in social science research. Because little research exists assessing when it is necessary to use CCREM, 2 studies were conducted. A real data set with a cross-classified structure was analyzed by comparing parameter estimates when ignoring versus modeling the cross-classified data structure. A follow-up simulation study investigated potential factors affecting the need to use CCREM. Results indicated that when the structure is ignored, fixed-effect estimates were unaffected, but standard error estimates associated with the variables modeled incorrectly were biased. Estimates of the variance components also displayed bias, which was related to several study factors.     

Generalized reliability estimation using repeated measurements

Reliability can be studied in a generalized way using repeated measurements. Linear mixed models are used to derive generalized test–retest reliability measures. The method allows for repeated measures with a different mean structure due to correction for covariate effects. Furthermore, different variance–covariance structures between measurements can be implemented. When the variance structure reduces to a random intercept (compound symmetry), classical methods are recovered. With more complex variance structures (e.g. including random slopes of time and/or serial correlation), time‐dependent reliability functions are obtained. The effect of time lag between measurements on reliability estimates can be evaluated. The methodology is applied to a psychiatric scale for schizophrenia.     

Effects of Modeling the Heterogeneity on Inferences Drawn from Multilevel Designs

This article uses Monte Carlo techniques to examine the effect of heterogeneity of variance in multilevel analyses in terms of relative bias, coverage probability, and root mean square error (RMSE). For all simulated data sets, the parameters were estimated using the restricted maximum-likelihood (REML) method both assuming homogeneity and incorporating heterogeneity into multilevel models. We find that (a) the estimates for the fixed parameters are unbiased, but the associated standard errors are frequently biased when heterogeneity is ignored; by contrast, the standard errors of the fixed effects are almost always accurate when heterogeneity is considered; (b) the estimates for the random parameters are slightly overestimated; (c) both the homogeneous and heterogeneous models produce standard errors of the variance component estimates that are underestimated; however, taking heterogeneity into account, the REML-estimations give correct estimates of the standard errors at the lowest level and lead to less underestimated standard errors at the highest level; and (d) from the RMSE point of view, REML accounting for heterogeneity outperforms REML assuming homogeneity; a considerable improvement has been particularly detected for the fixed parameters. Based on this, we conclude that the solution presented can be uniformly adopted. We illustrate the process using a real dataset.     

Separating sensitivity from response bias: implications of comparisons of yes-no and forced-choice tests for models and measures of recognition memory

A fundamental challenge to psychological research is the measurement of cognitive processes uncontaminated by response strategies resulting from different testing procedures. Test-free estimates of ability are vital when comparing the performance of different groups or different conditions. The current study applied several sets of measurement models to both forced-choice and yes-no recognition memory tests and concluded that the traditional signal-detection model resulted in distorted estimates of accuracy. Two-factor models were necessary to separate memory sensitivity from response bias. These models indicated that (a) memory accuracy did not differ across the tests and (b) the tests relied on the same underlying memory processes. The results illustrate the pitfalls of using a single-component model to measure accuracy in tasks that reflect 2 or more underlying processes.     

Using response ratios for meta-analyzing single-case designs with behavioral outcomes

Methods for meta-analyzing single-case designs (SCDs) are needed to inform evidence-based practice in clinical and school settings and to draw broader and more defensible generalizations in areas where SCDs comprise a large part of the research base. The most widely used outcomes in single-case research are measures of behavior collected using systematic direct observation, which typically take the form of rates or proportions. For studies that use such measures, one simple and intuitive way to quantify effect sizes is in terms of proportionate change from baseline, using an effect size known as the log response ratio. This paper describes methods for estimating log response ratios and combining the estimates using meta-analysis. The methods are based on a simple model for comparing two phases, where the level of the outcome is stable within each phase and the repeated outcome measurements are independent. Although auto-correlation will lead to biased estimates of the sampling variance of the effect size, meta-analysis of response ratios can be conducted with robust variance estimation procedures that remain valid even when sampling variance estimates are biased. The methods are demonstrated using data from a recent meta-analysis on group contingency interventions for student problem behavior.     

Physical activity levels in young adult Hispanics and Whites: Social cognitive theory determinants

The psychosocial determinants of health-impairing physical inactivity among Hispanic populations have not been well explored nor have systematic comparisons been made with White populations using Social Cognitive Theory (SCT) measures. Three exercise-relevant efficacy measures (task, scheduling, and response efficacy), three exercise-relevant expectancy measures (physical health, psychological health, and self-evaluative), and self reports of activity levels were obtained from 20-year-old male and female Hispanics (n?=?94) and Whites (n?=?94). Activity levels for the two groups were analyzed in separate regression analyses that included the six SCT measures and gender as predictors. The set of seven predictors accounted for 51% of the variance in self-reported activity levels in each analysis. For young adult Hispanics, task efficacy, response efficacy, mental-health expectancies, and self-evaluative expectancies predicted activity level. For young adult Whites, scheduling efficacy and self-evaluative expectancies predicted activity level. Gender was not a significant predictor of activity level for either group. A multivariate analysis of variance indicated that the only SCT predictor on which Hispanics and Whites significantly differed was mental-health expectancies. The results of this study indicate that the psychosocial determinants of exercise are qualitatively different for the two groups and that the determinants of physical activity levels for young adult Hispanics may not be as effective as those of young adult Whites in sustaining lifelong exercise habits. Thus the present study offers a promising strategy for detecting inactivity-related physical- and mental-health risks at an age when lifelong habits of physical activity are still being formed.     

Revisiting dispersion in count data item response theory models: The Conway–Maxwell–Poisson counts model

Count data naturally arise in several areas of cognitive ability testing, such as processing speed, memory, verbal fluency, and divergent thinking. Contemporary count data item response theory models, however, are not flexible enough, especially to account for over- and underdispersion at the same time. For example, the Rasch Poisson counts model (RPCM) assumes equidispersion (conditional mean and variance coincide) which is often violated in empirical data. This work introduces the Conway–Maxwell–Poisson counts model (CMPCM) that can handle underdispersion (variance lower than the mean), equidispersion, and overdispersion (variance larger than the mean) in general and specifically at the item level. A simulation study revealed satisfactory parameter recovery at moderate sample sizes and mostly unbiased standard errors for the proposed estimation approach. In addition, plausible empirical reliability estimates resulted, while those based on the RPCM were biased downwards (underdispersion) and biased upwards (overdispersion) when the simulation model deviated from equidispersion. Finally, verbal fluency data were analysed and the CMPCM with item-specific dispersion parameters fitted the data best. Dispersion parameter estimates indicated underdispersion for three out of four items. Overall, these findings indicate the feasibility and importance of the suggested flexible count data modelling approach.     

新世纪20年国内心理统计方法研究回顾

新世纪头20年, 国内心理学11本专业期刊一共发表了213篇统计方法研究论文。研究范围主要包括以下10类(按论文篇数排序)：结构方程模型、测验信度、中介效应、效应量与检验力、纵向研究、调节效应、探索性因子分析、潜在类别模型、共同方法偏差和多层线性模型。对各类做了简单的回顾与梳理。结果发现, 国内心理统计方法研究的广度和深度都不断增加, 研究热点在相互融合中共同发展; 但综述类论文比例较大, 原创性研究论文比例有待提高, 研究力量也有待加强。     

Affective and cognitive predictors of affective response to exercise: Examining unique and overlapping variance

ObjectivesAffective response to exercise has been suggested as an important factor in determining regular exercise. However, it is unclear the extent to which anticipatory affect factors (affective attitudes, implicit associations, and affective associations), anticipated affect factors (anticipated regret, anticipated pride), and cognitive factors (self-efficacy, intentions) explain overlapping or unique variance in affective response to exercise.DesignWe systematically examined the extent to which these various affective and cognitive factors relevant to exercise predict affective response, and determined the extent to which these factors account for unique or overlapping variance in affective response.MethodHealthy young adults (N = 69) completed measures of affective attitudes, affective associations, implicit associations, anticipated affect, self-efficacy, and exercise intentions. Participants then exercised for 20-min at moderate intensity on a treadmill, during and after which they reported their affective response. Using variables that were independent predictors, we conducted multivariate analyses to determine which factors account for unique variance in affective response to exercise.ResultsIn three of four multivariate models, both anticipated and anticipatory affect variables explained unique variance in affective response during exercise. Only anticipatory affect variables accounted for unique variance in affective response immediately post-exercise. Finally, the association between exercise self-efficacy and affective response during-exercise was rendered non-significant after controlling for affective factors in all three multivariate models.ConclusionsThe unique associations between affective response to exercise and affective, but not cognitive, factors elucidate key predictors of affective response during- and post-exercise.     

Common and specific cognitive deficits in schizophrenia: relationships to function

The goals of the present study were to assess the interrelationships among tasks from the MATRICS and CNTRACS batteries, to determine the degree to which tasks from each battery capture unique variance in cognitive dysfunction in schizophrenia, and to determine the ability of tasks from each battery to predict functional outcome. Subjects were 104 schizophrenia patients and 132 healthy control subjects recruited as part of the CNTRACS initiative. All subjects completed four CNTRACS tasks and two tasks from the MATRICS battery: Brief Assessment of Cognition in Schizophrenia Symbol Coding and the Hopkins Verbal Learning Test. Functional outcome was also assessed in the schizophrenia subjects. In both the patient and control groups, we found significant intercorrelations between all higher order cognitive tasks (episodic memory, goal maintenance, processing speed, verbal learning) but minimal relationships with the visual task. For almost all tasks, scores were significantly related to measures of functional outcome, with higher associations between CNTRACS tasks and performance-based measures of function and between one of the MATRICS tasks and self-reported functioning, relative to the other functioning measures. After regressing out variance shared by other tasks, we continued to observe group differences in performance among task residuals, particularly for measures of episodic memory from both batteries, although these residuals did not correlate as robustly with functional outcome as raw test scores. These findings suggest that there exists both shared and specific variance across cognitive tasks related to cognitive and functional impairments in schizophrenia and that measures derived from cognitive neuroscience can predict functional capacity and status in schizophrenia.     

Modeling individual differences in cognition

Many evaluations of cognitive models rely on data that have been averaged or aggregated across all experimental subjects, and so fail to consider the possibility of important individual differences between subjects. Other evaluations are done at the single-subject level, and so fail to benefit from the reduction of noise that data averaging or aggregation potentially provides. To overcome these weaknesses, we have developed a general approach to modeling individual differences using families of cognitive models in which different groups of subjects are identified as having different psychological behavior. Separate models with separate parameterizations are applied to each group of subjects, and Bayesian model selection is used to determine the appropriate number of groups. We evaluate this individual differences approach in a simulation study and show that it is superior in terms of the key modeling goals of prediction and understanding. We also provide two practical demonstrations of the approach, one using the ALCOVE model of category learning with data from four previously analyzed category learning experiments, the other using multidimensional scaling representational models with previously analyzed similarity data for colors. In both demonstrations, meaningful individual differences are found and the psychological models are able to account for this variation through interpretable differences in parameterization. The results highlight the potential of extending cognitive models to consider individual differences.