Empirical Bayes Derivative Estimates

Abstract

A dynamic system is a set of interacting elements characterized by changes occurring over time. The estimation of derivatives is a mainstay for exploring dynamics of constructs, particularly when the dynamics are complicated or unknown. The presence of measurement error in many social science constructs frequently results in poor estimates of derivatives, as even modest proportions of measurement error can compound when estimating derivatives. Given the overlap in the specification of latent differential equation models and latent growth curve models, and the equivalence of latent growth curve models and mixed models under some conditions, derivatives could be estimated from estimates of random effects. This article proposes a new method for estimating derivatives based on calculating the Empirical Bayes estimates of derivatives from a mixed model. Two simulations compare four derivative estimation methods: Generalized Local Linear Approximation, Generalized Orthogonal Derivative Estimates, Functional Data Analysis, and the proposed Empirical Bayes Derivative Estimates. The simulations consider two data collection scenarios: short time series (≤10 observations) from many individuals or occasions, and long individual time series (25–500 observations). A substantive example visualizing the dynamics of intraindividual positive affect time series is also presented.     

Calculate or wait: Is man an eager or a lazy intuitive statistician?

Research on people's ability to act as intuitive statisticians has mainly focused on the accuracy of estimates of central tendency and variability. In this paper, we investigate two hypothesised cognitive processes by which people make judgements of distribution shape. The first claims that people spontaneously induce abstract representations of distribution properties from experience, including about distribution shape. The second process claims that people construct beliefs about distribution properties post hoc by retrieval from long-term memory of small samples from the distribution, implying format dependence with accuracy that differs depending on judgement format. Results from two experiments confirm the predicted format dependence, suggesting that people are often constrained by the post hoc assessment of distribution properties by sampling from long-term memory. The results, however, also suggest that, although post hoc sampling from memory seems to be the default process, under certain predictable circumstances people do induce abstract representations of distribution shape.     

The locus of the benefits of repetition-lag memory training

The current study examined the effects of responses on error-adjacent trials (i.e., those immediately preceding or following errors) on age differences in measures of intraindividual variability and the shape of response time (RT) distributions on a two-back task. Removing error-adjacent responses reduced variability as measured by the coefficient of variation, but did so similarly for younger and older adults. However, older adults’ standard deviations (SDs) were less than those of younger adults with comparable RTs, raising questions regarding the validity of the coefficient of variation. An ex-Gaussian analysis revealed that removing the RTs on error-adjacent trials reduced the length of the tails of distributions and the skewness of the distributions. These properties were reduced more for older adults than for younger adults. These results indicate that the influence of error-adjacent trials should be considered when analyzing intraindividual variability and the shape of RT distributions to test theories of cognitive aging.     

Effect size, power, and sample size determination for structured means modeling and mimic approaches to between-groups hypothesis testing of means on a single latent construct

While effect size estimates, post hoc power estimates, and a priori sample size determination are becoming a routine part of univariate analyses involving measured variables (e.g., ANOVA), such measures and methods have not been articulated for analyses involving latent means. The current article presents standardized effect size measures for latent mean differences inferred from both structured means modeling and MIMIC approaches to hypothesis testing about differences among means on a single latent construct. These measures are then related to post hoc power analysis, a priori sample size determination, and a relevant measure of construct reliability.I wish to convey my appreciation to the reviewers and Associate Editor, whose suggestions extended and strengthened the article's content immensely, and to Ralph Mueller of The George Washington University for enhancing the clarity of its presentation.     

Intuitive theories of information: beliefs about the value of redundancy

In many situations, quantity estimates from multiple experts or diagnostic instruments must be collected and combined. Normatively, and all else equal, one should value information sources that are nonredundant, in the sense that correlation in forecast errors should be minimized. Past research on the preference for redundancy has been inconclusive. While some studies have suggested that people correctly place higher value on uncorrelated inputs when collecting estimates, others have shown that people either ignore correlation or, in some cases, even prefer it. The present experiments show that the preference for redundancy depends on one's intuitive theory of information. The most common intuitive theory identified is the Error Tradeoff Model (ETM), which explicitly distinguishes between measurement error and bias. According to ETM, measurement error can only be averaged out by consulting the same source multiple times (normatively false), and bias can only be averaged out by consulting different sources (normatively true). As a result, ETM leads people to prefer redundant estimates when the ratio of measurement error to bias is relatively high. Other participants favored different theories. Some adopted the normative model, while others were reluctant to mathematically average estimates from different sources in any circumstance. In a post hoc analysis, science majors were more likely than others to subscribe to the normative model. While tentative, this result lends insight into how intuitive theories might develop and also has potential ramifications for how statistical concepts such as correlation might best be learned and internalized.     

A post hoc correction procedure for systematic errors in time-sampling duration estimates

Numerous previous studies have shown that partial-interval sampling in direct observation systematically overestimates duration and underestimates frequency. Whole-interval sampling systematically underestimates both duration and frequency. This paper presents a post hoc method through which the systematic errors in duration estimates in partial-interval sampling and whole-interval sampling can be minimized.     

Robust Methods for Moderation Analysis with a Two-Level Regression Model

Moderation analysis has many applications in social sciences. Most widely used estimation methods for moderation analysis assume that errors are normally distributed and homoscedastic. When these assumptions are not met, the results from a classical moderation analysis can be misleading. For more reliable moderation analysis, this article proposes two robust methods with a two-level regression model when the predictors do not contain measurement error. One method is based on maximum likelihood with Student's t distribution and the other is based on M-estimators with Huber-type weights. An algorithm for obtaining the robust estimators is developed. Consistent estimates of standard errors of the robust estimators are provided. The robust approaches are compared against normal-distribution-based maximum likelihood (NML) with respect to power and accuracy of parameter estimates through a simulation study. Results show that the robust approaches outperform NML under various distributional conditions. Application of the robust methods is illustrated through a real data example. An R program is developed and documented to facilitate the application of the robust methods.     

The performance of robust test statistics with categorical data

This paper reports on a simulation study that evaluated the performance of five structural equation model test statistics appropriate for categorical data. Both Type I error rate and power were investigated. Different model sizes, sample sizes, numbers of categories, and threshold distributions were considered. Statistics associated with both the diagonally weighted least squares (cat‐DWLS) estimator and with the unweighted least squares (cat‐ULS) estimator were studied. Recent research suggests that cat‐ULS parameter estimates and robust standard errors slightly outperform cat‐DWLS estimates and robust standard errors ( Forero, Maydeu‐Olivares, & Gallardo‐Pujol, 2009 ). The findings of the present research suggest that the mean‐ and variance‐adjusted test statistic associated with the cat‐ULS estimator performs best overall. A new version of this statistic now exists that does not require a degrees‐of‐freedom adjustment ( Asparouhov & Muthén, 2010 ), and this statistic is recommended. Overall, the cat‐ULS estimator is recommended over cat‐DWLS, particularly in small to medium sample sizes.     

Modeling error distributions of growth curve models through Bayesian methods

Growth curve models are widely used in social and behavioral sciences. However, typical growth curve models often assume that the errors are normally distributed although non-normal data may be even more common than normal data. In order to avoid possible statistical inference problems in blindly assuming normality, a general Bayesian framework is proposed to flexibly model normal and non-normal data through the explicit specification of the error distributions. A simulation study shows when the distribution of the error is correctly specified, one can avoid the loss in the efficiency of standard error estimates. A real example on the analysis of mathematical ability growth data from the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 is used to show the application of the proposed methods. Instructions and code on how to conduct growth curve analysis with both normal and non-normal error distributions using the the MCMC procedure of SAS are provided.     

Problematic standard errors and confidence intervals for skewness and kurtosis

Many statistics packages print skewness and kurtosis statistics with estimates of their standard errors. The function most often used for the standard errors (e.g., in SPSS) assumes that the data are drawn from a normal distribution, an unlikely situation. Some textbooks suggest that if the statistic is more than about 2 standard errors from the hypothesized value (i.e., an approximate value for the critical value from the t distribution for moderate or large sample sizes when α = 5%), the hypothesized value can be rejected. This is an inappropriate practice unless the standard error estimate is accurate and the sampling distribution is approximately normal. We show distributions where the traditional standard errors provided by the function underestimate the actual values, often being 5 times too small, and distributions where the function overestimates the true values. Bootstrap standard errors and confidence intervals are more accurate than the traditional approach, although still imperfect. The reasons for this are discussed. We recommend that if you are using skewness and kurtosis statistics based on the 3^rd and 4^th moments, bootstrapping should be used to calculate standard errors and confidence intervals, rather than using the traditional standard. Software in the freeware R for this article provides these estimates.     

Replies to comments on "Which method delivers greater signal-to-noise ratio: Structural equation modelling or regression analysis with weighted composites?" by Yuan and Fang (2023)

Observational data typically contain measurement errors. Covariance-based structural equation modelling (CB-SEM) is capable of modelling measurement errors and yields consistent parameter estimates. In contrast, methods of regression analysis using weighted composites as well as a partial least squares approach to SEM facilitate the prediction and diagnosis of individuals/participants. But regression analysis with weighted composites has been known to yield attenuated regression coefficients when predictors contain errors. Contrary to the common belief that CB-SEM is the preferred method for the analysis of observational data, this article shows that regression analysis via weighted composites yields parameter estimates with much smaller standard errors, and thus corresponds to greater values of the signal-to-noise ratio (SNR). In particular, the SNR for the regression coefficient via the least squares (LS) method with equally weighted composites is mathematically greater than that by CB-SEM if the items for each factor are parallel, even when the SEM model is correctly specified and estimated by an efficient method. Analytical, numerical and empirical results also show that LS regression using weighted composites performs as well as or better than the normal maximum likelihood method for CB-SEM under many conditions even when the population distribution is multivariate normal. Results also show that the LS regression coefficients become more efficient when considering the sampling errors in the weights of composites than those that are conditional on weights.     

Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches

Multiple-baseline studies are prevalent in behavioral research, but questions remain about how to best analyze the resulting data. Monte Carlo methods were used to examine the utility of multilevel models for multiplebaseline data under conditions that varied in the number of participants, number of repeated observations per participant, variance in baseline levels, variance in treatment effects, and amount of autocorrelation in the Level 1 errors. Interval estimates of the average treatment effect were examined for two specifications of the Level 1 error structure (σ² I and first-order autoregressive) and for five different methods of estimating the degrees of freedom (containment, residual, between—within, Satterthwaite, and Kenward—Roger). When the Satterthwaite or Kenward—Roger method was used and an autoregressive Level 1 error structure was specified, the interval estimates of the average treatment effect were relatively accurate. Conversely, the interval estimates of the treatment effect variance were inaccurate, and the corresponding point estimates were biased.     

Examining Nonnormal Latent Variable Distributions for Non-Ignorable Missing Data

Missing not at random (MNAR) modeling for non-ignorable missing responses usually assumes that the latent variable distribution is a bivariate normal distribution. Such an assumption is rarely verified and often employed as a standard in practice. Recent studies for “complete” item responses (i.e., no missing data) have shown that ignoring the nonnormal distribution of a unidimensional latent variable, especially skewed or bimodal, can yield biased estimates and misleading conclusion. However, dealing with the bivariate nonnormal latent variable distribution with present MNAR data has not been looked into. This article proposes to extend unidimensional empirical histogram and Davidian curve methods to simultaneously deal with nonnormal latent variable distribution and MNAR data. A simulation study is carried out to demonstrate the consequence of ignoring bivariate nonnormal distribution on parameter estimates, followed by an empirical analysis of “don’t know” item responses. The results presented in this article show that examining the assumption of bivariate nonnormal latent variable distribution should be considered as a routine for MNAR data to minimize the impact of nonnormality on parameter estimates.     

A note on the sampling properties of the Vincentizing (quantile averaging) procedure

To assess the effect of a manipulation on a response time distribution, psychologists often use Vincentizing or quantile averaging to construct group or “average” distributions. We provide a theorem characterizing the large sample properties of the averaged quantiles when the individual RT distributions all belong to the same location-scale family. We then apply the theorem to estimating parameters for the quantile-averaged distributions. From the theorem, it is shown that parameters of the group distribution can be estimated by generalized least squares. This method provides accurate estimates of standard errors of parameters and can therefore be used in formal inference. The method is benchmarked in a small simulation study against both a maximum likelihood method and an ordinary least-squares method. Generalized least squares essentially is the only method based on the averaged quantiles that is both unbiased and provides accurate estimates of parameter standard errors. It is also proved that for location-scale families, performing generalized least squares on quantile averages is formally equivalent to averaging parameter estimates from generalized least squares performed on individuals. A limitation on the method is that individual RT distributions must be members of the same location-scale family.     

MCMC estimation for the p2 network regression model with crossed random effects

The p₂ model is a statistical model for the analysis of binary relational data with covariates, as occur in social network studies. It can be characterized as a multinomial regression model with crossed random effects that reflect actor heterogeneity and dependence between the ties from and to the same actor in the network. Three Markov chain Monte Carlo (MCMC) estimation methods for the p₂ model are presented to improve iterative generalized least squares (IGLS) estimation developed earlier, two of which use random walk proposals. The third method, an independence chain sampler, and one of the random walk algorithms use normal approximations of the binary network data to generate proposals in the MCMC algorithms. A large‐scale simulation study compares MCMC estimates with IGLS estimates for networks with 20 and 40 actors. It was found that the IGLS estimates have a smaller variance but are severely biased, while the MCMC estimates have a larger variance with a small bias. For networks with 20 actors, mean squared errors are generally comparable or smaller for the IGLS estimates. For networks with 40 actors, mean squared errors are the smallest for the MCMC estimates. Coverage rates of confidence intervals are good for the MCMC estimates but not for the IGLS estimates.     

A Monte Carlo investigation of conjoint analysis index-of-fit: Goodness of fit,significance and power

Researchers in the field of conjoint analysis know the index-of-fit values worsen as the judgmental error of evaluation increases. This simulation study provides guidelines on the goodness of fit based on distribution of index-of-fit for different conjoint analysis designs. The study design included the following factors: number of profiles, number of attributes, algorithm used and judgmental model used. Critical values are provided for deciding the statistical significance of conjoint analysis results. Using these cumulative distributions, the power of the test used to reject the null hypothesis of random ranking is calculated. The test is found to be quite powerful except for the case of very small residual degrees of freedom.The authors thank the editor, the three reviewers and Ellen Foxman for helpful comments on the paper. Sanjay Mishra was a doctoral student at Washington State University at the time this research was completed. He is currently in the Department of Marketing at the University of Kansas.     

The (mis)reporting of statistical results in psychology journals

In order to study the prevalence, nature (direction), and causes of reporting errors in psychology, we checked the consistency of reported test statistics, degrees of freedom, and p values in a random sample of high- and low-impact psychology journals. In a second study, we established the generality of reporting errors in a random sample of recent psychological articles. Our results, on the basis of 281 articles, indicate that around 18% of statistical results in the psychological literature are incorrectly reported. Inconsistencies were more common in low-impact journals than in high-impact journals. Moreover, around 15% of the articles contained at least one statistical conclusion that proved, upon recalculation, to be incorrect; that is, recalculation rendered the previously significant result insignificant, or vice versa. These errors were often in line with researchers’ expectations. We classified the most common errors and contacted authors to shed light on the origins of the errors.     

A Comparison of Inverse-Wishart Prior Specifications for Covariance Matrices in Multilevel Autoregressive Models

Multilevel autoregressive models are especially suited for modeling between-person differences in within-person processes. Fitting these models with Bayesian techniques requires the specification of prior distributions for all parameters. Often it is desirable to specify prior distributions that have negligible effects on the resulting parameter estimates. However, the conjugate prior distribution for covariance matrices—the Inverse-Wishart distribution—tends to be informative when variances are close to zero. This is problematic for multilevel autoregressive models, because autoregressive parameters are usually small for each individual, so that the variance of these parameters will be small. We performed a simulation study to compare the performance of three Inverse-Wishart prior specifications suggested in the literature, when one or more variances for the random effects in the multilevel autoregressive model are small. Our results show that the prior specification that uses plug-in ML estimates of the variances performs best. We advise to always include a sensitivity analysis for the prior specification for covariance matrices of random parameters, especially in autoregressive models, and to include a data-based prior specification in this analysis. We illustrate such an analysis by means of an empirical application on repeated measures data on worrying and positive affect.     

The normal distribution in scaling subjective stimulus differences: Less “normal” than we think?

The Gaussian, or “normal,” distribution is routinely used to model distributions of subjective quantities. This practice rests on a trust in the central limit theorem, yet this theorem does not cover the case of a mixture of Gaussian distributions with different standard deviations, which yields a distribution that is heavier tailed than the Gaussian. In psychophysical judgment experiments, this may result, for example, from fluctuating or interindividually varying attention. Two candidates for describing this commonly encountered type of distribution are the logistic distribution and thet distribution with a small number of degrees of freedom. In reanalyses of experimental data on three-category loudness comparisons, as well as in a Monte Carlo simulation,t(4) was found to model the underlying mixed inter- and intraindividual-distribution of subjective loudness differences quite satisfactorily.     

The relative efficiency of two tests as a function of ability level

A new formula is developed for the relative efficiency of two tests measuring the same trait. The formula expresses relative efficiency solely in terms of the standard errors of measurement and, surprisingly, the frequency distributions of true scores. Approximate methods for estimating relative efficiency may make this function routinely available. A numerical illustration compares new and old estimates of relative efficiency for subtests from the Scholastic Aptitude Test.Research reported in this paper has been supported by grant GB-32781X from National Science Foundation.