Model change and reliability in scientific inference

One persistent challenge in scientific practice is that the structure of the world can be unstable: changes in the broader context can alter which model of a phenomenon is preferred, all without any overt signal. Scientific discovery becomes much harder when we have a moving target, and the resulting incorrect understandings of relationships in the world can have significant real-world and practical consequences. In this paper, we argue that it is common (in certain sciences) to have changes of context that lead to changes in the relationships under study, but that standard normative accounts of scientific inquiry have assumed away this problem. At the same time, we show that inference and discovery methods can “protect” themselves in various ways against this possibility by using methods with the novel methodological virtue of “diligence.” Unfortunately, this desirable virtue provably is incompatible with other desirable methodological virtues that are central to reliable inquiry. No scientific method can provide every virtue that we might want.     

On accuracy in reliability estimation

This study in parametric test theory deals with the statistics of reliability estimation when scores on two parts of a test follow a binormal distribution with equal (Case 1) or unequal (Case 2) expectations. In each case biased maximum-likelihood estimators of reliability are obtained and converted into unbiased estimators. Sampling distributions are derived. Second moments are obtained and utilized in calculating mean square errors of estimation as a measure of accuracy. A rank order of four estimators is established. There is a uniformly best estimator. Tables of absolute and relative accuracies are provided for various reliability parameters and sample sizes.     

Examining body dissatisfaction in young men within a biopsychosocial framework

This study examined biopsychosocial factors related to body dissatisfaction in young men within multivariate and moderator contexts. A female sample was included as a gender comparison. Male (n = 111) and female (n = 236) undergraduates filled out self-report questionnaires assessing body mass index (BMI), media influence, a history of weight-related teasing, and socially prescribed perfectionism, along with various indices of body dissatisfaction. Perceived pressure from the media was consistently related to body dissatisfaction in men whereas multiple biopsychosocial variables accounted for body dissatisfaction in women. Socially prescribed perfectionism and a history of weight teasing each moderated the relationship between BMI and male body dissatisfaction, identifying men low in body dissatisfaction. Findings indicate that applying a biopsychosocial framework to the study of body dissatisfaction in men is useful and suggest the need for including other factors, such as male peers and sports involvement, in understanding contributors to male body image.     

Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates

Recent work has given rise to the view that reward-based decision making is governed by two key controllers: a habit system, which stores stimulus-response associations shaped by past reward, and a goal-oriented system that selects actions based on their anticipated outcomes. The current literature provides a rich body of computational theory addressing habit formation, centering on temporal-difference learning mechanisms. Less progress has been made toward formalizing the processes involved in goal-directed decision making. We draw on recent work in cognitive neuroscience, animal conditioning, cognitive and developmental psychology, and machine learning to outline a new theory of goal-directed decision making. Our basic proposal is that the brain, within an identifiable network of cortical and subcortical structures, implements a probabilistic generative model of reward, and that goal-directed decision making is effected through Bayesian inversion of this model. We present a set of simulations implementing the account, which address benchmark behavioral and neuroscientific findings, and give rise to a set of testable predictions. We also discuss the relationship between the proposed framework and other models of decision making, including recent models of perceptual choice, to which our theory bears a direct connection.     

The Model Human Processor and the older adult: parameter estimation and validation within a mobile phone task

The authors estimate weighted mean values for nine information processing parameters for older adults using the Card, Moran, and Newell (1983) Model Human Processor model. The authors validate a subset of these parameters by modeling two mobile phone tasks using two different phones and comparing model predictions to a sample of younger (N = 20; M-sub(age) = 20) and older (N = 20; M-sub(age) = 69) adults. Older adult models fit keystroke-level performance at the aggregate grain of analysis extremely well (R = 0.99) and produced equivalent fits to previously validated younger adult models. Critical path analyses highlighted points of poor design as a function of cognitive workload, hardware/software design, and user characteristics. The findings demonstrate that estimated older adult information processing parameters are valid for modeling purposes, can help designers understand age-related performance using existing interfaces, and may support the development of age-sensitive technologies.     

A longitudinal study of time estimation in psychotic disorders

5 healthy subjects and 9 patients with hospital diagnoses of schizophrenia were studied by means of repeated measurements of time estimation (production method) during a 1- to 2-yr period. The healthy subjects exhibited moderate variability in their time estimations and tended to over- or under-estimate somewhat. Both over- and under-estimation were also found among patients, but the patients were often more variable in their estimations and in some cases deviated more from the correct estimation than the healthy subjects. Two patients were decided under-estimators. They were both young, subchronic schizophrenics. Chronic schizophrenic patients tended to over-estimate time, but the amount of over-estimation had no noticeable correlation to the degree of clinical disturbance. Two patients gradually changed from under- to over-estimation during the observation period. One patient was evidently misdiagnosed. He suffered from a bipolar affective disorder, and his time estimations seemed to depend on his clinical state during the course of the disorder. The results are discussed in the context of earlier findings and physiological disturbances in chronic schizophrenia.     

Distinguishing volumetric content from perceptual presence within a predictive processing framework

Phenomenology and the Cognitive Sciences - I argue for an overlooked distinction between perceptual presence and volumetric content, and flesh it out in terms of predictive processing. Within the...     

Decisional separability, model identification, and statistical inference in the general recognition theory framework

Recent work in the general recognition theory (GRT) framework indicates that there are serious problems with some of the inferential machinery designed to detect perceptual and decisional interactions in multidimensional identification and categorization (Mack, Richler, Gauthier, & Palmeri, 2011). These problems are more extensive than previously recognized, as we show through new analytic and simulation-based results indicating that failure of decisional separability is not identifiable in the Gaussian GRT model with either of two common response selection models. We also describe previously unnoticed formal implicational relationships between seemingly distinct tests of perceptual and decisional interactions. Augmenting these formal results with further simulations, we show that tests based on marginal signal detection parameters produce unacceptably high rates of incorrect statistical significance. We conclude by discussing the scope of the implications of these results, and we offer a brief sketch of a new set of recommendations for testing relationships between dimensions in perception and response selection in the full-factorial identification paradigm.     

Crossover of job demands and emotional exhaustion within teams: a longitudinal multilevel study

This study investigated the crossover of job demands and emotional exhaustion among team members and the moderating effect of cohesiveness and social support on this process. Participants were 310 employees of an employment agency in the Netherlands, working in one of 100 teams. Multilevel analysis using a longitudinal design did not reveal a main effect of crossover. However, consistent with the study's hypotheses, the results showed a moderating effect of team cohesiveness and social support. We detected crossover of job demands and emotional exhaustion across time from the group to individual team members only in teams characterized by high levels of cohesiveness and social support. Teams characterized by low levels of cohesiveness and social support showed no crossover of job demands and exhaustion. The findings demonstrate that team-level moderators play an important role in crossover processes. Moreover, social support and cohesiveness may not always be positive.     

Bayesian inference for psychology,part IV: parameter estimation and Bayes factors

In the psychological literature, there are two seemingly different approaches to inference: that from estimation of posterior intervals and that from Bayes factors. We provide an overview of each method and show that a salient difference is the choice of models. The two approaches as commonly practiced can be unified with a certain model specification, now popular in the statistics literature, called spike-and-slab priors. A spike-and-slab prior is a mixture of a null model, the spike, with an effect model, the slab. The estimate of the effect size here is a function of the Bayes factor, showing that estimation and model comparison can be unified. The salient difference is that common Bayes factor approaches provide for privileged consideration of theoretically useful parameter values, such as the value corresponding to the null hypothesis, while estimation approaches do not. Both approaches, either privileging the null or not, are useful depending on the goals of the analyst.     

Bayesian inference for psychology,part III: Parameter estimation in nonstandard models

We demonstrate the use of three popular Bayesian software packages that enable researchers to estimate parameters in a broad class of models that are commonly used in psychological research. We focus on WinBUGS, JAGS, and Stan, and show how they can be interfaced from R and MATLAB. We illustrate the use of the packages through two fully worked examples; the examples involve a simple univariate linear regression and fitting a multinomial processing tree model to data from a classic false-memory experiment. We conclude with a comparison of the strengths and weaknesses of the packages. Our example code, data, and this text are available via https://osf.io/ucmaz/.     

The sexual sensation seeking scale: reliability and validity within a heterosexual college student sample

The Sexual Sensation Seeking Scale (SSSS; Kalichman & Rompa, 1995) is a measure of the propensity to seek out novel or risky sexual stimulation. This measure has been consistently related to HIV-risk behavior in several different samples of gay men and with behavioral correlates of sexual permissiveness in college students. Five hundred twenty-eight college students completed the SSSS and several other sexuality and personality measures. The results reveal significant gender differences on the SSSS and similar (although stronger among women) patterns of significant correlations with sexually permissive characteristics and behaviors. Furthermore, SSSS scores were more highly correlated with sexual measures, whereas a more general measure of sensation seeking was more highly correlated with general personality measures. We discuss implications for use of the SSSS by sexuality educators and sex therapists.     

A Bayesian approach to estimating variance components within a multivariate generalizability theory framework

In many behavioral research areas, multivariate generalizability theory (mG theory) has been typically used to investigate the reliability of certain multidimensional assessments. However, traditional mG-theory estimation—namely, using frequentist approaches—has limits, leading researchers to fail to take full advantage of the information that mG theory can offer regarding the reliability of measurements. Alternatively, Bayesian methods provide more information than frequentist approaches can offer. This article presents instructional guidelines on how to implement mG-theory analyses in a Bayesian framework; in particular, BUGS code is presented to fit commonly seen designs from mG theory, including single-facet designs, two-facet crossed designs, and two-facet nested designs. In addition to concrete examples that are closely related to the selected designs and the corresponding BUGS code, a simulated dataset is provided to demonstrate the utility and advantages of the Bayesian approach. This article is intended to serve as a tutorial reference for applied researchers and methodologists conducting mG-theory studies.     

Accounting for the phenomenology and varieties of auditory verbal hallucination within a predictive processing framework

Two challenges that face popular self-monitoring theories (SMTs) of auditory verbal hallucination (AVH) are that they cannot account for the auditory phenomenology of AVHs and that they cannot account for their variety. In this paper I show that both challenges can be met by adopting a predictive processing framework (PPF), and by viewing AVHs as arising from abnormalities in predictive processing. I show how, within the PPF, both the auditory phenomenology of AVHs, and three subtypes of AVH, can be accounted for.     

A note on reliability estimation for a test with components of unknown functional lengths

This research note proposes two reliability coefficients for tests with components of unknown functional lengths. The derived coefficients are extensions of the techniques devised by Kristof and Feldt and do not require a reduction of test components into parts. Simulation study indicates that the new coefficients yield reasonably stable reliability estimates when the number of test components is small.     

Intimacy and the magnitude and experience of episodic relational uncertainty within romantic relationships

This paper seeks to clarify the association between the intimacy and the magnitude of relational uncertainty generated by specific events within romantic relationships. More specifically, we suggest that episodic relational uncertainty peaks at moderate levels of intimacy. We conducted a cross–sectional study in which 328 romantic relationship participants reported their reactions to a hypothetical relational uncertainty increasing event. Although the effect size was small, findings documented a curvilinear association between intimacy and episodic relational uncertainty. Further, hierarchical regression results supported our predictions about the effects of intimacy and episodic relational uncertainty on people’s reactions to the events. We discuss the implications of our findings for understanding the roles of both intimacy and episodic relational uncertainty within romantic relationships.     

Controlling decremental and inflationary effects in reliability estimation resulting from violations of assumptions.

Two assumptions in classical test theory, essential tau-equivalence and independence of measurement errors, when violated may produce attenuated or inflated estimates of reliability, respectively. Inflation stemming from correlated errors can be controlled by a procedure in which systematically created equivalent halves of a given measuring instrument are administered across two occasions. When poor approximations to equivalent halves are constructed for this purpose, however, distortion in the opposite direction may result, being sometimes quite large when measuring instruments are not essentially tau-equivalent (or, at the practical level, unidimensional). The nature of these decrements are discussed and illustrated, and a number of procedures for eliminating them introduced.     

Correction to: Distinguishing volumetric content from perceptual presence within a predictive processing framework

The original version of this article unfortunately has missing statement in the Acknowledgments section.