Estimation of maximal reliability for multiple‐component instruments in multilevel designs

A procedure for point and interval estimation of maximal reliability of multiple‐component measuring instruments in multi‐level settings is outlined. The approach is applicable to hierarchical designs in which individuals are nested within higher‐order units and exhibit possibly related performance on components of a given homogeneous scale. The method is developed within the framework of multi‐level factor analysis. The proposed procedure is illustrated with an empirical example.     

Examining change in maximal reliability for multiple‐component measuring instruments

A method for examining change in maximal reliability for pre‐specified sets of congeneric measures when developing a multi‐component instrument is outlined. The approach is applicable for purposes of estimation and testing of gain or loss in the maximal reliability coefficient as a consequence of adding or dropping one or more measures from a homogeneous composite with uncorrelated errors, as well as when one is concerned with optimal component choice for highest increase or correspondingly smallest drop in maximal reliability. The method is compared with a procedure for ascertaining change in unweighted sum score reliability, and implications for instrument construction and revision are discussed. The approach is illustrated with a numerical example.     

Reliability of multiple‐component measuring instruments: Improved evaluation in repeated measure designs

A covariance structure analysis method for improved point and interval estimation of composite reliability in repeated measure designs is outlined that accounts for specificity variance. The approach also permits the testing of time‐invariance in reliability of multiple‐component instruments in terms of the ratio of ‘pure’ measurement error variance to observed scale score variance. In addition, the procedure allows interval estimation of the difference in composite reliability coefficients across assessment occasions. The method described is illustrated with data from a cognitive intervention study.     

Studying group and time invariance in maximal reliability for multiple‐component measuring instruments via covariance structure modelling

A method for examining invariance in maximal reliability for weighted combinations of congeneric measures is described. The approach is developed within the framework of covariance structure modelling and allows one to ascertain whether a multi‐component instrument consisting of homogeneous measures is associated with the same minimal relative error variance in distinct populations or over time. The procedure yields as a by‐product an interval measure of discrepancy in maximal reliability across independent groups or assessment occasions, and is illustrated with two examples.     

The concurrent validity and test–retest reliability of a visuospatial working memory task

To determine whether the visuospatial n-back working memory task is a reliable and valid measure of cognitive processes believed to underlie intelligence, this study compared the reaction times and accuracy of performance of 70 participants, with performance on the Multidimensional Aptitude Battery (MAB). Testing was conducted over two sessions separated by 1 week. Participants completed the MAB during the second test session. Moderate test–retest reliability for percentage accuracy scores was found across the four levels of the n-back task, whilst reaction times were highly reliable. Furthermore, participants' performance on the MAB was negatively correlated with accuracy of performance at the easier levels of the n-back task and positively correlated with accuracy of performance at the harder task levels. These findings confirm previous research examining the cognitive basis of intelligence, and suggest that intelligence is the product of faster speed of information processing, as well as superior working memory capacity.     

Enduring cognitive dysfunction in unipolar major depression: A test–retest study using the Stroop paradigm

Hammar, Å., Sørensen, L., Årdal, G., Oedegaard, K.J., Kroken, R., Roness, A. & Lund, A. (2009). Enduring cognitive dysfunction in unipolar major depression: A test–retest study using the Stroop‐paradigm. Scandinavian Journal of Psychology. The aim of the study was to investigate automatic and effortful information processing with the Stroop paradigm in a long term perspective in patients with major depressive disorder (MDD). Patients were tested at two test occasions: at inclusion with a Hamilton Depression Rating Scale (HDRS) score >18, and after 6 months, when most patients had experienced symptom reduction. The Stroop paradigm is considered to measure aspects of attention and executive functioning and consists of three conditions/cards: naming the color of the patches (Color), reading of the color‐words (Word) and naming the ink color of color‐words (Color‐Word). The Color‐Word condition is proved to be the most cognitive demanding task and requires the proband to actively suppress interference and is therefore considered to require more effortful information processing, whereas naming the color of the patches and reading the color‐words are expected to be more automatic and less cognitive demanding. A homogenous group of 19 patients with unipolar recurrent MDD according to DSM‐IV and a HDRS score of >18 were included in the study. A control group was individually matched for age, gender and level of education. Depressed patients performed equal to the control group on the Color and Word cards at both test occasions. However, the patients were impaired compared with the control group on the Color‐Word card task at both test occasions. Thus, the depressed patients showed no improvement of effortful attention/executive performance as a function of symptom reduction. The results indicate that the depressed patients showed impaired cognitive performance on cognitive demanding tasks when symptomatic and that this impairment prevailed after 6 months, despite significant improvement in their depressive symptoms.     

Proportion of general factor variance in a hierarchical multiple‐component measuring instrument: A note on a confidence interval estimation procedure

A confidence interval construction procedure for the proportion of explained variance by a hierarchical, general factor in a multi‐component measuring instrument is outlined. The method provides point and interval estimates for the proportion of total scale score variance that is accounted for by the general factor, which could be viewed as common to all components. The approach may also be used for testing composite (one‐tailed) or simple hypotheses about this proportion, and is illustrated with a pair of examples.     

Test–retest reliability of the soleus H-reflex excitability measured during human walking

The purpose of the study was to investigate with what accuracy the soleus H-reflex modulation and excitability could be measured during human walking on two occasions separated by days. The maximal M-wave (Mmax) was measured at rest in the standing position. During treadmill walking every stimulus elicited an M-wave of 25 ± 10% of Mmax in the soleus muscle and a supra-maximal stimulus elicited a maximal M-wave 60 ms after the first stimulus. Both Mmax during rest and during walking were later used for normalization. When normalized to resting Mmax, the peak reflex amplitude during walking was 5% lower on Day 2 than on Day 1 (p = .32). However, when the peak H-reflex was normalized to Mmax in every sweep, Day 2 showed a significant 15% lower amplitude (p = .037). The same pattern was found for the mean H-reflex. Spearman’s Rho was .92 when normalized to resting Mmax but .88 when normalized to Mmax in every sweep. The Pearson product was used to identify one participant at a time on Day 1 among all seven participants on Day 2. For both normalization procedures 5 of 7 participants were identified by this test. Since 5 of 7 participants were recognized between days, it must be recommended to use 10-15 participants for training or intervention studies as far as the H-reflex pattern of modulation during movement is concerned.     

On the assessment of the test–retest of Riding’s CSA: A commentary on Peterson, Deary, and Austin

Peterson, Deary, and Austin (2003) considered the reliability of the Cognitive Styles Analysis (CSA) (Riding, 1991). The CSA seeks to assess an individual’s position on each of two fundamental style dimensions – the Wholist-Analytic and the Verbal-Imagery dimensions. It presents a series of simple cognitive tasks, which the subjects may choose to process according to their preferred style. Performance on these test items is in terms of response times. The CSA comprises 40 items to assess the Wholist-Analytic and 48 for the Verbal-Imagery and typically takes 15–20 min to complete. It is intended to be suitable for a wide age and ability range, and applicable to a variety of contexts and cultures.The most important characteristic of any test of cognitive style is its temporal stability. Studies which attempt to establish test validity without definitive evidence of test reliability are lacking a basic foundation. Riding has not published any statistical data on the test–retest reliability of the CSA.Peterson et al. (2003) and Peterson (2003) claim to have carried out the primary evaluation of the CSA’s reliability. However we were the first to publish accurate test–retest reliability data on Riding’s CSA (Redmond, Mullally, & Parkinson, 2002).This brief report addresses the issue as to who initially established the unreliability of the CSA in the first place and why Peterson, Deary and Austin’s claims are misleading and unsubstantiated.     

The effect of item feedback on multiple‐choice test responses

This study examines the effects of item feedback on multiple‐choice test responses. In the first experiment, a positive effect was predicted under instructions advising a penalty for errors; a larger beneficial effect on female scores was expected, given the (presumed) tendency of female participants toward omission when uncertain. Feedback effects were either negligible or negative, and the expected interaction effect with gender was not found. The second experiment was a high‐powered replication of the feedback effect on errors, controlling for participants' ability. The discussion takes into account other evidence to state that recommendations of providing item feedback in the context of testing are neither theoretically nor empirically founded.     

Robust step‐down tests for multivariate independent group designs

A composite step‐down procedure, in which a set of step‐down tests are summarized collectively with Fisher's combination statistic, was considered to test for multivariate mean equality in two‐group designs. An approximate degrees of freedom (ADF) composite procedure based on trimmed/Winsorized estimators and a non‐pooled estimate of error variance is proposed, and compared to a composite procedure based on trimmed/Winsorized estimators and a pooled estimate of error variance. The step‐down procedures were also compared to Hotelling's T² and Johansen's ADF global procedure based on trimmed estimators in a simulation study. Type I error rates of the pooled step‐down procedure were sensitive to covariance heterogeneity in unbalanced designs; error rates were similar to those of Hotelling's T² across all of the investigated conditions. Type I error rates of the ADF composite step‐down procedure were insensitive to covariance heterogeneity and less sensitive to the number of dependent variables when sample size was small than error rates of Johansen's test. The ADF composite step‐down procedure is recommended for testing hypotheses of mean equality in two‐group designs except when the data are sampled from populations with different degrees of multivariate skewness.     

Ill-structured measurement designs in organizational research: implications for estimating interrater reliability

Organizational research and practice involving ratings are rife with what the authors term ill-structured measurement designs (ISMDs)--designs in which raters and ratees are neither fully crossed nor nested. This article explores the implications of ISMDs for estimating interrater reliability. The authors first provide a mock example that illustrates potential problems that ISMDs create for common reliability estimators (e.g., Pearson correlations, intraclass correlations). Next, the authors propose an alternative reliability estimator--G(q,k)--that resolves problems with traditional estimators and is equally appropriate for crossed, nested, and ill-structured designs. By using Monte Carlo simulation, the authors evaluate the accuracy of traditional reliability estimators compared with that of G(q,k) for ratings arising from ISMDs. Regardless of condition, G(q,k) yielded estimates as precise or more precise than those of traditional estimators. The advantage of G(q,k) over the traditional estimators became more pronounced with increases in the (a) overlap between the sets of raters that rated each ratee and (b) ratio of rater main effect variance to true score variance. Discussion focuses on implications of this work for organizational research and practice.     

A clusterwise simultaneous component method for capturing within‐cluster differences in component variances and correlations

This paper presents a clusterwise simultaneous component analysis for tracing structural differences and similarities between data of different groups of subjects. This model partitions the groups into a number of clusters according to the covariance structure of the data of each group and performs a simultaneous component analysis with invariant pattern restrictions (SCA‐P) for each cluster. These restrictions imply that the model allows for between‐group differences in the variances and the correlations of the cluster‐specific components. As such, clusterwise SCA‐P is more flexible than the earlier proposed clusterwise SCA‐ECP model, which imposed equal average cross‐products constraints on the component scores of the groups that belong to the same cluster. Using clusterwise SCA‐P, a finer‐grained, yet parsimonious picture of the group differences and similarities can be obtained. An algorithm for fitting clusterwise SCA‐P solutions is presented and its performance is evaluated by means of a simulation study. The value of the model for empirical research is illustrated with data from psychiatric diagnosis research.     

A pediatric screening instrument to detect problematic infant–parent interactions: Initial reliability and validity in a sample of high‐ and low‐risk infants

The purpose of this study was to determine the initial reliability and validity of a screening instrument developed to detect problematic interactions between infants and parents as part of a pediatric well‐baby exam. Participants included 117 infant–mother dyads (57 preterms and 60 full terms) assessed when infants were 6 to 9 months old. Mothers and infants were observed playing an interactional game such as peek‐a‐boo during the course of the pediatric exam. The game was scored for degree of interactional reciprocity using the Pediatric Infant Parent Exam (PIPE). Acceptable levels of interrater reliability were achieved. As predicted, higher risk infants and their mothers exhibited more problematic interactions than lower risk infants and their mothers. Results indicated that the PIPE was a reliable means of screening for interactional difficulties, that was sensitive to, but not synonymous with, neonatal health indices. ©2001 Michigan Association for Infant Mental Health.     

Using mixture models with known class membership to address incomplete covariance structures in multiple‐group growth models

Multi‐group latent growth modelling in the structural equation modelling framework has been widely utilized for examining differences in growth trajectories across multiple manifest groups. Despite its usefulness, the traditional maximum likelihood estimation for multi‐group latent growth modelling is not feasible when one of the groups has no response at any given data collection point, or when all participants within a group have the same response at one of the time points. In other words, multi‐group latent growth modelling requires a complete covariance structure for each observed group. The primary purpose of the present study is to show how to circumvent these data problems by developing a simple but creative approach using an existing estimation procedure for growth mixture modelling. A Monte Carlo simulation study was carried out to see whether the modified estimation approach provided tangible results and to see how these results were comparable to the standard multi‐group results. The proposed approach produced results that were valid and reliable under the mentioned problematic data conditions. We also present a real data example and demonstrate that the proposed estimation approach can be used for the chi‐square difference test to check various types of measurement invariance as conducted in a standard multi‐group analysis.     

An "open-transformed scale" for correcting ceiling effects and enhancing retest reliability: the example of pain

The use of closed scales (with anchors at each end) to measure pain was found to produce ceiling effects characterized by a deceleration of ratings toward the upper end of the scale. This was consistent with previous research. Apart from producing nonlinear functions, the closed scale also limited test-retest reliability because of subjects' tendencies to correct their distorted ratings in subsequent trials. However, an open-ended scale coupled with transformation of reported ratings into a decile scale virtually eliminated the ceiling effect, thus producing consistently linear functions and maximizing test-retest reliability. This finding may have implications for the measurement of other sensory and psychological phenomena, especially those in which the property evaluated varies in a continuous fashion.     

Examining time‐invariance in reliability in multi‐wave,multi‐indicator models: A covariance structure analysis approach accounting for indicator specificity

A covariance structure analysis method for testing time‐invariance in reliability in multiwave, multiple‐indicator models in outlined. The approach accounts for observed variable specificity and permits, in addition, estimation of reliability in terms of ‘pure’ measurement error variance. The proposed procedure is developed within a confirmatory factor analysis framework and illustrated with data from a cognitive intervention study.     

Mother–child interaction and sleep regulation in one‐year‐olds

The regulation of sleep–wake states is controlled not only by biological mechanisms but by care‐giving context as well. In this study the association between mother–child relationship and the infant's sleep was examined. Thirty‐seven 12‐month‐olds and their mothers participated in a 10‐minute laboratory play episode. The dyadic interaction was coded with the Early Parent–Child Relational Assessment (Clark, 1985) and with the Emotional Availability scales (Biringen, Robinson, & Emde, 1993). The child's sleep was measured at home with a small‐computerized activity monitor. Although mothers' behavior was not related to the child's sleep, infants who were more responsive in the play episode woke up more frequently compared to infants who were less involved in the interaction. The link between social‐emotional competency and fragmented sleep, among nonrisk infants, could be an age‐related phenomenon. ©2001 Michigan Association for Infant Mental Health.     

Social interdependence in close relationships: The actor–partner‐interdependence–investment model (API‐IM)

This article introduces the actor–partner‐interdependence–investment model (API‐IM) that was developed to add a dyadic perspective to Rusbult's investment model. The API‐IM is based on interdependence theoretical assumptions and the actor–partner interdependence model. Two studies were conducted to investigate the reliability of the API‐IM. Relationship satisfaction, investment size, quality of alternatives, and relationship commitment were assessed at both partners of 77 (Study 1) and 162 (Study 2) married and unmarried heterosexual couples. Path analyses that applied a structural equation modeling framework revealed a dyadic model that significantly predicts women's and men's commitment by actor effects of satisfaction, investments, and alternatives, and partner effects of satisfaction. Actor and partner effects of satisfaction were significantly moderated by relationship duration and marital status. Marital status also significantly moderated the actor effect of alternatives. The API‐IM supports the concept of social interdependence in close relationships, and it is discussed as a sound dyadic extension of the investment model. Copyright © 2012 John Wiley & Sons, Ltd.