Estimation of reliability for multiple‐component measuring instruments in test–retest designs

A covariance structure modelling method for the estimation of reliability for composites of congeneric measures in test–retest designs is outlined. The approach also allows an approximate standard error and confidence interval for scale reliability in such settings to be obtained. The procedure further permits measurement error components due to possible transient condition influences to be accounted for and evaluated, and is illustrated with a pair of examples.     

Examining change in maximal reliability for multiple‐component measuring instruments

A method for examining change in maximal reliability for pre‐specified sets of congeneric measures when developing a multi‐component instrument is outlined. The approach is applicable for purposes of estimation and testing of gain or loss in the maximal reliability coefficient as a consequence of adding or dropping one or more measures from a homogeneous composite with uncorrelated errors, as well as when one is concerned with optimal component choice for highest increase or correspondingly smallest drop in maximal reliability. The method is compared with a procedure for ascertaining change in unweighted sum score reliability, and implications for instrument construction and revision are discussed. The approach is illustrated with a numerical example.     

Studying group and time invariance in maximal reliability for multiple‐component measuring instruments via covariance structure modelling

A method for examining invariance in maximal reliability for weighted combinations of congeneric measures is described. The approach is developed within the framework of covariance structure modelling and allows one to ascertain whether a multi‐component instrument consisting of homogeneous measures is associated with the same minimal relative error variance in distinct populations or over time. The procedure yields as a by‐product an interval measure of discrepancy in maximal reliability across independent groups or assessment occasions, and is illustrated with two examples.     

Reliability of multiple‐component measuring instruments: Improved evaluation in repeated measure designs

A covariance structure analysis method for improved point and interval estimation of composite reliability in repeated measure designs is outlined that accounts for specificity variance. The approach also permits the testing of time‐invariance in reliability of multiple‐component instruments in terms of the ratio of ‘pure’ measurement error variance to observed scale score variance. In addition, the procedure allows interval estimation of the difference in composite reliability coefficients across assessment occasions. The method described is illustrated with data from a cognitive intervention study.     

Recommendations for reporting multiple‐baseline designs across participants

Two recommendations are offered for reporting multiple‐baseline designs across participants. These recommendations will better enable readers to (a) distinguish concurrent from nonconcurrent multiple‐baseline designs and (b) determine the temporal order in which sessions were conducted in concurrent multiple‐baseline designs. Copyright © 2005 John Wiley & Sons, Ltd.     

Polytomous multilevel testlet models for testlet‐based assessments with complex sampling designs

A pplications of standard item response theory models assume local independence of items and persons. This paper presents polytomous multilevel testlet models for dual dependence due to item and person clustering in testlet‐based assessments with clustered samples. Simulation and survey data were analysed with a multilevel partial credit testlet model. This model was compared with three alternative models – a testlet partial credit model (PCM), multilevel PCM, and PCM – in terms of model parameter estimation. The results indicated that the deviance information criterion was the fit index that always correctly identified the true multilevel testlet model based on the quantified evidence in model selection, while the Akaike and Bayesian information criteria could not identify the true model. In general, the estimation model and the magnitude of item and person clustering impacted the estimation accuracy of ability parameters, while only the estimation model and the magnitude of item clustering affected the item parameter estimation accuracy. Furthermore, ignoring item clustering effects produced higher total errors in item parameter estimates but did not have much impact on the accuracy of ability parameter estimates, while ignoring person clustering effects yielded higher total errors in ability parameter estimates but did not have much effect on the accuracy of item parameter estimates. When both clustering effects were ignored in the PCM, item and ability parameter estimation accuracy was reduced.     

A Monte Carlo evaluation of masked visual analysis in response‐guided versus fixed‐criteria multiple‐baseline designs

We developed masked visual analysis (MVA) as a structured complement to traditional visual analysis. The purpose of the present investigation was to compare the effects of computer‐simulated MVA of a four‐case multiple‐baseline (MB) design in which the phase lengths are determined by an ongoing visual analysis (i.e., response‐guided) versus those in which the phase lengths are established a priori (i.e., fixed criteria). We observed an acceptably low probability (less than .05) of false detection of treatment effects. The probability of correctly detecting a true effect frequently exceeded .80 and was higher when: (a) the masked visual analyst extended phases based on an ongoing visual analysis, (b) the effects were larger, (c) the effects were more immediate and abrupt, and (d) the effects of random and extraneous error factors were simpler. Our findings indicate that MVA is a valuable combined methodological and data‐analysis tool for single‐case intervention researchers.     

The measurement of reliability of social relations components from round‐robin designs

The social relations model (SRM) is a useful tool for measuring relationship effects, defined as the unique perceptions or behaviors of 2 people. The sources of variance in SRM studies are persons (actors and partners), groups, and items; the relationship effect is defined as the actor–partner interaction. By removing variance because of persons and groups, a measure of a “pure” relationship effect is obtained. In this article, generalizability theory (G Theory) is applied to estimate the reliability of SRM components from round‐robin data structures. Using G Theory, reliability formulas for actor, partner, group, and relationship are developed and interpretations for the reliability estimates are provided. The authors also discuss how these formulas can be used in both planning and interpreting results from relationship research.     

Estimation of maximal reliability: A note on a covariance structure modelling approach

A one‐step covariance structure analysis procedure for estimation of maximal reliability of linear composites with congeneric measures is outlined. The approach is readily employed within a single modelling session using popular covariance structure analysis software, and permits simultaneous estimation of the optimal measure weights with standard errors. The method is illustrated by a numerical example.     

A multiple‐stimulus‐without‐replacement assessment for sexual partners: Purchase task validation

The current study developed and tested a multiple‐stimulus‐without‐replacement (MSWO) assessment for potential sexual partners for use in research on human immunodeficiency virus. College students (N = 41) first completed an MSWO assessment and then completed a hypothetical purchase task for encounters with partners identified by the MSWO as high, median, and low preference. Overall, hypothetical purchase task responding was consistent with that from the MSWO, in that the highest valuation was observed for the high‐preference partner and the lowest for the low‐preference partner. Potentially interesting individual differences in purchase task responding, however, were obtained; some subjects showed differentiated responding among the 3 preference levels (n = 15), whereas others similarly valued high‐ and median‐preference partners (n = 5), and others similarly valued low‐ and median‐preference partners (n = 18).     

Evaluation of multiple‐alternative prompts during tact training

This study compared 2 methods of fading prompts while teaching tacts to 3 individuals who had been diagnosed with autism spectrum disorder (ASD). The 1st method involved use of an echoic prompt and prompt fading. The 2nd method involved providing multiple‐alternative answers and fading by increasing the difficulty of the discrimination. An adapted alternating‐treatments design showed that both procedures were more effective than a no‐intervention control condition. Providing multiple alternatives did not increase error rates or teaching time, and better maintenance was shown for tacts taught with the multiple‐alternative prompt.     

When mixed options are preferred in multiple‐trial decisions

We report three studies showing that in prospective multiple‐trial decisions people often select a mix of sure and risky options over pure bundles of either option. Such a preference is not ‘rational’ because a mixed option cannot be the EV‐maximizing choice. Experiment 1 confirmed a mixed‐option preference for gains but not for losses. Showing a graph of the multiple‐trial outcome distribution reduced but did not eliminate this effect, suggesting that it is not due purely to a failure to aggregate correctly over the multiple trials. Experiment 2 replicated the mixed option preference using a wider range of problems. Experiment 3 compared choices in the trinary choice conditions used in Experiments 1 and 2 with binary choices between pairs of the multiple‐trial sure, mixed, and risky options. In the binary choice condition the mixed option was no longer the modal choice, suggesting that the strong mixed option preference found in the trinary choice conditions is mainly due to a compromise effect. However, the binary choice probabilities did show violations of strong stochastic transitivity in a pattern that suggested a slight bias toward the mixed option. Copyright © 2006 John Wiley & Sons, Ltd.     

Proportion of general factor variance in a hierarchical multiple‐component measuring instrument: A note on a confidence interval estimation procedure

A confidence interval construction procedure for the proportion of explained variance by a hierarchical, general factor in a multi‐component measuring instrument is outlined. The method provides point and interval estimates for the proportion of total scale score variance that is accounted for by the general factor, which could be viewed as common to all components. The approach may also be used for testing composite (one‐tailed) or simple hypotheses about this proportion, and is illustrated with a pair of examples.     

Rank‐based permutation approaches for non‐parametric factorial designs

Inference methods for null hypotheses formulated in terms of distribution functions in general non‐parametric factorial designs are studied. The methods can be applied to continuous, ordinal or even ordered categorical data in a unified way, and are based only on ranks. In this set‐up Wald‐type statistics and ANOVA‐type statistics are the current state of the art. The first method is asymptotically exact but a rather liberal statistical testing procedure for small to moderate sample size, while the latter is only an approximation which does not possess the correct asymptotic α level under the null. To bridge these gaps, a novel permutation approach is proposed which can be seen as a flexible generalization of the Kruskal–Wallis test to all kinds of factorial designs with independent observations. It is proven that the permutation principle is asymptotically correct while keeping its finite exactness property when data are exchangeable. The results of extensive simulation studies foster these theoretical findings. A real data set exemplifies its applicability.     

Connection‐based and object‐based grouping in multiple‐object tracking: A developmental study

Developmental research on Gestalt laws has previously revealed that, even as young as infancy, we are bound to group visual elements into unitary structures in accordance with a variety of organizational principles. Here, we focus on the developmental trajectory of both connection‐based and object‐based grouping, and investigate their impact on object formation in participants, aged 9–21 years old (N = 113), using a multiple‐object tracking paradigm. Results reveal a main effect of both age and grouping type, indicating that 9‐ to 21‐year‐olds are sensitive to both connection‐based and object‐based grouping interference, and tracking ability increases with age. In addition to its importance for typical development, these results provide an informative baseline to understand clinical aberrations in this regard.

Statement of contribution

What is already known on this subject?

• The origin of the Gestalt principles is still an ongoing debate: Are they innate, learned over time, or both?
• Developmental research has revealed how each Gestalt principle has its own trajectory and unique relationship to visual experience.
• Both connectedness and object‐based grouping play an important role in object formation during childhood.

What does this study add?

• The study identifies how sensitivity to connectedness and object‐based grouping evolves in individuals, aged 9–21 years old.
• Using multiple‐object tracking, results reveal that the ability to track multiple objects increases with age.
• These results provide an informative baseline to understand clinical aberrations in different types of grouping.

     

A component analysis of a parent‐conducted multi‐component treatment for food selectivity

Treatment packages including differential reinforcement of alternative (DRA) behavior and escape prevention in the form of a non‐removal of the spoon procedure have been shown to successfully increase food consumption. However, when these treatment components are introduced simultaneously, the treatment component(s) responsible for behavior change cannot be determined. The purpose of this study was to conduct a sequential component analysis of the following treatment components: Bite fading, manipulation of reinforcer magnitude, and escape prevention. For two participants, food consumption did not increase until after escape prevention was introduced. For one participant, increased food consumption was observed after the magnitude of reinforcement was increased; therefore, escape prevention was not necessary. Results were maintained at a 12‐week follow‐up for all participants. Copyright © 2010 John Wiley & Sons, Ltd.     

Further refinement of video‐based brief multiple‐stimulus without replacement preference assessments

We compared the results of a brief video‐based multiple‐stimulus without replacement preference assessment with no access to chosen activities (MSWO‐NO) to the results of the same assessment with access (MSWO‐WA) with four children with autism. We also compared instructor rankings of activities to MSWO‐WA results. Strong to moderate correlations between MSWO‐NO and MSWO‐WA assessment results were found across all participants. The correlation between MSWO‐WA and instructor rankings ranged from strong to low across all participants. Implications and directions for future research are discussed.     

The effect of item feedback on multiple‐choice test responses

This study examines the effects of item feedback on multiple‐choice test responses. In the first experiment, a positive effect was predicted under instructions advising a penalty for errors; a larger beneficial effect on female scores was expected, given the (presumed) tendency of female participants toward omission when uncertain. Feedback effects were either negligible or negative, and the expected interaction effect with gender was not found. The second experiment was a high‐powered replication of the feedback effect on errors, controlling for participants' ability. The discussion takes into account other evidence to state that recommendations of providing item feedback in the context of testing are neither theoretically nor empirically founded.     

The reliability of component scores

A method is given for determining the reliability of each of the components resulting from a factor analysis by the principal axis method.On leave for military service.