The reliability coefficient

The reliability coefficient is unlike other measures of correlation in that it is a quantitative statement of an act of judgment,—usually the test maker's, — that the things correlated are similar measures. Attempts to divorce it from this act of judgment are misdirected, just as would be an attempt to eliminate judgment of sameness of function of items when a test is originally drawn up. A coefficient of cohesion, entirely devoid of judgment, measuring the singleness of test function is proposed as an essential datum with reference to a test, but not as a substitute for the similar-form reliability co-efficient.     

Objectivity and reliability

Scanlon’s Being Realistic about Reasons is a beautiful book – sleek, sophisticated, and programmatic. One of its key aims is to demystify knowledge of normative and mathematical truths, realistically construed – i.e. construed, roughly, as being true relevantly independent of minds and languages, when interpreted at face-value. In this article, I develop an epistemological problem that Scanlon fails to explicitly address. I argue that his ‘metaphysical pluralism’ can be understood as a response to that problem. However, it resolves the problem only if it undercuts the objectivity of normative and mathematical inquiry.     

The reliability of romantic love: A reliability generalization meta‐analysis

The present study is a reliability‐generalization meta‐analysis of 5 commonly used measures of romantic love: Rubin’s Loving and Liking scales, the Love Attitudes Scale, the Love Attitudes Scale–Short Form, the Passionate Love Scale, and the Triangular Love Scale. Data from 127 studies representing 38,132 participants provided internal consistency reliability estimates. The studies provide information on the average score reliability across studies and the effect of sample characteristics on score reliability across measures. The reliability of scores from several measures proved to be susceptible to the influence of sample characteristics, most notably, the ethnicity of the sample. The discussion provides recommendations to researchers and focuses on the ramifications of score reliability on love research.     

Components of attentional set-switching

A series of distinct event-related potentials (ERPs) have been recorded from the scalp of human subjects as they switch from one task to another. It is possible that task switching may depend on different mechanisms depending on whether the switch requires a change in attentional set, in other words the redirecting of attention to different aspects of a sensory stimulus, or whether it requires a change in intentional set, in others words a change in the way that responses are selected. To address this issue, the current study recorded ERPs while subjects switched between attentional sets and the results were compared with those of a previous investigation in which subjects switched between intentional sets. Subjects selected stimuli according to two conflicting attentional sets, each emphasizing one visual stimulus dimension (colour, shape). Pairs of stimuli, only one of which was to be attended, were presented for between eight and seventeen trials then either a switch or a stay cue was shown. The switch cue instructed subjects to switch from the current attentional set to the other set, while the stay cue instructed subjects to maintain the current set. Comparing ERPs time-locked to the switch and stay cues revealed neural correlates of the initiation of a task switch. Comparing the ERPs time locked to the first stimuli after either stay or switch cues identified neural correlates of the implementation of a task switch. A similar modulation over parietal electrodes was seen when subjects were switching between either attentional or intentional sets. While an intentional set switch began with a medial frontal modulation, attentional set switching began with a lateral frontal modulation. Implementing a new attentional set was associated with modulation of relatively early visual potentials, while implementing a new intentional set was associated with modulation of later response-related potentials. The results confirm that task switching consists of a number of constituent processes which may be taxed to different degrees depending on whether a task-switch paradigm requires subjects to change the way in which they select stimuli or responses.     

Components of visual memory

Visual recognition memory for a sequence of non-verbalized patterns is shown to have a large and clearly defined recency effect. This recency effect occurs with random list lengths and therefore cannot be due to differential processing of the end items. The effect is completely removed by just 3 s of mental arithmetic but survives for at least 10 s over unfilled intervals. Recognition memory for patterns at other serial positions is slower, less accurate, and shows no primacy effect; performance at these earlier serial positions is dependent upon the time for which patterns are initially presented, but is unaffected by the duration of the retention interval, mental arithmetic, and the time between patterns on initial presentation. These findings provide evidence that visual memory has two components that are closely analogous to the short-term (STM) and long-term (LTM) components of verbal memory. Visual STM, here called visualization, has a capacity of one pattern, cannot be activated LTM, and does not seem to be the gateway to LTM.     

Components of metaphoric processing

Three experiments examined whether sentences interpretable as metaphorical and literal expressions differed on three components of processing: perceptual decoding, sense selection, and integration of terms. In Experiments 1 and 2 metaphorical words were identified more readily than literal words on separate tests of perceptual identification and word recognition. In Experiment 3 the conveyed meaning of a metaphor was not recalled better than a literal interpretation of the same target sentence. It is concluded that metaphorical and literal sentences utilize separate perceptual and selectional decoding strategies, but do not differ with respect to comprehension processes once metaphorical and literal referents are instantiated. Discussion is given as to whether these differences constitute a separate metaphor strategy.Shinjo and Myers also described a fourth component process,retrieval, involved in the foregrounding of prime words used in their experiments.     

Bootstrapping and knowledge of reliability

This is a critical discussion of a paper on the problem of bootstrapping by Jose Zalabardo.

Components of syllogistic reasoning

The present research sought to understand the components of syllogistic reasoning used in a syllogistic evaluation task. In this task, subjects must indicate whether a conclusion such as “Some Yale professors are humbugs” is definitely true, or never true of a set of premises such as “Some humbugs study syllogistic reasoning; some Yale professors study syllogistic reasoning”. A modified form of componential analysis (Sternberg 1977, 1978) was used to decompose the syllogistic evaluation task with abstract content into encoding and encoding plus combination subtasks. The response-choice data from these subtasks were used to provide (a) direct tests of a proposed theory of syllogistic reasoning, and in particular, of its assumptions about sources of error in syllogistic reasoning; and (b) direct inferences regarding the representation of relations between the subject and predicate of the premises as encoded and combined. The results supported a proposed transitive-chain model of syllogistic reasoning.     

Test-retest reliability of psychophysiological assessment

Thirty subjects participated in a study of the test-retest reliability of five psychophysiological assessment procedures: habituation. Quiz Electrocardiogram, anger imagery, aversive slides and startle. The procedures were presented in identical fashion, 2 weeks apart. Skin conductance level and response, heart rate, vasomotor response, skin temperature, frontalis and forearm electromyogram, respiration rate. and systolic and diastolic blood pressure were recorded continuously. The following analyses were performed on Test-Retest data: correlations, baseline changes, profile similarity, and stability of ranked standardized physiological scores. Significant but modest Test-Retest correlations were found for skin conductance level and response, skin temperature, respiration rate, heart rate and systolic blood pressure. Absolute scores were more often stable than change scores. All baselines were stable from Test to Retest. Four baselines and four test procedures had Test-Retest Profile Similarity Index scores indicating reliability. Chi-square analyses of Retest ranks of standardized physiological scores showed that the two most and least responsive measures accounted for Test-Retest stability. Conclusions were: (a) Psychophysiological response patterns are reliable for baselines and procedures, and single psychophysiological measures (especially absolute scores) are moderately stable; (b) Traditional (correlational) estimates of the reliability of single psychophysiological responses are limited by individual-response stereotypy, but multivariate analysis of response pattern provides a better index of reliability.     

Exploring fantasies of TAT reliability

It has been argued that low TAT stability estimates are an artifact of retest instructions to be original and creative. This view was supported in a recent study by Winter and Stewart (1977) who found that test-retest reliability was higher when subjects were told on the second occasion to write the same stories than when they were told to write different stories. This study attempted to replicate Winter and Stewart's results and also to determine whether the similarity of stories did vary as hypothesized. The replication was only partially successful. More importantly, it was concluded that while instructions to write similar stories did improve reliability, the increase reflected only subjects' memory capabilities and not greater stability in motive expression. While true TAT stability may still be commonly underestimated, altering retest instructions to improve reliability does not appear to be the solution.     

The reliability of component scores

A method is given for determining the reliability of each of the components resulting from a factor analysis by the principal axis method.On leave for military service.     

The reliability of speeded tests

Some methods are presented for estimating the reliability of a partially speeded test without the use of a parallel form. The effect of these formulas on some test data is illustrated. Whenever an odd-even reliability is computed it is probably desirable to use one of the formulas noted in Section 2 of this paper in addition to the usual Spearman-Brown correction. Since the formulas given here involve the mean and the standard deviation of the “number unattempted score,” a method is given in Section 4 for computing this mean and standard deviation from item analysis data. If the item analysis data are available, this method will save considerable time as compared with rescoring answer sheets.     

A note on reliability

A formula for internal consistency reliability is developed within the framework of the analysis of variance. The test items are assumed to be homogeneous, but may have any weights. Data needed for computation are the student test scores, and the total number of items answered so as to have the same weight. It is shown that this formula reduces to the Kuder-Richardson (21) for item weights of one and zero. Some empirical validation is offered.Social Security Board (on military leave.)     

Stable unstable reliability theory

Classical reliability theory assumes that individuals have identical true scores on both testing occasions, a condition described as stable. If some individuals' true scores are different on different testing occasions, described as unstable, the estimated reliability can be misleading. A model called stable unstable reliability theory (SURT) frames stability or instability as an empirically testable question. SURT assumes a mixed population of stable and unstable individuals in unknown proportions, with w(i) the probability that individual i is stable. w(i) becomes i's test score weight which is used to form a weighted correlation coefficient r(w) which is reliability under SURT. If all w(i) = 1 then r(w) is the classical reliability coefficient; thus classical theory is a special case of SURT. Typically r(w) is larger than the conventional reliability r, and confidence intervals on true scores are typically shorter than conventional intervals. r(w) is computed with routines in a publicly available R package.     

Estimation of the reliability of ratings

A procedure for estimating the reliability of sets of ratings, test scores, or other measures is described and illustrated. This procedure, based upon analysis of variance, may be applied both in the special case where a complete set of ratings from each ofk sources is available for each ofn subjects, and in the general case wherek ₁,k ₂, ...,k _n ratings are available for each of then subjects. It may be used to obtain either a unique estimate or a confidence interval for the reliability of either the component ratings or their averages. The relations of this procedure to others intended to serve the same purpose are considered algebraically and illustrated numerically.The writer wishes to acknowledge the helpful comments and suggestions of Professors E. E. Cureton, Harold Gulliksen, and E. F. Lindquist.     

Components of events and processes

Rips and Estin (1998) provided evidence that mental events such as dreaming are more homogeneous than physical events such as checking out a book; that is, their parts are more difficult to distinguish. In their experiment, participants listed more distinctive properties for the parts of physical events than for the parts of mental events. However, the physicality of stimuli was confounded with temporal aspects. Mental stimuli tended to be processes, and physical stimuli, events. This study tested homogeneity with new stimuli separating out the factors of physicality and aspect. Consistently, both physicality and aspect had significant effects on the perceived homogeneity of activities, as measured by the number of listed parts, the number of distinctive properties of each part, and homogeneity ratings. The study shows that homogeneity is strongly influenced by aspect but that physicality remains a robust factor for homogeneity, even after taking aspect into account.