Teachers' ratings of gross motor skills suffer from low concurrent validity

In this study an attempt was made to construct a reliable and valid unifactorial teachers' rating scale for gross motor ability. Study 1 (132 children from 3 to 7 years) revealed that reliability of the scale was acceptable and that the scale represented an unifactorial dimension. Two studies on concurrent validity of the scale with an experimental gross motor task (stepping-stone crossing), the unifactorial subtest Locomotion of the Test of Gross Motor Development and the subtest Balance of the Movement Assessment Battery for Children as criterion measures, did not produce acceptable validity coefficients. In both validation studies an age effect was found. It was concluded that factor specificity does not seem the answer to the usual low validity coefficients of multifactorial teachers' rating scales. An alternative approach is suggested in which the assessment of functional activities in daily situations is stressed. Finally, the inclusion of atypical groups in random samples, which is common practice in research on concurrent validity of screening instruments for children's motor problems, is discussed.     

The value of relative-time-spent ratings in task-oriented job analysis

Recent studies have called for the abandonment of the relative-time-spent scale in task inventories. This recommendation is based on findings that the job profile created with the scale data is highly correlated with the profile created from a much simpler “Do you perform this task?” checklist. We examined this issue using 3 inventories and 42 jobs (N=2252). Profile correlations were computed on only the tasks actually performed by incumbents to avoid possibly inflated rs due to including irrelevant tasks. The specificity of task inventory items was proposed as an explanation for the high correlation between the two job profiles. Specificity of items was examined by looking at both the type (job duties versus tasks) and the amount (number of items in job profile and average number of items relevant to each job) of items used in the inventory. Correlations between time spent and checklist profiles were in the .80's and .90's regardless of the number of irrelevant tasks or the specificity of tasks. We agree with previous military research and conclude that the relative-time-spent scale has limited incremental utility beyond a dichotomous checklist.     

Operating characteristic analysis of attribute ratings

The analysis of receiver operating characteristics as employed in psychophysics is suggested as a way of obtaining several useful measures in the context of attribute ratings. These include the difference between two stimuli on the attribute, the tendency for Ss to favor one pole of the rating scale, width of rating categories, and equal interval properties of the rating scale. The underlying measurement model is described along with means of evaluating its basic assumptions.     

The perception of concurrent sound objects in harmonic complexes impairs gap detection

Since the introduction of the concept of auditory scene analysis, there has been a paucity of work focusing on the theoretical explanation of how attention is allocated within a complex auditory scene. Here we examined signal detection in situations that promote either the fusion of tonal elements into a single sound object or the segregation of a mistuned element (i.e., harmonic) that "popped out" as a separate individuated auditory object and yielded the perception of concurrent sound objects. On each trial, participants indicated whether the incoming complex sound contained a brief gap or not. The gap (i.e., signal) was always inserted in the middle of one of the tonal elements. Our findings were consistent with an object-based account in which perception of two simultaneous auditory objects interfered with signal detection. This effect was observed for a wide range of gap durations and was greater when the mistuned harmonic was perceived as a separate object. These results suggest that attention may be initially shared among concurrent sound objects thereby reducing listeners' ability to process acoustic details belonging to a particular sound object. These findings provide new theoretical insight for our understanding of auditory attention and auditory scene analysis.     

A critical analysis of students' teacher ratings

Analysis of data obtained from 258 second-year undergraduate students revealed that sex differences are not associated with differences in students' teacher ratings. Significant differences in ratings were found to be associated with differences in academic performance and with differences in preference for anonymity when submitting ratings. These findings suggest that students' ratings of teaching performance suffer from significant validity problems and that such ‘performance indicators’ are likely to provide misleading information.     

The statistical analysis of data from small groups

The authors elaborate the complications and the opportunities inherent in the statistical analysis of small-group data. They begin by discussing nonindependence of group members' scores and then consider standard methods for the analysis of small-group data and determine that these methods do not take into account this nonindependence. A new method is proposed that uses multilevel modeling and allows for negative nonindependence and mutual influence. Finally, the complications of interactions, different group sizes, and differential effects are considered. The authors strongly urge that the analysis model of data from small-group studies should mirror the psychological processes that generate those data.     

The joint analysis of direct ratings,pairwise preferences,and dissimilarities

In studies involving judgments of similarity or dissimilarity, a variety of other variables may also be measured. Examples might be direct ratings of the stimuli, pairwise preference judgments, and physical measurements of the stimuli with respect to various properties. In such cases, there are important advantages to joint analyses of the dissimilarity and collateral variables. A variety of models are described for relating these and algorithms described for fitting these to data. A number of hypothesis tests are developed and an example offered.The research reported here was supported by grant number APA 320 to the author by the National Research Council of Canada.     

Self-serving bias effects on job analysis ratings

The purpose of this study was to investigate whether worker-oriented job analysis importance ratings were influenced by subject matter experts' (SME) standing (as measured by self-rated performance) on a competency. This type of relationship (whereby SMEs indicate that the traits they have are important for successful job performance) is an example of the self-serving bias (which is widely described in the social cognition literature and rarely described in the industrial/organizational psychology literature). An archival dataset covering 57 clerical and technical occupations with 26,682 participants was used. Support was found for the relationship between self-rated performance and importance ratings. Significant relationships (typically in the .30s) were observed for all 31 competencies that were studied. Controls were taken to account for common method bias and differences in the competencies required for each of the 57 occupations. Past research has demonstrated the effects of the self-serving bias on personality-based job analysis ratings. This study was the first to extend these findings to traditional job analysis, which covers other competencies in addition to personality. In addition, this study is the first to use operational field data instead of laboratory data.     

Agreement of adolescent ratings with mother ratings and teacher ratings of ADHD symptom groups: A correlated trait-correlated method minus one analysis

This study examined the level of agreement of adolescent ratings with mother ratings, and adolescent ratings with teacher ratings of the inattention (IA) and hyperactivity/impulsivity (HI) symptom groups of ADHD. A total of 214 adolescents provided self-ratings of IA and HI, and their IA and HI were also rated by their mothers and teachers. The correlated trait-correlated method minus one model was applied, with adolescent ratings as the reference method, and the other two ratings as the non-reference methods. The findings indicated no additional variance in adolescent ratings for IA and HI that could not be accounted by mother ratings of IA and HI, respectively. In contrast, there was additional variance in adolescent ratings for IA and HI that could was not accounted by teacher ratings of IA and HI, respectively. The findings suggest that when diagnosing ADHD in adolescents, their reports of their own ADHD behaviors are not needed when mother reports of such behaviors are used.     

Error detection processes in statistical problem solving

Problem solvers' error detection processes were studied by instructing 16 subjects to think aloud when solving two statistical problems. The evaluative episodes occurring in subjects' protocols were analyzed into Affirmative evaluation, Direct error-hypotheses, Error suspicion, and Standard check episodes, the last three of which are assumed to cover all main types of error detection processes. Most errors (78%) were found to have contributed to a solution part that triggered some evaluative episode. However, only one-third of the undetected errors had contributed to such a solution part. The Standard check episodes, seen as centrally-invoked, only led to the detection of few errors in proportion to the number of times they were performed. Evidence was found for two types of spontaneous error detections, one occurring abruptly and the other as a result of a more elaborated error detection process, initiated by the solver perceiving the solution as dissatisfying or strange. The perception of a symptom was a fairly reliable source of information about errors. However, subjects often did not manage to detect the error after having noticed a symptom. The closer a relevant Error suspicion episode followed an error, the greater was the probability of detecting the error. Good problem solvers detected a higher proportion of their errors compared to poor problem solvers, probably due to differences in the processes leading up to the triggering of an evaluative episode rather than to differences after the episode had been triggered.     

The use of factor analysis in the statistical analysis of multiple time series

Factor analysis has been proposed and used as a method of statistical analysis of several measurements made on one individual repeatedly over time. This paper discusses some difficulties in applying factor analysis to multiple time series and attempts to indicate to what extent such methods can accomplish the goals of time series analysis. Some other methods are suggested.Research sponsored by contract AF41(657)-214 between the USAF School of Aerospace Medicine and Teachers College, Columbia University.     

The effect of judges' attitudes on ratings of attitude statements. A theoretical analysis

Judgments of attitude statements with the method of equal-appearing intervals have been found to vary as a function of the judges' attitudes. In this paper explanations of the relationship between judges' attitudes and judgments of attitude statements in terms of models of psychophysical judgment are discussed. It is argued that psychophysical models such as adaptation-level theory, the range-frequency model, and the ‘rubber-band’ model and its derivations, cannot account satisfactorily for judges' performance of the attitude rating task in a great number of studies. The reason for this failure, it is argued, is that the stimulus series employed in the psychophysical judgment research on which these models are based typically varied only on the dimension being judged. The sets of statements judged in attitude rating studies, however, vary not only on the dimension of interest (favourability—Unfavourability) but also on a number of other dimensions. It is suggested that this incidental stimulus variation of attitude statements may account for the failure of psychophysical models to predict accurately the performance of judges in the attitude rating task. It is argued that if principles which could account for the effects of this incidental stimulus variation on attitude ratings could be incorporated into psychophysical models, the predictive qualities of these models could be improved considerably. One such model is discussed.     

The relevance of specific conversational behaviors to ratings of social skill: An experimental analysis

Research on the social effectiveness of specific conversational behaviors is reviewed and a new study is presented. Eight versions of the same tape-recorded conversation between a man and a woman were prepared in a 2×2×2 design, where the woman systematically varied her conversation either high or low on questions, compliments, and 4-sec latencies. Subjects listened to one of these tapes and rated the female's social skill. Results of the major social skill variable showed that high questions and high compliments elicited higher skill ratings. The effect of low latencies was only marginally significant. While other factors such as personal appearance or self-evaluation differences may also be important, this study suggests that it may be valuable to include assessment and training relevant to compliments and questions (and possibly latencies) in programs for women who are socially anxious.     

A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings

How should we measure metacognitive (“type 2”) sensitivity, i.e. the efficacy with which observers’ confidence ratings discriminate between their own correct and incorrect stimulus classifications? We argue that currently available methods are inadequate because they are influenced by factors such as response bias and type 1 sensitivity (i.e. ability to distinguish stimuli). Extending the signal detection theory (SDT) approach of Galvin, Podd, Drga, and Whitmore (2003), we propose a method of measuring type 2 sensitivity that is free from these confounds. We call our measure meta-d′, which reflects how much information, in signal-to-noise units, is available for metacognition. Applying this novel method in a 2-interval forced choice visual task, we found that subjects’ metacognitive sensitivity was close to, but significantly below, optimality. We discuss the theoretical implications of these findings, as well as related computational issues of the method. We also provide free Matlab code for implementing the analysis.