SELECTING A COST-EFFECTIVE SCREENING MEASURE FOR THE ASSESSMENT OF PRESCHOOL SOCIAL WITHDRAWAL

The utility of teacher judgment (rankings, ratings) and peer judgment (sociometric ratings) were studied as screening variables for preschool social withdrawal/responsiveness. Observational measures of preschoolers' interaction were used as the validation criterion based upon theoretical, empirical, and practical considerations related to development of a multipurpose behavioral assessment system. Results indicated that teacher rankings of students' verbal interaction frequencies (1) were most highly correlated to interaction rate in two preschool settings, (2) had test-retest reliability consistently above Rho = .80, and (3) were useful in that 77% of teachers using this procedure could identify their least socially responsive student within five rankings. The peer nomination sociometric rating was the least reliable measure at retest (Rho = .35) and showed consistently the lowest correlation with observational indices of interaction partner preferences, e.g., reciprocal rate (Rho = .29 to .39). The tendency for relationships among screening measures and the criterion to increase at retest suggested a training effect on screening accuracy. The use of teacher rankings in combination with observational measures to confirm selection accuracy and for monitoring treatment progress is discussed as a cost-effective behavioral assessment procedure for preschool social withdrawal/responsiveness.     

The predictive validity of peer assessment in law enforcement: a 6 year follow-up

This study examined the relations between peer assessment and long-term law enforcement/job performance. Sixty-nine law enforcement cadets completed peer rankings of likelihood of success in job performance. These peer rankings were significantly correlated with supervisor ratings, as well as absenteeism and retention, after 1 year of employment. In the present study, both subjective and objective job performance data were collected from a variety of employing job performance agencies after 6 years of employment. Peer rankings were still significantly correlated (r=.40, p<.05) with retention after 6 years, but were not significantly correlated to any job performance index. These results support the potential usefulness of peer assessment data even after several years have passed. © 1998 John Wiley & Sons, Ltd.     

Of Nobel class: A citation perspective on high impact research authors

The purpose of this paper was to determine if quantitative rankings of highly cited research authors confirm Nobel prize awards. Six studies covering different time periods and author sample sizes were reviewed. The number of Nobel laureates at the time each study was published was tabulated, as was the number of high impact authors who later became laureates. The Nobelists and laureates-to-be were also compared with non-Nobelists to see if they differed in terms of impact and productivity. The results indicate that high rankings by citation frequency identify researchersof Nobel class — that is, a small set of authors that includes a high proportion of actual Nobelists and laureates-to-be. Also, the average impact (citations per author) of Nobelists and laureates-to-be is sufficiently high to distinguish them from non-Nobelists in these rankings. In conclusion, a simple, quantitative, and objective algorithm based on citation data can effectively corroborate —and even forecast — a complex, qualitative, and subjective selection process based on human judgement.     

Information and preference in partial orders: A bimatrix representation

This paper presents a bimatrix structure for examining ordinal partial rankings. A set of axioms is given similar to those of Kemeny and Snell (1962) and Bogart (1973), which uniquely determines the distance between any pair of such rankings. Thel ¹ norm is shown to satisfy this set of axioms, and to be equivalent to the Kemeny and Snell distance on their subspace of weak orderings. Consensus formation is discussed.This research was supported by a NSERC grant A8966.     

What Goes Up Apparently Needn't Come Down: Asymmetric Predictions of Ascent and Descent in Rankings

In eight studies, we document an upward mobility bias, or a tendency to predict that a rise in rankings is more likely than a decline. This asymmetry was observed in predictions of classroom performance, NBA and NFL standings, business school rankings, and employee performance rankings. The bias was found for entities people care about and want to see improve their standing, as well as entities in which people are not invested. It appears to result from people's tendency to give considerable weight to a focal agent's intentions and motivation, but to give less weight to the intentions of competitors and other factors that would thwart the focal agent's improvement. We show that this bias is most pronounced for implicit incremental theorists, who believe that performance is malleable (and hence assign more weight to intentions and effort). We discuss implications of this asymmetry for decision making and for an understanding of the underdog bias. Copyright © 2015 John Wiley & Sons, Ltd.     

Leading North American Programs in Clinical Assessment Research: An Assessment of Productivity and Impact

To identify doctoral programs with strong concentrations in clinical assessment, I measured productivity and impact of faculty at North American institutions with American Psychological Association accredited clinical programs. Publications, citations, and h-indexes derived from 4 top assessment journals were calculated over a 10-year period (1999–2009). I identified a total of 42 leading programs that collectively accounted for more than half of the publications and citations in these journals. I found a moderate relationship between assessment productivity and both US News & World Report program rankings as well as general productivity rankings of clinical programs reported in an earlier study.     

A new rank correlation coefficient with application to the consensus ranking problem

The consensus ranking problem has received much attention in the statistical literature. Given m rankings of n objects the objective is to determine a consensus ranking. The input rankings may contain ties, be incomplete, and may be weighted. Two solution concepts are discussed, the first maximizing the average weighted rank correlation of the solution ranking with the input rankings and the second minimizing the average weighted Kemeny–Snell distance. A new rank correlation coefficient called τ_x is presented which is shown to be the unique rank correlation coefficient which is equivalent to the Kemeny‐Snell distance metric. The new rank correlation coefficient is closely related to Kendall's tau but differs from it in the way ties are handled. It will be demonstrated that Kendall's τ_b is flawed as a measure of agreement between weak orderings and should no longer be used as a rank correlation coefficient. The use of τ_x in the consensus ranking problem provides a more mathematically tractable solution than the Kemeny–Snell distance metric because all the ranking information can be summarized in a single matrix. The methods described in this paper allow analysts to accommodate the fully general consensus ranking problem with weights, ties, and partial inputs. Copyright © 2002 John Wiley & Sons, Ltd.     

An index of specific behaviors in the moral domain

Studies on attribution in the moral domain often involve the use of specific behavior examples. To make valid comparisons across trait dimensions (such as honesty and friendliness), it is important to equate the intensities of the specific behaviors used. Pretesting specific behaviors can be a costly effort, but it is often necessary for research in social psychology. Our study provides a rich source of such pretested behaviors. Positive and negative examples of behaviors in the categories of honesty, loyalty, friendliness, charitableness, and cooperativeness were solicited from participants and then rated on the relevant trait dimension by an independent group. The result is data representing rankings, raw scores, andz-scores in an index of 500 behaviors across 10 trait categories that can be used by researchers to study moral and immoral behaviors. The full index of behaviors is available at www .psychonomic.org/archive/.     

Multidimensional unfolding: Determining the dimensionality of ranked preference data

A model is proposed which treats rankings given by a group of judges as representing regions in an isotonic space of dimensionalityr. Three possible criteria for estimating lower bound dimensionality are discussed: mutual boundary, cardinality, and the occurrence of transposition groups. Problems associated with each criterion are mentioned.Deceased.     

Scholarly productivity,impact, and quality among Academic Psychologists at Group of Eight Universities

Scholarly research among academics in Schools and Departments of Psychology in the Group of Eight Australian universities was assessed for each academic level (Lecturer to Professor) on productivity (number of articles), impact (number of citations to those articles), and quality (journal rankings). Publications were determined over an 8‐year period, 2001–2008. Citations to these works over the same period were also measured. Quality was determined using the Excellence in Research for Australia draft journal rankings. Scholarly productivity was a function of academic level: as academic level increased (Lecturer through Professor) so too did number of publications. Research impact (citations) was also greatest among established Staff (Professors). Research quality was independent of academic level and largely independent of gender. Normative research profiles are provided for each academic level.     

On approximate confidence intervals for measures of concordance

The use ofU-statistics based on rank correlation coefficients in estimating the strength of concordance among a group of rankers is examined for cases where the null hypothesis of random rankings is not tenable. The studentizedU-statistics is asymptotically distribution-free, and the Student-t approximation is used for small and moderate sized samples. An approximate confidence interval is constructed for the strength of concordance. Monte Carlo results indicate that the Student-t approximation can be improved by estimating the degrees of freedom.Research partially supported on ONR Contract N00014-82-K-0207.     

Estimation ofJ scales for unidimensional unfolding

In Coombs' unidimensional unfolding theory each individual ranks all stimuli, often in terms of preference judgments. These ranks, calledI scales, are used to infer the latent continuum called aJ scale which is presumed to have generated theI scale rankings. A major problem concerns the inference of theJ scale given a set ofI scales because anI scale is not unique to any oneJ scale. This paper presents a procedure for estimatingJ scale probabilities given a set ofI scales. Models for three and four or more stimuli are presented. Results of computer tests are reported.     

Saturday Night’s Alright for Fighting: Analysis of Criminal Behavior During the SEC Football Season

Evidence suggests that live sporting events may lead fans to engage in criminal behaviors. The current study examined the relationship between criminal behavior and college football game days. Archival data of criminal offenses were collected from Southeastern Conference member institution police departments to analyze reported criminal offenses during the regular season. Multilevel linear models revealed an increase in criminal behavior. A factorial ANOVA indicated an interaction of team rankings and game outcome was related to specific types of criminal behavior in the host city (F (1, 9142) = 20.07, p < .001). Results provide insight to more effectively allocate resources.     

A multistage ranking model

In this paper, we propose a (n–1)² parameter, multistage ranking model, which represents a generalization of Luce's model. We propose then×n item-rank relative frequency matrix (p-matrix) as a device for summarizing a set of rankings. As an alternative to the traditional maximum likelihood estimation, for the proposed model we suggest a method which estimates the parameters from thep-matrix. An illustrative numerical example is given. The proposed model and its differences from Luce's model are briefly discussed. We also show some specialp-matrix patterns possessed by the Thurstonian models and distance-based models.     

A COMPARISON OF CRITERIA FOR TEST VALIDATION: A META-ANALYTIC INVESTIGATION

Meta-analyses of validity coefficients from tests of clerical abilities for five criteria—supervisor ratings, supervisor rankings, work samples, production quantity, and production quality—were conducted, and the resulting expected true validities were compared. Ratings, rankings, work samples, and production quantity all resulted in high test validities. Validities resulting from ratings and quantity-of-production criteria were highly similar across tests. Validities resulting from rankings and work samples were on the average higher than those from ratings and quantity of production. The fifth criterion, quality of production, had low predictability and did not generalize across situations.     

Do Ranks Suffice? A Comparison of Alternative Weighting Approaches in Value Elicitation

We compared both attribute weights and overall evaluations for students′ preferences among apartments described to them in terms of nine independent attributes. Methods used for eliciting attribute weights were (a) 7-point scales; (b) value hierarchy; (c) swing weights; and two methods using importance rankings only: (d) rank order centroid and (e) rank sum weights. Multiple linear regression was also used to infer attribute weights. Test–retest reliability of overall evaluations was found to be modest. Evaluation models based on all five weight elicitation methods were superior to an equal weights model, with rank order centroid weights modestly superior to other methods.     

Content Analysis of the Journal of Counseling & Development: Volumes 67 to 74

This study examined the content of the Journal of Counseling & Development. Collected data were used to identify various changes and trends over time. A content analysis was conducted for a 9-year period (Volumes 67–74). Articles were cross-classified into 11 categories of topics with 7 subcategories and then grouped by the 3 editorship periods during this time. Topics were then evaluated and ranked for each editorship to identify possible trends. An overall ranking of topics is provided, as well as rankings of contributing authors, institutional affiliations, and states represented by contributing authors. Results are compared with an earlier content analysis (Pelsma & Cesari, 1989) of Volumes 48 to 66.     

Comparability of Self‐Reported Conscientiousness Across 21 Countries

In cross‐national studies, mean levels of self‐reported phenomena are often not congruent with more objective criteria. One prominent explanation for such findings is that people make self‐report judgements in relation to culture‐specific standards (often called the reference group effect), thereby undermining the cross‐cultural comparability of the judgements. We employed a simple method called anchoring vignettes in order to test whether people from 21 different countries have varying standards for Conscientiousness, a Big Five personality trait that has repeatedly shown unexpected nation‐level relationships with external criteria. Participants rated their own Conscientiousness and that of 30 hypothetical persons portrayed in short vignettes. The latter type of ratings was expected to reveal individual differences in standards of Conscientiousness. The vignettes were rated relatively similarly in all countries, suggesting no substantial culture‐related differences in standards for Conscientiousness. Controlling for the small differences in standards did not substantially change the rankings of countries on mean self‐ratings or the predictive validities of these rankings for objective criteria. These findings are not consistent with mean self‐rated Conscientiousness scores being influenced by culture‐specific standards. The technique of anchoring vignettes can be used in various types of studies to assess the potentially confounding effects of reference levels. Copyright © 2011 John Wiley & Sons, Ltd.     

A comparison of men's and women's consciousness-raising groups

This study contrasted the nature and function of consciousness-raising (CR) groups for men and women. Questionnaires were filled out by 61 individuals representing 10 CR groups. There were no significant differences between sexes in terms of rankings for services rendered by the CR groups. This suggests that consciousness-raising groups for men and women may be more alike than different. Combining sexes, it was found that emotional support was rated significantly higher than sociopolitical change and behavior change/advice. CR participants appear to be more concerned with the fulfillment of emotional and social needs in their groups and less concerned with behavioral and political change.