Assessing reading comprehension with narrative and expository texts: Dimensionality and relationship with fluency,vocabulary and memory

Reading comprehension assessment should rely on valid instruments that enable adequate conclusions to be taken regarding students' reading comprehension performance. In this article, two studies were conducted to collect validity evidence for the vertically scaled forms of two Tests of Reading Comprehension for Portuguese elementary school students in the second to fourth grades, one with narrative texts (TRC‐n) and another with expository ones (TRC‐e). Two samples of 950 and 990 students participated in Study 1, the study of the dimensionality of the TRC‐n and TRC‐e forms, respectively. Confirmatory factor analyses provided evidence of an acceptable fit for the one‐factor solution for all test forms. Study 2 included 218 students to collect criterion‐related validity. The scores obtained in each of the test forms were significantly correlated with the ones obtained in other reading comprehension measures and with the results obtained in oral reading fluency, vocabulary and working memory tests. Evidence suggests that the test forms are valid measures of reading comprehension.     

Do Assessors Have Too Much on their Plates? The Effects of Simultaneously Rating Multiple Assessment Center Candidates on Rating Quality

It has been suggested that the large cognitive demands during the observation of assessment center (AC) participants can impair the quality of the assessors' ratings. An aspect that is especially relevant in this regard is the number of candidates that assessors have to observe simultaneously during group discussions, which are one of the most commonly used AC exercises. The present research evaluated potential impairments of the quality of the assessors' ratings (construct‐ and criterion‐related validity and rating accuracy) related to the number of to‐be‐observed candidates. Study 1 (N=1046) was a quasi‐experimental field study and Study 2 (N=71) was an experimental laboratory study. Both studies found significant impairments of assessors' rating quality when a larger in comparison to a lower number of candidates had to be observed simultaneously. These results suggest that assessors should not have to observe too many candidates at the same time during AC group discussions.     

INTRAORGANIZATIONAL EMPLOYEE NAVIGATION AND SOCIALLY DERIVED OUTCOMES: CONCEPTUALIZATION,VALIDATION, AND EFFECTS ON OVERALL PERFORMANCE

Intraorganizational employee navigation (IEN) is conceptualized as a means of better understanding how the organizational actor proactively works across their firm's internal environment in the execution of their jobs. Navigation is argued to be a precursor to the employee's overall performance through a class of mediating variables labeled “socially derived outcomes,” which are variables inside the organization that are bestowed upon the employee as a result of them first engaging in proactive behavior (e.g., IEN). Two studies are reported. Study I sees IEN psychometrically validated versus a range of existing proactive behaviors and individual traits (discriminant, nomological, and criterion‐related validity) with a heterogeneous sample of 704 employees. Study II then tests a model relating IEN to performance through six mediating “socially derived outcomes” by leveraging data from 2 Fortune 500 firms. The results of Study II show that IEN significantly impacts multiple measures of the employee's overall performance through mediating effects brought about by key socially derived outcomes, such as the employee's “manager alignment.” The contributions, broader implications, and limitations of the research are then put into context.     

A video‐based procedure for the assessment of prospective memory

Prospective memory, or the everyday activity of remembering to remember, is frequently deficient in persons with brain injuries. However, few clinically useful measures of this ability are available. We report the results of three studies using student participants, conducted to examine the psychometric credentials of a video‐based method of assessing prospective remembering. This procedure requires participants to watch a video segment recorded moving through a shopping precinct and to recall preassigned instructions (e.g. ‘Buy a Big Mac at McDonald's’) in response to cues appearing on the videotape. Study 1 showed that the video procedure is reliable and easier to complete when the video footage is set in a familiar location. Study 2 confirmed that familiarity enhances recall and demonstrated that pre‐exposure to the unfamiliar video stimuli could partially attenuate this effect. In Study 3 it was found that performance on the video‐based task was correlated with performance on an equivalent real life memory task providing evidence for criterion validity. The findings from these studies demonstrate that clinical and experimental measures of prospective memory with high ecological validity can be constructed based on video recordings of everyday activities. Copyright © 2001 John Wiley & Sons, Ltd.     

Validation of a Chinese achievement goal orientation questionnaire

Background . The mixed findings of previous studies on the nature and effects of performance goals have led to a call for re‐examination of the dichotomous framework of achievement goal orientation theory. While the call for a revised achievement goal orientation theory has received considerable discussion in Western studies, it is not clear whether the revised theory can also be applied in other ethnic and cultural contexts. Aims . Our aim was to validate the Chinese version of Elliot and Church's (1007) Goal Orientation Questionnaire and to initially test the revised goal orientation theory in the context of Chinese students in Hong Kong. Sample . A total of 270 Chinese students (137 boys and 133 girls) from a primary and secondary school in Hong Kong participated in Study 1. Study 2 involved a total of 9,440 students (5,420 boys and 4,020 girls) from 10 primary and 18 secondary schools in Hong Kong. Method . Participants in Study 1 completed a Chinese Goal Orientation Questionnaire (CGOQ) measuring three types of goal orientation. Exploratory factor analysis, item–total correlation, and reliability analyses were undertaken to assess the psychometric quality of the CGOQ. In Study 2, confirmatory factor analysis was used to provide further validation for the revised questionnaire. The relationships among different goal orientations and that with students' perceived classroom environment and self‐efficacy were also explored using structural equation modelling. Results . Findings of exploratory factor analysis in Study 1 and confirmatory factor analysis in Study 2 supported the proposed factor structure of the CGOQ. All the subscales in the questionnaire also showed good internal consistency reliabilities. The construct validity of the CGOQ was supported by its significant relationship with criterion measures. While most of the relationships between the three types of goal orientation and the learning‐related variables measured in our study were consistent with the revised goal orientation theory, the positive relationships between performance‐avoidance goals, mastery goals, and perceived classroom environment were different from previous studies. Conclusion . Our findings generally supported the trichotomous framework of the revised goal orientation theory with Chinese students in Hong Kong. Consistent with the revised goal orientation theory, our findings indicated that both mastery and performance‐approach goals had positive impacts on students' learning. However, the positive relationships between performance‐avoidance goals, mastery goals, and perceived classroom environment were contradictory to the conceptualization of performance‐avoidance goals in the revised theory. Cultural and social factors affecting Hong Kong students' goal orientations are discussed.     

VALIDITY GENERALIZATION RESULTS FOR LAW ENFORCEMENT OCCUPATIONS

The Schmidt-Hunter interactive validity generalization procedure was applied to validity data for cognitive abilities tests for law enforcement occupations. Both assumed artifact distributions, and distributions of artifacts constructed from information contained in the current sample of studies were used to test the hypothesis of situational specificity and to estimate validity generalizability. Results for studies using a criterion of performance in training programs showed that validities ranged from .41 to .71, and for four test types the hypothesis of situational specificity could be rejected using the 75% decision rule. For the remaining test types, validity was generalizable, based on 90% credibility values ranging from .37 to .71. Results for studies using a criterion of performance on the job indicated that the hypothesis of situational specificity was not tenable for three test types, which had validities between .17 and .31. For the remaining test types, estimated mean true validities ranged from .10 to .26 and were generalizable to a majority of situations. Results for both groups of studies were essentially identical for the two types of artifact distribution. Possible reasons for the apparently lower validities and lesser generalizability for job performance criteria are discussed, including possible low validity of the criterion (due to lack of opportunity by supervisors to observe behavior) and the potential role of noncognitive factors in the determination of law enforcement job success. Suggestions for specifically targeted additional research are made.     

Measuring Relationships between Personality,Knowledge, and Performance using Single‐response Situational Judgment Tests

We report two studies that investigate single‐response situational judgment tests (SJTs) as measures of job knowledge. Study 1 examines relationships between job knowledge measured by a single‐response SJT, personality, and performance for museum tour guides. Study 2 extends Study 1's findings with a sample of volunteers using a single‐response SJT about volunteerism. In both studies, personality was related to knowledge, and knowledge predicted performance. In Study 2, knowledge accounted for incremental variance in performance beyond personality, but personality added no incremental variance beyond knowledge. Results suggest that knowledge of effective behavior and knowledge of ineffective behavior are separate constructs. These studies demonstrate the validity of single‐response SJTs and provide evidence that knowing what to do and what not to do are separate domains.     

Development and initial validation of an indirect measure of transformational leadership integrity

An indirect measure of transformational leadership integrity was developed across three studies. In Study 1, the transformational leadership integrity implicit association test (TLI-IAT) was developed and tested with 65 leaders across heterogeneous organizational contexts. Study 2 involved 51 coaches from 18 sports. Results from Studies 1 and 2 supported the construct validity of the instrument, providing evidence of the instrument's convergent and discriminant validity. Study 3 involved 32 coaches and 133 players from six sports. Findings supported the criterion validity of the measure, providing evidence for the instrument's predictive validity. In sum, evidence is presented that supports the TLI-IATs construct and criterion validity. As such, the present research has made significant advancements to the transformational leadership integrity literature and provides researchers with an indirect measure of automatic transformational leadership integrity self-attitudes.     

Development and validation of a work‐specific measure of cognitive failure: Implications for occupational safety

Two studies were conducted to develop and validate a work‐specific measure of cognitive failure, the Workplace cognitive failure scale (WCFS). In initial item development, content validity was gained via sorting. In Study 1, data were obtained from an employee sample that supported the factorial, construct, and criterion‐related validity of the WCFS. In particular, results supported the expected relationships of workplace cognitive failure with facets of personality, role overload, components of self‐regulation, and self‐reported measures of safety outcomes. Study 2 examined the WCFS in two additional employee samples, and provided further criterion‐related validity using objective measures of injury and supervisor ratings of safety behaviour. Overall, results supported the validity and utility of the new measure in assessing organizational safety behaviour and outcomes, more so than the general cognitive failure scale.     

Transparency of Assessment Centers: Lower Criterion‐related Validity but Greater Opportunity to Perform?

Assessment centers (ACs) are popular selection devices in which assessees are assessed on several dimensions during different exercises. Surveys indicate that ACs vary with regard to the transparency of the targeted dimensions and that the number of transparent ACs has increased during recent years. Furthermore, research on this design feature has put conceptual arguments forward regarding the effects of transparency on criterion‐related validity, impression management, and fairness perceptions. This study is the first to examine these effects using supervisor‐rated job performance data as the criterion. We conducted simulated ACs with transparency as a between‐subjects factor. The sample consisted of part‐time employed participants who would soon be applying for a new job. In line with our hypothesis, results showed that ratings from an AC with nontransparent dimensions were more criterion valid than ratings from an AC with transparent dimensions. Concerning impression management, our results supported the hypothesis that transparency moderates the relationship between self‐promotion and job performance, such that self‐promotion in the nontransparent AC was more positively related to job performance than self‐promotion in the transparent AC. The data lent no support for the hypothesis that participants’ perceptions of their opportunity to perform are higher in the transparent AC.     

The positive and negative semantic dimensions of relationship satisfaction

Semantic differential items were reconfigured to assess relationship satisfaction across separate positive and negative attitude dimensions. Study 1 (N = 1,656) supported a 2‐factor model for the Positive and Negative Semantic Differential (PN–SMD), as well as its convergent, criterion‐related, and incremental validity over the 16‐item Couples Satisfaction Index (CSI; J. L. Funk & R. D. Rogge, 2007) using known correlates of relationship satisfaction as criteria. Study 2 (N = 89) replicated the convergent, criterion‐related, and incremental validity findings of Study 1 using different criterion measures, the CSI, a bipolar semantic differential measure designed for assessing relationship satisfaction, and an existing 2‐dimensional measure of relationship satisfaction. The authors demonstrated across studies that the PN–SMD captures criterion‐relevant information about ambivalence versus indifference toward the relationship—associations that are only detectable when using a 2‐dimensional satisfaction measure.     

The Three‐option Format for Knowledge and Ability Multiple‐choice Tests: A case for why it should be more commonly used in personnel testing

Multiple‐choice (MC) tests are arguably the most widely used testing format in applied settings. In the psychometric and education literatures, research on the optimal number of options for knowledge and ability MC tests has revealed that three‐option tests are psychometrically equivalent and, in some cases, superior to five‐option tests. In addition, there are a number of practical, economic, and administrative advantages associated with the use of three‐option MC tests. Yet, despite its advantages, the three‐option format is underutilized in personnel selection. Across two studies, we compared test‐taker perceptions, criterion‐related validity, and sex‐based subgroup differences, and in Study 1, we compared race‐based subgroup differences on three‐ and five‐option tests. Participants in the two studies completed a three‐ or five‐option version of ACT. Test perceptions, criterion‐related validity, and race‐ and sex‐based subgroup differences were similar across test formats. The implications for the expanded use of three‐option tests in applied settings and future directions for research are discussed.     

Development and Validation of the Performance Management Behavior Questionnaire

This study integrates literature on performance management and organizational behavior with input from managers to develop a 27‐item measure of performance management behavior. We used 8 samples, including multisource data, to develop a reliable multidimensional scale that replicated across samples. The scale demonstrated content validity, both convergent and discriminant validity, and incremental validity, and criterion‐related validity was established through the scale's relation with measures of management effectiveness and job attitudes. Overall, this study suggests that the Performance Management Behavior Questionnaire (PMBQ) is a reliable and valid measure and that performance management behaviors have important implications for both employees and organizations.     

The Knowledge,Skill, and Ability Requirements for Teamwork: Revisiting the Teamwork‐KSA Test's validity

The Teamwork – Knowledge, Skills, and Ability (KSA) Test was developed by Stevens and Campion to operationalize their comprehensive taxonomy of teamwork competencies. The test is generally considered ‘valid’ and has been used frequently in organizations. Our review of the literature found an average criterion validity of.20 for the Teamwork‐KSA Test, although there was considerable variability across studies. We could find no research on the item properties, factor structure, or subscale reliabilities, and no extensive investigations of the nomological net of this test. In our field sample, we found subscale reliabilities to be generally inadequate, no meaningful factor structure, and low predictivenes of employees' performance on team‐related dimensions. Although the taxonomy it purports to measure is preeminent, the Teamwork‐KSA Test itself may have serious limitations.     

Measurement and Application of Egoistic and Moralistic Self‐enhancement

Five studies were conducted to develop and validate the Egoistic and Moralistic Self‐enhancement (EMS) scale, a new self‐report measure of egoistic and moralistic self‐enhancement. Exploratory and confirmatory factor analysis provided evidence for a two‐factor structure comprising the tendencies to promote agentic (egoistic) and communal (moralistic) qualities (Study 1). Construct validity was supported by significant correlations with established measures of socially desirable responding, such as the Balanced Inventory of Desirable Responding (Study 2) and the Impression Management scale of the 16 Personality Factors (Study 3). When the EMS scale has been applied in organizational settings, we found that job applicants were more inclined than nonapplicants to promote their own agentic and communal qualities, although differences were more pronounced for egoistic self‐enhancement (Study 4). Findings also suggested that exaggeration of agentic qualities decrease the criterion‐related validity of conscientiousness in predicting job performance (Study 5). Taken together, findings suggest that the EMS scale provides a valid and reliable measure that can be fruitfully used for both theoretical and applied research.     

Exploring the Capacity of NEO PI‐R Facets to Predict Job Performance in Two French‐Canadian Samples

The capacity of personality facets to predict contextual performance (i.e., interpersonal facilitation and job dedication), task performance, and global performance was explored by means of two predictive validity field studies. In a preliminary way, Study 1 demonstrated that four personality facets – Self‐Consciousness, Excitement Seeking, Trust, and Competence – were related to at least one job performance dimension in a well‐structured occupation sample. Study 2 revealed that eight personality facets were linked to specific job performance dimensions in a professional occupation sample. Results indicate that a combination of facets is useful for predicting job performance.     

Video‐based Testing: A high‐fidelity job simulation that demonstrates reliability,validity, and utility

A conceptual rationale is presented for the use of a high‐fidelity simulation: a video‐based test (VBT), in which applicants view job‐related scenarios, respond orally, and are later scored on five dimensions by trained raters. Using applicant data, we empirically investigate the criterion‐related validity, reliability, and utility of the VBT. Overall, we found that the VBT predicted performance in training and on the job. In addition, different raters strongly agreed with one another on each applicant's scores. Despite somewhat higher development costs, the VBT demonstrated high utility. We suggest that organizations consider the use of VBTs as a selection instrument for jobs that require the use of applied social and interpersonal skills.     

WORK ENGAGEMENT: A QUANTITATIVE REVIEW AND TEST OF ITS RELATIONS WITH TASK AND CONTEXTUAL PERFORMANCE

Many researchers have concerns about work engagement's distinction from other constructs and its theoretical merit. The goals of this study were to identify an agreed‐upon definition of engagement, to investigate its uniqueness, and to clarify its nomological network of constructs. Using a conceptual framework based on Macey and Schneider (2008; Industrial and Organizational Psychology, 1, 3–30), we found that engagement exhibits discriminant validity from, and criterion related validity over, job attitudes. We also found that engagement is related to several key antecedents and consequences. Finally, we used meta‐analytic path modeling to test the role of engagement as a mediator of the relation between distal antecedents and job performance, finding support for our conceptual framework. In sum, our results suggest that work engagement is a useful construct that deserves further attention.     

Testing primary‐school children's understanding of the nature of science

Understanding the nature of science (NOS) is a critical aspect of scientific reasoning, yet few studies have investigated its developmental beginnings and initial structure. One contributing reason is the lack of an adequate instrument. Two studies assessed NOS understanding among third graders using a multiple‐select (MS) paper‐and‐pencil test. Study 1 investigated the validity of the MS test by presenting the items to 68 third graders (9‐year‐olds) and subsequently interviewing them on their underlying NOS conception of the items. All items were significantly related between formats, indicating that the test was valid. Study 2 applied the same instrument to a larger sample of 243 third graders, and their performance was compared to a multiple‐choice (MC) version of the test. Although the MC format inflated the guessing probability, there was a significant relation between the two formats. In summary, the MS format was a valid method revealing third graders' NOS understanding, thereby representing an economical test instrument. A latent class analysis identified three groups of children with expertise in qualitatively different aspects of NOS, suggesting that there is not a single common starting point for the development of NOS understanding; instead, multiple developmental pathways may exist.     

In‐basket Validity: A systematic review

In‐baskets are high‐fidelity simulations often used to predict performance in a variety of jobs including law enforcement, clerical, and managerial occupations. They measure constructs not typically assessed by other simulations (e.g., administrative and managerial skills, and procedural and declarative job knowledge). We compiled the largest known database (k = 31; N = 3,958) to address the criterion‐related validity of in‐baskets and possible moderators. Moderators included features of the in‐basket: content (generic vs. job specific) and scoring approach (objective vs. subjective) and features of the validity studies: design (concurrent vs. predictive) and source (published vs. unpublished). Sensitivity analyses assessed how robust the results were to the influence of various biases. Results showed that the operational criterion‐related validity of in‐baskets was sufficiently high to justify their use in high‐stakes settings. Moderator analyses provided useful guidance for developers and users regarding content and scoring.