Incorporating Mobility in Growth Modeling for Multilevel and Longitudinal Item Response Data

Multilevel data often cannot be represented by the strict form of hierarchy typically assumed in multilevel modeling. A common example is the case in which subjects change their group membership in longitudinal studies (e.g., students transfer schools; employees transition between different departments). In this study, cross-classified and multiple membership models for multilevel and longitudinal item response data (CCMM-MLIRD) are developed to incorporate such mobility, focusing on students' school change in large-scale longitudinal studies. Furthermore, we investigate the effect of incorrectly modeling school membership in the analysis of multilevel and longitudinal item response data. Two types of school mobility are described, and corresponding models are specified. Results of the simulation studies suggested that appropriate modeling of the two types of school mobility using the CCMM-MLIRD yielded good recovery of the parameters and improvement over models that did not incorporate mobility properly. In addition, the consequences of incorrectly modeling the school effects on the variance estimates of the random effects and the standard errors of the fixed effects depended upon mobility patterns and model specifications. Two sets of large-scale longitudinal data are analyzed to illustrate applications of the CCMM-MLIRD for each type of school mobility.     

The impacts of ignoring individual mobility across clusters in estimating a piecewise growth model

A three-level piecewise growth model (3L-PGM) can be used to break up nonlinear growth into multiple components, providing the opportunity to examine potential sources of variation in individual and contextual growth within different segments of the model. The conventional 3L-PGM assumes that the data are strictly hierarchical in nature, where measurement occasions (level 1) are nested within individuals (level 2) who are members of a single cluster (level 3). However, in longitudinal research, it is sometimes difficult for data structures to remain purely clustered during a study, such as when some students change classrooms or schools over time. One resulting data structure in this situation is known as a multiple membership structure, where some lower-level units are members of more than one higher-level unit. The new multiple membership PGM (MM-PGM) extends the 3L-PGM to handle multiple membership data structures frequently found in the social sciences. This study sought to examine the consequences of ignoring individual mobility across clusters when estimating a 3L-PGM in comparison to estimating a MM-PGM. MM-PGM estimates were less biased (especially in the cluster-level coefficient estimates), although we found substantial bias in cluster-level variance components across some conditions for both models.     

INTERVIEWER VALIDITY AND RELIABILITY: AN INDIVIDUAL ANALYSIS APPROACH

Interviewer reliability, validity, and strategy for information integration were studied by analysis of data across interviewers and, also, by within, individual interviewer analysis. Candidates ( N = 412) for selection to a military division of a national defense organization were interviewed by 10 female interviewers and assessed on nine behaviorally anchored dimensions. Candidates ( N = 131) subsequently admitted to officers' training school were evaluated, for the purposes of this study, on 19 dimensions and on an overall evaluation taken at six and twelve week points. Results of analyses of data across interviewers indicated that interviewers functioned in a similar fashion, using few of the dimensions in their decisions whereas analyses of individual interviewers indicated higher reliability and individual differences among interviewers' strategy formation. Analysis across interviewers of the relationship of the interview decision to six and twelve week training performance evaluations indicated no validity for the interview decision. Analysis of individual interview strategies revealed differences among the interviewers only at the six week point. Results are discussed with regard to methodological problems, interview strategy differences, criteria dimensionality, fruitfulness of individual, within interviewer analyses, and purpose of the interview.     

Incorporating Student Mobility in Achievement Growth Modeling: A Cross-Classified Multiple Membership Growth Curve Model

Multiple membership random effects models (MMREMs) have been developed for use in situations where individuals are members of multiple higher level organizational units. Despite their availability and the frequency with which multiple membership structures are encountered, no studies have extended the MMREM approach to hierarchical growth curve modeling (GCM). This study introduces a cross-classified multiple membership growth curve model (CCMM-GCM) for modeling, for example, academic achievement trajectories in the presence of student mobility. Real data are used to demonstrate and compare growth curve model estimates using the CCMM-GCM and a conventional GCM that ignores student mobility. Results indicate that the CCMM-GCM represents a promising option for modeling growth for multiple membership data structures.     

From a single-level analysis to a multilevel analysis of single-case experimental designs

Multilevel modeling provides one approach to synthesizing single-case experimental design data. In this study, we present the multilevel model (the two-level and the three-level models) for summarizing single-case results over cases, over studies, or both. In addition to the basic multilevel models, we elaborate on several plausible alternative models. We apply the proposed models to real datasets and investigate to what extent the estimated treatment effect is dependent on the modeling specifications and the underlying assumptions. By considering a range of plausible models and assumptions, researchers can determine the degree to which the effect estimates and conclusions are sensitive to the specific assumptions made. If the same conclusions are reached across a range of plausible assumptions, confidence in the conclusions can be enhanced. We advise researchers not to focus on one model but conduct multiple plausible multilevel analyses and investigate whether the results depend on the modeling options.     

Lower level mediation in multilevel models

Multilevel models are increasingly used to estimate models for hierarchical and repeated measures data. The authors discuss a model in which there is mediation at the lower level and the mediational links vary randomly across upper level units. One repeated measures example is a case in which a person's daily stressors affect his or her coping efforts, which affect his or her mood, and both links vary randomly across persons. Where there is mediation at the lower level and the mediational links vary randomly across upper level units, the formulas for the indirect effect and its standard error must be modified to include the covariance between the random effects. Because no standard method can estimate such a model, the authors developed an ad hoc method that is illustrated with real and simulated data. Limitations of this method and characteristics of an ideal method are discussed.     

INDIVIDUAL DIFFERENCES IN INTERVIEWER RATINGS: THE IMPACT OF STANDARDIZATION, CONSENSUS DISCUSSION, AND SAMPLING ERROR ON THE VALIDITY OF A STRUCTURED INTERVIEW

This research investigated empirical issues regarding the validity of individual interviewers'( N = 62) ratings collected after a structured interview. Each interviewer rated an average of 25 interviewees. One hypothesis examined but not supported was that systematic interviewer errors will attenuate interview validity when data are aggregated across interviewers. Also investigated was the validity of ratings averaged across interviewers compared to consensus ratings; consensus ratings were shown to have significantly but probably not practically higher validities. Third, a meta-analysis of individual interviewer validities revealed that all of the variance in validities could be attributed to sampling error. Results and implications are discussed.     

CORRELATES OF COUNSELING EFFECTIVENESS IN EMPLOYMENT INTERVIEWERS

The purpose of this study was to identify correlates of effective counseling behavior in employment interviewers as measured by the Kelz Counselor Performance Rating Scale. Four demographic variables were studied: age, length of employment as an employment interviewer for a state employment agency, sex, and level of education. Seventeen employment interviewers employed by the Pennsylvania Bureau of Employment Security served as subjects. Significant negative correlations were found between effective counseling behavior as measured by the Kelz Scale and both age and length of employment as an interviewer for a state agency. No relationship was found between the criterion variable and the sex or level of education of the interviewers. Some possible implications of these findings are discussed and tentative hypotheses presented.     

A Comparison of Population-Averaged and Cluster-Specific Approaches in the Context of Unequal Probabilities of Selection

Sampling designs of large-scale survey studies are typically complex, involving multiple design features such as clustering and unequal probabilities of selection. Single-level (i.e., population-averaged) methods that use adjusted variance estimators and multilevel (i.e., cluster-specific) methods provide two alternatives for modeling clustered data. Although the literature comparing these methods is vast, comparisons have been limited to the context in which all sampling units are selected with equal probabilities (thus circumventing the need for sampling weights). The goal of this study was to determine under what conditions single-level and multilevel estimators outperform one another in the context of a two-stage sampling design with unequal probabilities of selection. Monte Carlo simulation methods were used to evaluate the impact of several factors, including population model, informativeness of the design, distribution of the outcome variable, intraclass correlation coefficient, cluster size, and estimation method. Results indicated that the unweighted estimators performed similarly across conditions, whereas the weighted single-level estimators tended to outperform the weighted multilevel estimators, particularly under nonideal sample conditions. Multilevel weight approximation methods did not perform well when the design was informative. An empirical example is provided to demonstrate how researchers might investigate the implications of the simulation results in practice.     

Selection of gender‐incongruent applicants: No gender bias with structured interviews

Research suggests that the use of structured interviews can reduce gender bias in hiring. However, studies have been limited to gender‐neutral professions or laboratory simulations. The current study evaluated two components of the structured interview in a field sample of 691 applicants interviewing for a male‐dominated position of public transit operator. Multilevel modeling results show no significant differences in ratings across applicant gender with this highly structured interview. This relation was found in individual interviewer ratings and consensus panel ratings, as well as irrespective of interviewer gender and interviewer participation in comprehensive versus minimal training in structured interviewing. This study provided a conservative test in a male‐dominated profession to further validate the value of the structured interview for promoting equal hiring practices.     

Estimation of a Latent Variable Regression Growth Curve Model for Individuals Cross-Classified by Clusters

The cross-classified multiple membership latent variable regression (CCMM-LVR) model is a recent extension to the three-level latent variable regression (HM3-LVR) model which can be utilized for longitudinal data that contains individuals who changed clusters over time (for instance, student mobility across schools). The HM3-LVR model can include the initial status on growth effect as varying across those clusters and allows testing of more flexible hypotheses about the influence of initial status on growth and of factors that might impact that relationship, but only in the presence of pure clustering of participants within higher-level units. This Monte Carlo study was conducted to evaluate model estimation under a variety of conditions and to measure the impact of ignoring cross-classified data when estimating the incorrectly specified HM3-LVR model in a scenario in which true values for parameters are known. Furthermore, results from a real-data analysis were used to inform the design of the simulation. Overall, it would be recommended for researchers to utilize the CCMM-LVR model over the HM3-LVR model when individuals are cross-classified, and to use a bare minimum of more than 100 clustering units in order to avoid overestimation of the level-3 variance component estimates.     

The developmental ecology of urban males' youth violence

Data from a longitudinal study of 294 African American and Latino adolescent boys and their caregivers living in poor urban communities were used to test a developmental-ecological model of violence. Six annual waves of data were applied to evaluate the relations between microsystem influences of parenting and peer deviance (peer violence and gang membership), macrosystem influences of community structural characteristics and neighborhood social organization, and individual involvement in violence (level and growth). Structural equation modeling analyses showed that community structural characteristics significantly predicted neighborhood social processes. Parenting practices partially mediated the relation between neighborhood social processes and gang membership. Parenting practices was fully mediated in its relation to peer violence by gang membership. Gang membership was partially mediated by peer violence level in its relation to individual violence level. Although the overall set of relations does not satisfy mediation requirements fully in all instances, the model was validated for the most part, supporting a focus on a multilevel ecological model of influences on risk development.     

The impact of ignoring multiple membership data structures in multilevel models

This study compared the use of the conventional multilevel model (MM) with that of the multiple membership multilevel model (MMMM) for handling multiple membership data structures. Multiple membership data structures are commonly encountered in longitudinal educational data sets in which, for example, mobile students are members of more than one higher-level unit (e.g., school). While the conventional MM requires the user either to delete mobile students' data or to ignore prior schools attended, MMMM permits inclusion of mobile students' data and models the effect of all schools attended on student outcomes. The simulation study identified underestimation of the school-level predictor coefficient, as well as underestimation of the level-two variance component with corresponding overestimation of the level-one variance when multiple membership data structures were ignored. Results are discussed along with limitations and ideas for future MMMM methodological research as well as implications for applied researchers.     

Applicant impression management in job interview: The moderating role of interviewer affectivity

In light of the organizational need to obtain talented personnel, an appropriate evaluation of applicant behaviour in the selection interview is crucial. Extending past research on applicant use of impression management (IM) tactics, this study examines the effects of a broad set of IM tactics in a field setting, and also investigates the moderating roles of two rarely tested interviewer characteristics: interviewer positive affectivity (PA) and negative affectivity (NA). Due to the nested nature of the data, consisting of 142 job applicants and 33 interviewers, we adopt hierarchical linear modelling (HLM) to examine the proposed hypotheses. The results indicate that three IM tactics (self‐focused IM, SFIM; other‐focused IM, OFIM; and non‐verbal IM, NVIM) are significantly positively related to interviewer evaluations. Furthermore, interviewer PA appears to strengthen the positive effects of SFIM tactics on interviewer evaluations, while the effects of NVIM tactics may be weaker when interviewers are high in NA. In addition, these findings suggest the importance of interviewer trait affectivity in explaining interviewer's decision‐making variability, which may lead to low inter‐rater reliability and in turn restrict the level of achievable validity.     

Evaluating the Evaluators: Perceptions of Interviewers by Rejected Job Applicants as a Function of Interviewer and Applicant Sex1

Students who had recently had an unsuccessful job interview rated the competence of their interviewer and completed the Women As Managers Scale (WAMS; Terborg, Peters, Ilgen, & Smith, 1977 ). The results showed an impact of the sex of interviewer on judgments of interviewer competence and WAMS scores, but only for male participants. Male participants gave lower ratings of interviewer competence to female than to male interviewers, and ratings for the female interviewers were also lower than those given by female participants. Scores on the WAMS were lower for male participants who were interviewed by a female interviewer than those interviewed by a male interviewer, and were lower for male than for female participants with a female interviewer.     

Predictors of change in postmerger identification during a merger process: a longitudinal study

Adopting an intergroup perspective, the authors examined predictors of change in postmerger identification throughout a merger. Data were collected over 3 points of measurement from 157 students of a newly merged university. The 1st questionnaire was distributed 4 months after the implementation of the merger; the following 2 were distributed 6 months and 1 year thereafter. With its longitudinal design, this study replicates and extends past results by revealing predictors of change in organizational identification for members of the dominant and subordinate organizations throughout a merger process. As predicted, postmerger identification increased only slowly for members of both the dominant and the subordinate organizations. Multilevel models for change confirmed that the predictive effect of premerger identification on postmerger identification for members of the dominant organization dissipates over time. The effect of in-group typicality unexpectedly varied as a function of organizational membership and was stable over time. Perceived fairness in the merger process positively influenced postmerger identification across members of both organizations; over time the effect of fairness amplified.     

面试考官的权重策略对评分和录用的影响

从个体水平研究面试考官的有效性是面试研究的一个热点问题．本研究对面试考官权重策略的类型及类型的比较进行了实证研究。研究运用线性规划策略捕捉方法,分析了公务员考试中28名考官在结构化面试和无领导小组讨论中的权重策略,研究了权重策略对评分和录用的影响,结果发现：（1）在结构化面试中存在三种策略,在无领导小组讨论中存在两种策略,这些策略大部分是规范的（符合评分规则的要求）,它们会随面试情景发生变化;（2）权重策略影响了考官的评分区分度和最终考生的录用结果,规范性策略具有较高的区分度和录用预测效度。研究最后对结果的理论和实践意义进行了讨论。     

The Variability of Criterion‐Related Validity Estimates Among Interviewers and Interview Panels

The authors examined differences in criterion‐related validity estimates among ratings from individual interviewers and interview panels within a structured interview. Senior non‐commissioned officers (NCOs) in the U.S. Army (N=64) conducted panel interviews with 944 junior NCOs during a concurrent validation project. Analysis of the data revealed considerable variation in interviewer validity coefficients in relation to multiple performance criteria. Results also indicated the importance of adopting a multivariate perspective when evaluating interviewer validity differences in that the amount of variation in validity coefficients differed both by interview dimension and criterion. A similar pattern of findings emerged when analyses were performed on ratings averaged within interview panel. Nonetheless, when meta‐analysis was used to estimate the amount of true variance in interviewer‐ and panel‐level validity coefficients, most or all of the variance for some interview‐criterion combinations appeared to be due to statistical artifacts.     

Often biased but rarely in doubt: How initial reactions to stigmatized applicants affect interviewer confidence

Building on a metacognitive framework of heuristic judgments, we investigate the effect of applicant stigma on interviewers’ overconfidence in their (biased) judgments. There were 193 experienced interviewers conducting a face-to-face interview with an applicant who was facially stigmatized or not, and who was visible (traditional interview) or not (partially blind interview), to the interviewer during the rapport-building stage. In traditional interviews, interview judgments of stigmatized applicants were negatively biased, and interviewers reported overconfidence in these judgments. This effect was partially mediated by the interviewer’s professional performance during rapport building. Interview procedure moderated both the direct and indirect effect (through professional performance) of applicant stigma on interviewer confidence. Results show that interviewer (over)confidence in biased judgments is driven by the initial effects of, and reactions to, the stigmatized applicant.