The Use of Multitrait–Multimethod Data for Detecting Nonlinear Relationships: The Case of Psychological Climate and Job Satisfaction

Previous monomethod research has found mixed support for nonlinear effects between certain job characteristics (e.g., job autonomy, job complexity) and outcome variables (e.g., job performance). We hypothesized that these weak nonlinear findings may be due to the prevalence of monomethod research that can lead to a lack of complete measurement and/or introduce common methods variance, either of which may mask the true shape of relationships. Using hierarchical regression analyses and a multitrait–multimethod (MTMM) research design, we found strikingly different results between monomethod and MTMM data when considering the relationships between three psychological climate variables and job satisfaction. While the monomethod results mirrored earlier inconclusive findings, the MTMM data indicated that nonlinear equations explained significantly more of the relationship between all three climate dimensions and job satisfaction. These results suggest that the use of MTMM data allows one to more effectively test for nonlinear effects. Furthermore, these nonlinear results suggest that the format of employee questionnaires should probably change from asking how much an employee has of certain constructs to asking does the employee want more or less of these constructs.     

A Multitrait–Multisource Confirmatory Factor Analytic Approach to the Construct Validity of ADHD and ODD Rating Scales with Malaysian Children

Confirmatory factor analysis (CFA) was used to model a multitrait by multisource matrix to determine the convergent and discriminant validity of measures of attention-deficit hyperactivity disorder (ADHD)-inattention (IN), ADHD-hyperactivity/impulsivity (HI), and oppositional defiant disorder (ODD) in 917 Malaysian elementary school children. The three trait factors were ADHD-IN, ADHD-HI, and ODD. The two source factors were parents and teachers. Similar to earlier studies with Australian and Brazilian children, the parent and teacher measures failed to show convergent and discriminant validity with Malaysian children. The study outlines the implications of such strong source effects in ADHD-IN, ADHD-HI, and ODD measures for the use of such parent and teacher scales to study the symptom dimensions.     

The Elusive General Factor of Personality: The Acquaintance Effect

A general factor (g_p) at the apex of personality has been suggested to account for the correlations between the Big Five. Although the g_p has received ample support from monomethod studies, results from studies incorporating different methods have remained rather ambiguous; some have identified a g_p across different informants, whereas others have not. It was hypothesized that these divergent findings are a result of varying lengths of acquaintance between raters. To this end, the current study presents a multitrait multi‐informant meta‐analysis (total N = 11 941) that found weak support for a g_p as a substantive trait of personality. Evidence for a g_p was susceptible to the length of acquaintance between informants. Although a g_p could be identified for short‐term acquaintances, it remained elusive at long‐term acquaintance. Thus, the g_p in other ratings more likely reflects normative ratings of an average individual rather than ratings of the specific target person. Copyright © 2013 European Association of Personality Psychology     

Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study

When multisource feedback instruments, for example, 360-degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non-standard multilevel structural equation model that incorporates self-ratings (level-2 variables) and others’ ratings from different additional perspectives (level-1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self-ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self-ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade-off effects between the sample sizes of self-ratings and others’ ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ² test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level-specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.     

Qualitatively driven mixed‐methods approaches to counselling and psychotherapy research

Many researchers and clinicians take an ‘either, or’ position regarding factors responsible for change when conducting research. Qualitatively driven mixed methods privilege the qualitative approach and offer the opportunity to generate multi‐dimensional material, permitting a more holistic insight into experiences that can be understood from a combination of epistemological and ontological stances. A qualitatively driven mixed‐methods approach apply a ‘both, and’ position when exploring the elements that produce change or that are under investigation, which can be of particular value to counselling and psychotherapy research. There are various ways of engaging with qualitatively driven mixed methods. Some designs include both qualitative and quantitative components, where the former is the core element. The secondary component may also be qualitative, known as a multimethod design. Yet, other designs mix different qualitative approaches, through the application of different qualitative analyses to the same data (pluralistic qualitative research). This paper discusses the application and value of qualitatively driven mixed methods in counselling and psychotherapy research through the presentation of two research case studies; one which mixes both qualitative and quantitative components to investigate the experiences of pain alleviation following a CBT pain management programme; and one which applies a pluralistic approach to a counselling psychology doctorate exploring the experiences and meanings attached to self‐harming. Through illustration and discussion of the case studies’ mixed‐methods approaches, this paper demonstrates that qualitatively driven mixed methods produce a more enhanced and holistic understanding into phenomena, and therefore a more balanced perspective to counselling and psychotherapy research.     

Measuring Schwartz's values in childhood: Multidimensional scaling across instruments and cultures

The Picture-Based Value Survey for Children (PBVS-C; Döring et al., 2010) assesses children's values through self-report and thereby depicts Schwartz's theory of universal human values at an early age. Despite evidence on the PBVS-C's structural validity, there remains an open question: Does the PBVS-C measure the same values as does Schwartz's Portrait Values Questionnaire (PVQ; Schwartz et al., 2001), an established assessment instrument for adult and adolescent respondents? To answer this question about the PBVS-C's validity, Polish (N = 167) and German (N = 119) children's values were measured with both instruments. The multitrait–multimethod correlation matrix (Campbell & Fiske, 1959) of value items from both instruments was analysed with multidimensional scaling (MDS), as proposed by Borg and Groenen (1997). Our structural expectations for the MDS output about the distinctiveness (1) of Schwartz's higher order value types (i.e., traits) and (2) of the two assessment methods (PBVS-C and PVQ) were confirmed in both samples.     

Bayesian analysis of longitudinal multitrait–multimethod data with ordinal response variables

A new multilevel latent state graded response model for longitudinal multitrait–multimethod (MTMM) measurement designs combining structurally different and interchangeable methods is proposed. The model allows researchers to examine construct validity over time and to study the change and stability of constructs and method effects based on ordinal response variables. We show how Bayesian estimation techniques can address a number of important issues that typically arise in longitudinal multilevel MTMM studies and facilitates the estimation of the model presented. Estimation accuracy and the impact of between‐ and within‐level sample sizes as well as different prior specifications on parameter recovery were investigated in a Monte Carlo simulation study. Findings indicate that the parameters of the model presented can be accurately estimated with Bayesian estimation methods in the case of low convergent validity with as few as 250 clusters and more than two observations within each cluster. The model was applied to well‐being data from a longitudinal MTMM study, assessing the change and stability of life satisfaction and subjective happiness in young adults after high‐school graduation. Guidelines for empirical applications are provided and advantages and limitations of a Bayesian approach to estimating longitudinal multilevel MTMM models are discussed.     

Infant Distress During Immunization: A Multimethod Assessment

The purpose of this study was to comprehensively describe infant procedural distress and pain across assessment modalities, and to compare similarities and differences across measures. A multimethod assessment of distress was conducted to investigate infants (N = 37) undergoing routine immunizations. Measures of infant distress included Parent report, nurse report, infant heart rate, and an observational measure of infant distress. Parents rated their infant's distress and pain significantly higher than did nurses. Observational and physiological ratings of infant distress were found to vary significantly by phase, and there were no correlations between adult ratings of pain and distress and physiological ratings. Findings suggest that infant procedural distress can be assessed in a number of manners. The discordance between these measures emphasizes the need for multimethod assessment of pediatric procedural distress in both research and clinical settings. Given the differences between parent and nurse ratings, clinicians should be aware that different assessment methods might lead to different conclusions about infant procedural distress.     

Multimethod Assessment of Psychopathology Among DSM-IV Subtypes of Children With Attention-Deficit/Hyperactivity Disorder: Self-, Parent, and Teacher Reports

Using data based on self-, parent, and teacher reports, we assessed various aspects of psychopathology in a large sample of control children and those with ADHD. Confirmatory factor analysis was employed to extract response bias from latent constructs of aggression, anxiety, attention problems, depression, conduct disorder, and hyperactivity. These latent constructs were then entered into logistic regression equations to predict membership in control versus ADHD groups, and to discriminate between ADHD subtypes. Results of the regression equations showed that higher levels of attention problems and aggression were the best predictors of membership in the ADHD group relative to controls. Logistic regression also indicated that a higher degree of aggression was the only significant predictor of membership in the ADHD-Combined group compared to the ADHD-Inattentive group. However, when comorbid diagnoses of Oppositional Defiant Disorder and Conduct Disorder were controlled for in the logistic regression, greater hyperactivity rather than aggression was the sole variable with which to distinguish the ADHD-Combined from the ADHD-Inattentive subtype. Results are discussed in the context of the DSM-IV ADHD nosology and the role of instrument and source bias in the diagnosis of ADHD.