A nonparametric item analysis of a selected item subset of the Learning Process Questionnaire

Background: Although there have been numerous studies conducted on the psychometric properties of Biggs' Learning Process Questionnaire (LPQ), these have involved the use of traditional omnibus measures of scale quality such as corrected item total correlations, internal consistency estimates of reliability, and factor analysis. However, these omnibus measures of scale quality are sample dependent and fail to model item responses as a function of trait level. And since the item trait relationship is typically nonlinear, traditional factor analytic methods are inappropriate. Aims: The purpose of this study was to identify a unidimensional subset of LPQ items and examine the effectiveness of these items and their options in discriminating between changes in the underlying trait level. In addition to assessing item quality, we were interested in assessing overall scale quality with non‐sample dependent measures. Method: The sample was split into two nearly equal halves, and a undimensional subset of items was identified in one of these samples and cross‐validated in the other. The nonlinear relationship between the probability of endorsing an item option and the underlying trait level was modelled using a nonparametric latent trait technique known as kernel smoothing and implemented with the program TestGraf. After item and scale quality were established, maximum likelihood estimates of participants' trait level were obtained and used to examine grade and gender differences. Results: A undimensional subset of 16 deep and achieving items was identified. Slightly more than half of these items needed some of their options combined so that the probability of endorsing an item option as a function of increasing trait level corresponded to the ideal rank ordering of the item options. With this adjustment, scale quality as measured by the information function and standard error function was found to be good. However, no statistically significant gender differences were observed and, although statistically significant grade differences were observed, they were not substantively meaningful. Conclusions: The use of nonparametric kernel‐smoothing techniques is advocated over parametric latent trait methods for the analysis of attitudinal and psychological measures involving polychotomous ordered‐response categories. It is also suggested that latent trait methods are more appropriate than traditional test‐based measures for studying differential item functioning both within and between cultures. Nonparametric kernel‐smoothing techniques hold particular promise in identifying and understanding cross‐cultural differences in student approaches to learning at both the item and scale level.     

The construct validity of the stanford-binet 5 measures of working memory

This study examines the validity of the measures of verbal and nonverbal working memory on the Stanford-Binet Fifth Edition (SB5). The validity evidence included Rasch-based, criterion-referenced item mapping, correlations with other clinical measures of memory, and prediction of reading and mathematics scores. The item mapping clearly demonstrates a parallel between increasing item difficulty and a progression of item characteristics that placed increasing demands on verbal and nonverbal working memory. The higher correlations of the SB5 verbal and nonverbal working memory subtests with other measures of verbal and nonverbal memory, respectively, and lower correlations with nonverbal and verbal memory measures, respectively, clearly show convergent and divergent validity. The higher correlations between SB5 verbal working memory and reading skills and between SB5 nonverbal memory and mathematics skills are consistent with past research.     

Reliability measures in item response theory: Manifest versus latent correlation functions

For item response theory (IRT) models, which belong to the class of generalized linear or non‐linear mixed models, reliability at the scale of observed scores (i.e., manifest correlation) is more difficult to calculate than latent correlation based reliability, but usually of greater scientific interest. This is not least because it cannot be calculated explicitly when the logit link is used in conjunction with normal random effects. As such, approximations such as Fisher's information coefficient, Cronbach's α, or the latent correlation are calculated, allegedly because it is easy to do so. Cronbach's α has well‐known and serious drawbacks, Fisher's information is not meaningful under certain circumstances, and there is an important but often overlooked difference between latent and manifest correlations. Here, manifest correlation refers to correlation between observed scores, while latent correlation refers to correlation between scores at the latent (e.g., logit or probit) scale. Thus, using one in place of the other can lead to erroneous conclusions. Taylor series based reliability measures, which are based on manifest correlation functions, are derived and a careful comparison of reliability measures based on latent correlations, Fisher's information, and exact reliability is carried out. The latent correlations are virtually always considerably higher than their manifest counterparts, Fisher's information measure shows no coherent behaviour (it is even negative in some cases), while the newly introduced Taylor series based approximations reflect the exact reliability very closely. Comparisons among the various types of correlations, for various IRT models, are made using algebraic expressions, Monte Carlo simulations, and data analysis. Given the light computational burden and the performance of Taylor series based reliability measures, their use is recommended.     

Latent variable models for multivariate longitudinal ordinal responses

The paper proposes a full information maximum likelihood estimation method for modelling multivariate longitudinal ordinal variables. Two latent variable models are proposed that account for dependencies among items within time and between time. One model fits item‐specific random effects which account for the between time points correlations and the second model uses a common factor. The relationships between the time‐dependent latent variables are modelled with a non‐stationary autoregressive model. The proposed models are fitted to a real data set.     

How satisfied are you with your job? Estimating the reliability of scores on a single‐item job satisfaction measure

Estimating the reliability of scores on single‐item measures can be difficult because commonly used internal consistency estimates of reliability cannot be calculated. When longitudinal data is available, statistical models can be used to decompose the variability in the latent variable at each wave into trait versus state variance. Then, reliability can be estimated as a ratio of the sum of the trait variance that is captured in repeated assessments over the total variance. The current study used latent trait‐state‐error models on a nine‐year longitudinal data (N = 5,003) to estimate the test–retest reliability of scores on a single‐item measure of job satisfaction. Results showed that job satisfaction scores were somewhat unreliable (r_xx = .49–.59) and amenable to change.     

Testing the latent factor structure and construct validity of the Ten-Item Personality Inventory

Recent efforts have aimed to develop relatively short measures of the Five-Factor Model (FFM) of personality, particularly for when time and/or space is limited. We evaluate the Ten-Item Personality Inventory (TIPI), a non-proprietary FFM measure with two items per dimension. We use a latent variable methodology to examine the TIPI’s factor structure and convergent validity with the 50-item International Personality Item Pool (IPIP) FFM measure. We provide correlations between the scale scores and latent factors, and compare each measure’s pattern of correlations with measures of other individual difference constructs. Results were favorable in terms of the factor structure and convergent validity of the TIPI, particularly regarding the correlations between the respective latent factors of the TIPI and the IPIP–FFM measures.     

Improving measurement precision of test batteries using multidimensional item response models

A conventional way to analyze item responses in multiple tests is to apply unidimensional item response models separately, one test at a time. This unidimensional approach, which ignores the correlations between latent traits, yields imprecise measures when tests are short. To resolve this problem, one can use multidimensional item response models that use correlations between latent traits to improve measurement precision of individual latent traits. The improvements are demonstrated using 2 empirical examples. It appears that the multidimensional approach improves measurement precision substantially, especially when tests are short and the number of tests is large. To achieve the same measurement precision, the multidimensional approach needs less than half of the comparable items required for the unidimensional approach.     

Measuring the Big Five with single items using a bipolar response scale

The convergence on the Big Five in personality theory has produced a demand for efficient yet psychometrically sound measures. Therefore, five single‐item measures, using bipolar response scales, were constructed to measure the Big Five and evaluated in terms of their convergent and off‐diagonal divergent properties, their pattern of criterion correlations and their reliability when compared with four longer Big Five measures. In a combined sample (N = 791) the Single‐Item Measures of Personality (SIMP) demonstrated a mean convergence of r = 0.61 with the longer scales. The SIMP also demonstrated acceptable reliability, self–other accuracy, and divergent correlations, and a closely similar pattern of criterion correlations when compared with the longer scales. It is concluded that the SIMP offer a reasonable alternative to longer scales, balancing the demands of brevity versus reliability and validity. Copyright © 2005 John Wiley & Sons, Ltd.     

The linear transformation model with frailties for the analysis of item response times

The item response times (RTs) collected from computerized testing represent an underutilized source of information about items and examinees. In addition to knowing the examinees’ responses to each item, we can investigate the amount of time examinees spend on each item. In this paper, we propose a semi‐parametric model for RTs, the linear transformation model with a latent speed covariate, which combines the flexibility of non‐parametric modelling and the brevity as well as interpretability of parametric modelling. In this new model, the RTs, after some non‐parametric monotone transformation, become a linear model with latent speed as covariate plus an error term. The distribution of the error term implicitly defines the relationship between the RT and examinees’ latent speeds; whereas the non‐parametric transformation is able to describe various shapes of RT distributions. The linear transformation model represents a rich family of models that includes the Cox proportional hazards model, the Box–Cox normal model, and many other models as special cases. This new model is embedded in a hierarchical framework so that both RTs and responses are modelled simultaneously. A two‐stage estimation method is proposed. In the first stage, the Markov chain Monte Carlo method is employed to estimate the parametric part of the model. In the second stage, an estimating equation method with a recursive algorithm is adopted to estimate the non‐parametric transformation. Applicability of the new model is demonstrated with a simulation study and a real data application. Finally, methods to evaluate the model fit are suggested.     

Comparing the Performance of Native North Americans and Predominantly White Military Recruits on Verbal and Nonverbal Measures of Cognitive Ability

This study compared the cognitive ability of adult Canadian First Nations (American Indian) aboriginals (N=101) living in remote areas to recruits (N=131) undergoing military training in the Canadian Forces. Comparisons involved both verbal and nonverbal measures: The Canadian Forces Aptitude Test (CFAT), Wonderlic Personnel Test, (WPT), Raven's Standard Progressive Matrices (SPM) and Mill Hill Vocabulary (MHV) test. All measures were examined for differential test and item functioning; as well, the CFAT was analyzed for adverse impact. Confirming past research, First Nations (FNs) members scored, on average, 18 points lower in IQ estimates based on the WPT, a verbal measure of cognitive ability; however, the differences between the groups were less on the nonverbal tests: SPM (5 points lower) and the MHV (9 points lower). Differential Item Functioning (DIF) analysis detected a few items from the CFAT, SPM and MHV that displayed DIF, but none from the WPT. The SPM and WPT appear to be unbiased measures with respect to differential test functioning (DTF), once language and education are controlled. Making employment decisions solely on verbal cognitive ability test scores, however, is likely to produce adverse impact against members of FN, unless verbal ability is a bona fide occupational requirement for the positions under consideration. Both the verbal and nonverbal cognitive ability tests assessed the same latent structure suggesting that the nonverbal tests could be used in place of the verbal tests with little loss in predictive ability for occupations involving a high degree of spatial ability.     

Attachment,Mental Health,and Verbal Aggressiveness in Personal Relationships

This study examined the association between verbal aggressiveness and individual differences in attachment orientations and mental health, defined as a state of emotional well-being. One hundred and thirty-two college-aged students completed measures assessing their attachment orientations, mental health, and verbal aggressiveness. Results indicated that security in attachment was negatively associated with verbal aggressiveness, and preoccupation with relationships was positively associated with verbal aggressiveness. In addition, mental health was negatively associated with verbal aggressiveness. Results indicated support for a statistical model in which mental health mediated the association between attachment orientations and verbal aggressiveness.     

Testing the latent factor structure and construct validity of the Ten-Item Personality Inventory

Recent efforts have aimed to develop relatively short measures of the Five-Factor Model (FFM) of personality, particularly for when time and/or space is limited. We evaluate the Ten-Item Personality Inventory (TIPI), a non-proprietary FFM measure with two items per dimension. We use a latent variable methodology to examine the TIPI’s factor structure and convergent validity with the 50-item International Personality Item Pool (IPIP) FFM measure. We provide correlations between the scale scores and latent factors, and compare each measure’s pattern of correlations with measures of other individual difference constructs. Results were favorable in terms of the factor structure and convergent validity of the TIPI, particularly regarding the correlations between the respective latent factors of the TIPI and the IPIP–FFM measures.     

Instrumental and expressive representations of aggression: One scale or two?

The Expagg questionnaire was developed to measure a subject's view of their own aggression as a relatively instrumental or relatively expressive act. Two issues have been raised pertaining to the dimensional structure of the questionnaire: the use of principal components analysis on dichotomous responses and the possibility that instrumental and expressive representations might be independent dimensions rather than opposite ends of a single continuum. In study 1, dichotomous Expagg data from 405 subjects were subjected to microfact, principal components, and factor analysis. Each produced a first general factor, and the correlations between the item loadings were in excess of r = .99. In study 2, a 40‐item Likert scale version of Expagg was given to 295 subjects. Principal components analysis, paired item correlations, and subscale correlations suggested partial independence of instrumental and expressive items. Two new 8‐item scales measuring instrumental and expressive representations were constructed that maximise their independence. Potential uses of these revised scales are discussed. Aggr. Behav. 25:435–444, 1999. © 1999 Wiley‐Liss, Inc.     

Aggression and bullying

Models are presented describing how bullying others and being bullied are related to proactive and reactive aggressiveness. This was investigated among 1801 pupils in Norwegian schools at the end of grade five and among 2083 pupils at the end of grade eight. Pupils in the two samples were approximately 11 and 14 years old, respectively. The procedure of structural equation modeling with latent variables was followed using LISREL 8.30 to estimate the parameters. Analyses were conducted for boys and girls separately at fifth and eighth grades. Support for the assumptions of factor solution and factorial invariance of the concepts across gender and age was obtained independently from estimating structural parameters. Both proactive and reactive aggressiveness were related to bullying others and to being bullied at Level 5. Among the older pupils, there was a strong relationship between proactive aggressiveness and bullying others, while reactive aggressiveness was much more weakly related to bullying others. Being bullied, on the other hand, was only weakly related to proactive and reactive aggressiveness among the older pupils. Age was therefore an important factor in determining how proactive and reactive aggressiveness were related to bullying. When pupil gender is concerned, power‐related proactive aggressiveness in predicting bullying was a more important factor for boys, while affiliation‐related proactive aggressiveness was a more powerful factor for girls. Aggr. Behav. 27:446–462, 2001. © 2001 Wiley‐Liss, Inc.     

Examining the cross‐cultural validity of the positive affect and negative affect schedule between an Asian (Singaporean) sample and a Western (American) sample

The positive affect and negative affect schedule (PANAS) is a popular measure of positive (PA) and negative affectivity (NA). Developed and validated in Western contexts, the 20‐item scale has been frequently administered on respondents from Asian countries with the assumption of cross‐cultural measurement invariance. We examine this assumption via a rigorous multigroup confirmatory factor analysis, which allows us to assess between‐group differences in both strength of scale item‐to‐latent factor relationship (metric invariance test) and mean of each scale item (scalar invariance test), on a large sample of 1,065 respondents recruited from Singapore (Asian sample) and the United States (Western sample). We found that two items assessing PA (“excited” and “proud”) and three items assessing NA (“guilty,” “hostile,” and “ashamed”) exhibited metric noninvariance whereas 11 of the remaining metric invariant items exhibited scalar noninvariance, suggesting that the PA and NA constructs differ from what the PANAS is expected to measure for Asian respondents. Our findings serve as a cautionary note to researchers who intend to administer the PANAS in future studies as well as to researchers interpreting the results of past studies involving respondents from Asian countries.     

On the Validity of Idiographic and Generic Self‐Concept Implicit Association Tests: A Core‐Concept Model

We present a core‐concept model (CCM) suggesting that stimulus centrality is an important factor in category representations in implicit measures. We tested the hypothesis that idiographic stimuli (first name, birthday) are more central and therefore assess self‐concept in Implicit Association Tests (IATs) more validly than generic and nonspecific stimuli (me, you). Superior validity of the idiographic variant emerged across three different domains of self‐concept. First, an idiographic self‐esteem IAT displayed higher correlations than a generic IAT with self‐assessments and observer‐assessments of self‐esteem. Second, an idiographic body scheme‐IAT predicted subjective ratings of body image and objective body‐mass index. Third, an idiographic aggressiveness‐IAT had higher incremental validity for unprovoked aggression when interacting with explicit measures of aggressiveness. We conclude that idiographic stimuli focus participants' attention on the core features of the self, hence, tapping into self‐related associations to a stronger degree than generic stimuli. Copyright © 2011 John Wiley & Sons, Ltd.     

Verbal aggressiveness and state anxiety of volleyball players and coaches

The purpose of the present study was to examine the relation of verbal aggressiveness and state anxiety (somatic, cognitive, and self-confidence) in sports settings based on the ratings by volleyball coaches and their athletes. The sample consisted of volleyball athletes (n=208; 98 men and 110 women) and their coaches (n=20; 16 men and 4 women). Analysis showed that male volleyball players rated somatic anxiety higher and were more affected by the verbal aggressiveness of their coaches than female volleyball players. No mean differences were significant for male and female coaches on somatic or cognitive anxiety, self-confidence, or verbal aggressiveness. Also, correlation between subscale scores for male and female volleyball players and coaches was found. The correlations of verbal aggressiveness with self-confidence and anxiety were positive for these athletes, leading them to better behavior. This relationship needs further examination in sport settings.     

Syntax for calculation of discounting indices from the monetary choice questionnaire and probability discounting questionnaire

The 27‐item Monetary Choice Questionnaire (MCQ; Kirby, Petry, & Bickel, 1999) and 30‐item Probability Discounting Questionnaire (PDQ; Madden, Petry, & Johnson, 2009) are widely used, validated measures of preferences for immediate versus delayed rewards and guaranteed versus risky rewards, respectively. The MCQ measures delayed discounting by asking individuals to choose between rewards available immediately and larger rewards available after a delay. The PDQ measures probability discounting by asking individuals to choose between guaranteed rewards and a chance at winning larger rewards. Numerous studies have implicated these measures in addiction and other health behaviors. Unlike typical self‐report measures, the MCQ and PDQ generate inferred hyperbolic temporal and probability discounting functions by comparing choice preferences to arrays of functions to which the individual items are preconfigured. This article provides R and SPSS syntax for processing the MCQ and PDQ. Specifically, for the MCQ, the syntax generates k values, consistency of the inferred k, and immediate choice ratios; for the PDQ, the syntax generates h indices, consistency of the inferred h, and risky choice ratios. The syntax is intended to increase the accessibility of these measures, expedite the data processing, and reduce risk for error.     

Associations of students' self-reports of their teachers' verbal aggression, intrinsic motivation, and perceptions of reasons for discipline in Greek physical education classes

In this study were examined associations among physical education teachers' verbal aggressiveness as perceived by students and students' intrinsic motivation and reasons for discipline. The sample consisted of 265 Greek adolescent students who completed four questionnaires, the Verbal Aggressiveness Scale, the Lesson Satisfaction Scale, the Reasons for Discipline Scale, and the Intrinsic Motivation Inventory during physical education classes. Analysis indicated significant positive correlations among students' perceptions of teachers' verbal aggressiveness with pressure/ tension, external reasons, introjected reasons, no reasons, and self-responsibility. Significant negative correlations were noted for students' perceptions of teachers' verbal aggression with lesson satisfaction, enjoyment/interest, competence, effort/importance, intrinsic reasons, and caring. Differences between the two sexes were observed in their perceptions of teachers' verbal aggressiveness, intrinsic motivation, and reasons for discipline. Findings and implications for teachers' type of communication were also discussed and suggestions for research made.     

The Impact of Fallible Item Parameter Estimates on Latent Trait Recovery

In this paper we propose an upward correction to the standard error (SE) estimation of [^(q)]_ML\hat{\theta}_{\mathrm{ML}} , the maximum likelihood (ML) estimate of the latent trait in item response theory (IRT). More specifically, the upward correction is provided for the SE of [^(q)]_ML\hat{\theta}_{\mathrm{ML}} when item parameter estimates obtained from an independent pretest sample are used in IRT scoring. When item parameter estimates are employed, the resulting latent trait estimate is called pseudo maximum likelihood (PML) estimate. Traditionally, the SE of [^(q)]_ML\hat{\theta}_{\mathrm{ML}} is obtained on the basis of test information only, as if the item parameters are known. The upward correction takes into account the error that is carried over from the estimation of item parameters, in addition to the error in latent trait recovery itself. Our simulation study shows that both types of SE estimates are very good when θ is in the middle range of the latent trait distribution, but the upward-corrected SEs are more accurate than the traditional ones when θ takes more extreme values.