Do overall dimension ratings from assessment centres show external construct-related validity?

ABSTRACT

There have been repeated calls for an external construct validation approach to advance our understanding of the construct-related validity of assessment centre dimension ratings beyond existing internal construct-related validity findings. Following an external construct validation approach, we examined whether linking assessment centre overall dimension ratings to ratings of the same dimensions that stem from sources external to the assessment centre provides evidence for construct-related validity of assessment centre ratings. We used data from one laboratory assessment centre sample and two field samples. External ratings of the same dimensions stemmed from assessees, assessees’ supervisors, and customers. Results converged across all three samples and showed that different dimension-same source correlations within the assessment centres were larger than same dimension-different source correlations. Moreover, confirmatory factor analyses revealed source factors but no dimension factors in the latent factor structure of overall dimension ratings from the assessment centre and from external sources. Hence, consistent results across the three samples provide no support that assessment centre overall dimension ratings and ratings of the same dimensions from other sources can be attributed to dimension factors. This questions arguments that assessment centre overall dimension ratings should have construct-related validity.     

Assessment of fun in interactive systems: A survey

Fun is fundamental in life, since it fosters interaction and learning. But the design of fun is not trivial since it is subjective: it depends upon context, preferences and the history of users. That is why assessment is an important issue in the design process. Traditional assessment methods involve observation and inquiring of users while interacting, while more recent methods involve data collection and physiological measurements. This article presents a survey on the existing methods for the assessment of fun from its constituent elements – attention, flow, immersion and emotion.     

Utility of Model of Human Occupation Screening Tool in Sweden

abstract

Occupational therapists in Sweden are expected to use evidence-based practice. To meet this expectation, a helpful tool could be the Model of Human Occupational Screening Tool (MOHOST). The aim of this study was to examine the utility of the Swedish version of MOHOST. Thirty-seven occupational therapists were invited to take part in the examination. A questionnaire covering transferability, feasibility and clinical relevance was developed. The results show that MOHOST-S seems to have suitable utility. The study supports its implementation potential and clinical relevance. It gives a broad picture of the client’s occupational participation and it supports the treatment planning process.     

Development and validation of PediaTrac™: A web-based tool to track developing infants

ObjectivePediaTrac™, a 363-item web-based tool to track infant development, administered in modules of ∼40-items per sampling period, newborn (NB), 2­-, 4­-, 6­-, 9­- and 12­-months was validated. Caregivers answered demographic, medical, and environmental questions, and questions covering the sensorimotor, feeding/eating, sleep, speech/language, cognition, social-emotional, and attachment domains.MethodsExpert Panel Reviews and Cognitive Interviews (CI) were conducted to validate the item bank. Classical Test Theory (CTT) and Item Response Theory (IRT) methods were employed to examine the dimensionality and psychometric properties of PediaTrac with pooled longitudinal and cross-sectional cohorts (N = 132).ResultsIntraclass correlation coefficients (ICC) for the Expert Panel Review revealed moderate agreement at 6 ­months and good reliability at other sampling periods. ICC estimates for CI revealed moderate reliability regarding clarity of the items at NB and 4 months, good reliability at 2-­, 9­- and 12­-months and excellent reliability at 6 ­months. CTT revealed good coefficient alpha estimates (α ≥ 0.77 for five of the six ages) for the Social-Emotional/Communication, Attachment (α ≥ 0.89 for all ages), and Sensorimotor (α ≥ 0.75 at 6-months) domains, revealing the need for better targeting of sensorimotor items. IRT modeling revealed good reliability (r = 0.85–0.95) for three distinct domains (Feeding/Eating, Social-Emotional/Communication and Attachment) and four subdomains (Feeding Breast/Formula, Feeding Solid Food, Social-Emotional Information Processing, Communication/Cognition). Convergent and discriminant construct validity were demonstrated between our IRT-modeled domains and constructs derived from existing developmental, behavioral and caregiver measures. Our Attachment domain was significantly correlated with existing measures at the NB and 2-month periods, while the Social-Emotional/Communication domain was highly correlated with similar constructs at the 6-, 9- and 12-month periods.ConclusionPediaTrac has potential for producing novel and effective estimates of infant development via the Sensorimotor, Feeding/Eating, Social-Emotional/Communication and Attachment domains.     

Using targeted visceroception to improve interoceptive sensibility and emotion regulation

Focusing on one's body can improve the awareness and regulation of emotion. Interoception – sensing the physiological condition of the body, particularly of the viscera – appears to play an important role. While the majority of previous research studies have examined interoceptive sensitivity (i.e., detection of objectively measurable physiological changes), there has been relatively limited investigation of interoceptive sensibility (i.e., subjective awareness of those changes), and even fewer studies permitting causal conclusions. The current study is part of a randomised controlled trial on visceroception in the context of emotion regulation, using an 8-week intervention involving focused attention on either cardiac or gastrointestinal activity. Group differences in emotion regulation and reactivity were assessed using the Multidimensional Assessment of Interoceptive Awareness (MAIA), the Difficulties in Emotion Regulation Scale, and the Emotion Reactivity Scale (ERS). The findings suggest that focussed attention on gastrointestinal activity (i.e., gastroception), in particular, improves interoceptive sensibility and emotion regulation, as evidenced by increases on the MAIA's Noticing, Body Listening, and Self-regulation subscales, and decreases on the Arousal subscale of the ERS. Gastroception may make distinct contributions to the benefits of body focus in a research context where the gut is often overlooked.     

Novel use of a virtual driving assessment to classify driver skill at the time of licensure

Motor vehicle crash rates are highest immediately after licensure, and driver error is one of the leading causes. Yet, few studies have quantified driving skills at the time of licensure, making it difficult to identify at-risk drivers before independent driving. Using data from a virtual driving assessment implemented into the licensing workflow in Ohio, this study presents the first population-level study classifying degree of skill at the time of licensure and validating these against a measure of on-road performance: license exam outcomes. Principal component and cluster analysis of 33,249 virtual driving assessments identified 20 Skill Clusters that were then grouped into 4 major summary “Driving Classes”; i) No Issues (i.e. careful and skilled drivers); ii) Minor Issues (i.e. an average new driver with minor vehicle control skill deficits); iii) Major Issues (i.e. drivers with more control issues and who take more risks); and iv) Major Issues with Aggression (i.e. drivers with even more control issues and more reckless and risk-taking behavior). Category labels were determined based on patterns of VDA skill deficits alone (i.e. agnostic of the license examination outcome). These Skill Clusters and Driving Classes had different distributions by sex and age, reflecting age-related licensing policies (i.e. those under 18 and subject to GDL and driver education and training), and were differentially associated with subsequent performance on the on-road licensing examination (showing criterion validity). The No Issues and Minor Issues classes had lower than average odds of failing, and the other two more problematic Driving Classes had higher odds of failing. Thus, this study showed that license applicants can be classified based on their driving skills at the time of licensure. Future studies will validate these Skill Cluster classes in relation to their prediction of post-licensure crash outcomes.     

Assessing competencies of trainee sport psychologists: An examination of the ‘structured case presentation’ assessment method

ObjectivesThere is virtually no literature on how to assess competencies of applied sport psychologists. We assessed casework of applied sport psychology students and compared written case report assessment (WCRA) with structured case presentation assessment (SCPA) on reliability and acceptability (e.g., validity, transparency, feedback function and preference of methods, as perceived by students and assessors).DesignA quantitative, comparative study of two assessment methods.MethodParticipants were 11 students, nine supervisors and three exam committee members. A number of 18 cases were evaluated with both WCRA by the supervisor and SCPA by two exam committee members. Ten of these cases were also evaluated with WCRA by exam committee members. Interrater reliability measures were calculated and compared for the different assessment methods. Participants' perception of the validity, transparency, and feedback function of the methods, and the preferences for assessment methods were surveyed with a brief questionnaire.ResultsSCPA by the exam committee resulted in higher interrater reliability than WCRA by supervisor and exam committee. The feedback function of SCPA seemed superior to WCRA by either supervisor or exam committee. For assessment by the exam committee, the perceived validity and transparency of SCPA seemed higher than of WCRA. Students and exam committee had the highest preference for SCPA by supervisor and exam committee.ConclusionsOverall it can be concluded that, for assessment by the exam committee, structured case presentations provided a more reliable and acceptable method of assessment than written case reports only.