一种基于多阶认知诊断模型测评科学素养的方法

科学素养是指作为一名有反思意识的公民所具有的解决科学问题和运用科学理念的能力。为实现在认知诊断中对科学素养的测评, 本文基于PISA 2015科学素养测评框架首次提出科学素养包含的三阶潜在结构, 使用新提出的多阶认知诊断模型对PISA 2015科学测评数据进行分析, 并通过模拟研究探究新模型的心理测量学性能。结果表明：(1)新模型能够较好地分析包含三阶潜在结构的科学素养; (2)科学知识对科学素养的影响最大, 科学背景次之, 科学能力的影响最小; (3)全贝叶斯MCMC算法能够为新模型提供较精准的参数估计。     

Item Calibration Samples and the Stability of Achievement Estimates and System Rankings: Another Look at the PISA Model

Using an empirically-based simulation study, we show that typically used methods of choosing an item calibration sample have significant impacts on achievement bias and system rankings. We examine whether recent PISA accommodations, especially for lower performing participants, can mitigate some of this bias. Our findings indicate that standard operational methods, while not ideal, recover underlying proficiency reasonably well and generally outperform methods that more completely include all participants. Translating results onto the PISA scale, the calibration sample can induce bias of up to 12.49 points, which is important given that standard errors are around three points. Although ranking correlations are at least.95, we note the policy implications of slight ranking changes. Our findings indicate that limited accommodations targeted at low achieving educational systems do not outperform either of the other methods considered. Research that further explores accommodations for heterogeneous populations is recommended.     

International Semiotics: Item Difficulty and the Complexity of Science Item Illustrations in the PISA-2009 International Test Comparison

We examined multimodality (the representation of information in multiple semiotic modes) in the context of international test comparisons. Using Program of International Student Assessment (PISA)-2009 data, we examined the correlation of the difficulty of science items and the complexity of their illustrations. We observed statistically significant mean correlation coefficient differences favoring higher-ranking over lower-ranking jurisdictions and non-Western over Western jurisdictions. We examined this correlation as a function of the variety of forms of representation of information used in illustrations. For the highest-ranking jurisdictions, the correlation tended to be positive and its magnitude tended to increase with the number of forms of representation. For the lowest-ranking jurisdictions, the correlation tended to be negative and its magnitude tended to increase with the number of forms of representation. Increased illustration complexity appears to be an affordance in making sense of science items for students from the highest-ranking jurisdictions and a challenge for students from the lowest-ranking jurisdictions. Our findings support the notion that integrating information provided in multiple semiotic modes is critical to science achievement and underscore the importance for science standards and international assessment frameworks to address this skill as intrinsic to (rather than an aspect of) science proficiency.     

Incongruence Between Native and Test Administration Languages: Towards Equal Opportunity in International Literacy Assessment

Previous research has primarily addressed the effects of language on the Program for International Student Assessment (PISA) mathematics and science assessments. More recent research has focused on the effects of language on PISA reading comprehension and literacy assessments on student populations in specific Organization for Economic Cooperation and Development (OECD) and non-OECD countries. Recognizing calls to highlight the impact of language on student PISA reading performance across countries, the purpose of this study was to examine the effect of home languages versus test languages on PISA reading literacy across OECD and non-OECD economies, while considering other factors. The results of Ordinary Least Squares regression showed that about half of the economies demonstrated a positive and significant effect of students' language status on their reading performance. This finding is consistent with observations in the parallel analysis of PISA 2009 data, suggesting that students' performance on reading literacy assessment was higher when they were tested in their home language. Our findings highlight the importance of the role of context, the need for new approaches to test translation, and the potential similarities in language status for youth from OECD and non-OECD countries that have implications for interpreting their PISA reading literacy assessments.     

Modeling differentiation of cognitive abilities within the higher-order factor model using moderated factor analysis

Receptive foreign language proficiency is usually measured with reading and listening comprehension tasks. A novel approach to assess such proficiencies – viewing comprehension – is based on the presentation of short instructional videos followed by one or more comprehension questions concerning the preceding video stimulus. In order to evaluate a newly developed viewing comprehension test 485 German high school students completed reading, listening, and viewing comprehension tests, all measuring the receptive proficiency in English as a foreign language. Fluid and crystallized intelligence were measured as predictors of performance. Relative to traditional comprehension tasks, the viewing comprehension task has similar psychometric qualities. The three comprehension tests are very highly but not perfectly correlated with each other. Relations with fluid and crystallized intelligence show systematic differences between the three comprehension tasks. The high overlap between foreign language comprehension measures and between crystallized intelligence and language comprehension ability can be taken as support for a uni-dimensional interpretation. Implications for the assessment of language proficiency are discussed.     

Rash客观等距测量在PISA中国试测研究中的实践

Rasch测量是当前心理测量中具有客观等距量尺的测量,克服了传统经典测量的测验依赖和样本依赖的局限。以学生能力国际评价PISA中国试测研究为例,说明PISA如何应用Rasch测量达到跨越国家和地区教育成效比较的测量目的。客观等距量尺研究对改进和完善我国心理测量与教育评价具有重要参考价值。     

Cross-country gender DIF in PISA science literacy items

This study investigated gender based differential item functioning (DIF) in science literacy items included in the Program for International Student Assessment (PISA) 2012. Prior research has suggested presence of such DIF in large scale surveys. Our study extends the empirical literature by examining gender based DIF differences at the country level in order to gain a better overall picture of how cultural and national differences affect occurrence of uniform and nonuniform DIF. Our statistical results indicate existence of widespread gender based DIF in PISA with estimates of percentage of potentially biased items ranging between 2 and 44% (M = 16, SD = 9.9). Our reliance on nationally representative country samples allow these findings to have wide applicability.     

Attribution of apparent arousal and proficiency of recovery from sympathetic activation affecting excitation transfer to aggressive behavior

In a pretest, subjects' proficiency to recover from sympathetic arousal induced by strenuous exercise was assessed. The results were used to determine conditions of high, intermediate, and low recovery proficiency (fitness). After an assessment of subjects' unprovoked aggressiveness, subjects were aggressively provoked. Within proficiency blocks, they were then given one of two treatments, (a) sitting followed by exercising (no decay) or (b) exercising followed by sitting (partial decay), and were there-after provided with an opportunity to retaliate against their tormentor. Under conditions of no decay, in which the high levels of arousal experienced were attributable to exertion, the provocation treatment failed to increase aggressiveness significantly, and there were no differences in aggressiveness in the various proficiency conditions in spite of differentiations in the magnitude of prevailing excitatory residues. Under partial decay, in the absence of cues linking arousal to exertion, the magnitude of residual arousal did affect aggressive behavior: In the conditions of intermediate and low recovery proficiency, aggressiveness increased significantly with provocation and was more pronounced than in the condition of high proficiency (best fitness); in the condition of low proficiency (least fitness) aggressiveness was higher than in the condition of intermediate proficiency, but not reliably so.     

Being transformed: Delivering a psychotherapeutic group intervention targeting repeat suicide attempts

Clients who make repeat suicide attempts have complex needs, are at high risk of completed suicide and pose a range of challenges for clinicians. Targeted interventions are required. Acceptability, the perceived appropriateness of an intervention, is associated with successful implementation, engagement and effectiveness of interventions. Despite the advantages of group interventions, there is limited research into the acceptability of groups with this population. Using Interpretative Phenomenological Analysis, this paper reports on facilitators’ experiences (n = 9) of the acceptability of a therapeutic group intervention (PISA – a Psychoeducational/psychosocial Intervention for persons who make recurrent Suicide Attempts). Analysis of focus group data revealed how involvement with the intervention profoundly influenced facilitators’ perceptions of and approach to their work with this client group. They embraced the model, developed a deeper appreciation of clients’ resources and became unburdened from sole responsibility for the clients, the therapeutic process and outcomes. PISA provided a model that helped facilitators to work in a focused, compassionate and creative manner as they addressed the central concern of these clients, their suicidal desire, with newly found confidence and eagerness. PISA was experienced as acceptable to facilitators as it was useful to them and deemed appropriate and beneficial to the target population.     

Analyses of Model Fit and Robustness. A New Look at the PISA Scaling Model Underlying Ranking of Countries According to Reading Literacy

This paper addresses methodological issues that concern the scaling model used in the international comparison of student attainment in the Programme for International Student Attainment (PISA), specifically with reference to whether PISA’s ranking of countries is confounded by model misfit and differential item functioning (DIF). To determine this, we reanalyzed the publicly accessible data on reading skills from the 2006 PISA survey. We also examined whether the ranking of countries is robust in relation to the errors of the scaling model. This was done by studying invariance across subscales, and by comparing ranks based on the scaling model and ranks based on models where some of the flaws of PISA’s scaling model are taken into account. Our analyses provide strong evidence of misfit of the PISA scaling model and very strong evidence of DIF. These findings do not support the claims that the country rankings reported by PISA are robust.     

“Equal access” without “establishing” religion: The necessity for assessing social perspective-taking skills and institutional atmosphere

We concur withMoshman (1990) that in public high schools there should be unimpeded religious freedom if an empirical assessment shows students have the proficiency to evaluate and freely choose religious systems and practices and to discern attempts by authorities to establish a religion. We differ with Moshman,however, regarding (a) the relative importance of a school's ethical, moral and political climate in influencing the exercise of free religious and political choice; (b) his focus on religious over political freedoms; (c) what kinds of proficiencies need to be assessed; (d) how to assess these proficiencies; (e) his claim that that the reasoning ability of high school and college students is similar; and (f) the manner in which decisions permitting religious clubs in high schools should be made. Reasoning proficiency must be assessed in high school, as well as post-high school, populations of students, teachers, and administrators using a social perspective-taking task with establishment of a religion content. A school's institutional atmosphere must be assessed.     

Reading Achievement and Science Proficiency: International Comparisons From the Programme on International Student Assessment

Students need to develop scientific literacy in order to participate fully as citizens, community members, and in the globalized economy. But what is the relationship between scientific literacy and reading literacy? Three international data sets from the Programme on International Student Assessment (PISA) were used to calculate correlations between scientific literacy and reading literacy for 15-year-old students. Mean correlations at the individual student level across countries were .840 for the PISA 2000 data set, .805 for the PISA 2003 data set, and .819 for the PISA 2006 data set. In all three data sets, this correlation varied among countries, and the reading-science relationship was weakest in countries with low country mean reading scores. Three possible interpretations are discussed, favoring the interpretation that knowledge and skills that drive higher reading comprehension also drive higher science achievement.     

Assessing Training Needs: Do Work Experience and Capability Matter?

Despite the crucial nature of needs assessment in training design and development, very little empirical work examining factors that influence such ratings has been undertaken. We investigated the impact of individuals' levels of work experience, self-efficacy, and skill proficiency on their subsequent ratings of training needs. Our results indicate that self-efficacy and skill proficiency are positively related to importance and frequency ratings for both skills and job tasks. However, when considered collectively, skill proficiency was a more potent influence, suggesting that an individual's actual capability may have a greater impact on ratings of training needs than his or her perceived capability (i.e., self-efficacy). Implications for future research and training needs assessment practice are discussed.     

National IQs predict educational attainment in math,reading and science across 56 nations

The results of the 2006 PISA (Program for International Student Assessment) study of reading comprehension, mathematical ability, and science understanding administered to 15 year olds in 56 countries [OECD (2007). PISA 2006: Science Competencies for Tomorrow's World. Paris: OECD.] are examined to assess the predictive validity of the national IQs presented by Lynn and Vanhanen [Lynn, R., & Vanhanen, T. (2002). IQ and the wealth of nations. Westport, CT: Praeger., Lynn, R., & Vanhanen, T. (2006). IQ and global inequality. Augusta, GA: Washington Summit Books.], and to assess the contributions of national differences in IQ and educational variables to national differences in educational attainment. It was found that national scores in reading comprehension, mathematical ability, and science understanding are correlated with Lynn & Vanhanen (L & V) national IQs at 0.84; corrected for attenuation, 0.935. This establishes the high validity of Lynn & Vanhanen national IQs. The contribution of national differences in IQ and education variables to national differences in educational attainment obtained in the 2006 PISA 56 nation study showed that the predictive validity of IQ alone was 0.84, and that national IQs together with one economic and two education variables had the validity 0.90 in predicting PISA 2006 results.     

Comparison of Hispanic children with and without limited English proficiency on the Naglieri Nonverbal Ability Test

Hispanic children with (n=148) and without (n=148) limited English proficiency were given the Naglieri Nonverbal Ability Test (NNAT; J. A. Naglieri. 1997a) and the Stanford Achievement Test-9th edition (SAT-9: 1995). The groups were selected from the NNAT standardization sample (N=22.620) and matched on geographic region, gender, socioeconomic status, urbanicity. and ethnicity. There was a very small difference (d ratio=0.1) between the NNAT standard scores for the children with limited English proficiency (M - 98.0) and those without limited English proficiency (M=96.7). The NNAT correlated moderately and similarly with achievement for the 2 groups. The sample of children with limited English proficiency earned considerably lower scores on SAT-9 Reading and Verbal subtests. Results suggest that the NNAT may be useful for the assessment of Hispanic children with and without limited English proficiency.     

Protocols for the Investigation of Information Processing in Human Assessment of Fundamental Movement Skills

Fundamental movement skill (FMS) assessment remains an important tool in classifying individuals’ level of FMS proficiency. The collection of FMS performances for assessment and monitoring has remained unchanged over the last few decades, but new motion capture technologies offer opportunities to automate this process. To achieve this, a greater understanding of the human process of movement skill assessment is required. The authors present the rationale and protocols of a project in which they aim to investigate the visual search patterns and information extraction employed by human assessors during FMS assessment, as well as the implementation of the Kinect system for FMS capture.     

Does Wechsler Intelligence Scale administration and scoring proficiency improve during assessment training?

Studies have found that Wechsler scale administration and scoring proficiency is not easily attained during graduate training. These findings may be related to methodological issues. Using a single-group repeated measures design, this study documents statistically significant, though modest, error reduction on the WAIS-III and WISC-III during a graduate course in assessment. The study design does not permit the isolation of training factors related to error reduction, or assessment of whether error reduction is a function of mere practice. However, the results do indicate that previous study findings of no or inconsistent improvement in scoring proficiency may have been the result of methodological factors. Implications for teaching individual intelligence testing and further research are discussed.     

The Reading Literacy test for Secondary Education (CompLEC)

A new test to evaluate reading literacy, the Test of Reading Literacy for Secondary Education (CompLEC) is presented. CompLEC is based on the PISA assessment framework and new definitions of reading literacy. The test, easy to apply and score, assesses the level of reading literacy of children between 11 and 14 years of age in several reading situations (i.e., public, educational, personal and occupational) and with different types of texts (i.e., continuous and non-continuous). The scale has been standardized with a sample of 1,854 students from five different Spanish regions. Empirical results show that CompLEC is a homogeneous, reliable and valid instrument.     

A latent topic model with Markov transition for process data

We propose a latent topic model with a Markov transition for process data, which consists of time-stamped events recorded in a log file. Such data are becoming more widely available in computer-based educational assessment with complex problem-solving items. The proposed model can be viewed as an extension of the hierarchical Bayesian topic model with a hidden Markov structure to accommodate the underlying evolution of an examinee's latent state. Using topic transition probabilities along with response times enables us to capture examinees' learning trajectories, making clustering/classification more efficient. A forward-backward variational expectation-maximization (FB-VEM) algorithm is developed to tackle the challenging computational problem. Useful theoretical properties are established under certain asymptotic regimes. The proposed method is applied to a complex problem-solving item in the 2012 version of the Programme for International Student Assessment (PISA).     

A cognitive perspective on Singaporean primary school pupils' use of reading strategies in learning to read in English

Background . This study is conducted in Singapore, where learning to read in English is regarded as essential because it is offered as a First Language (L1) subject in the curriculum and is stipulated as the medium of instruction in the education system, and the mother tongues are offered as Second Language (L2) subjects, although the majority still learn English as an L2. Aims . The paper reports on the reading strategies used by Singaporean primary school pupils from a cognitive perspective, which is part of a larger study that aims to investigate these pupils' language learning strategies. Sample . The participants were 18 pupils from three neighbourhood primary schools, in grades Primary 4, 5 and 6. Method . The data were collected from high‐ and low‐proficiency pupils at each of the three grades in each school, who read two texts at each level. Grounded in an information‐processing theory and based on successful experiences of scholars using think‐aloud for data collection, we asked the pupils to read and report what they were thinking about while reading. The think‐aloud protocols were recorded, transcribed verbatim, coded and analysed. Results . The results suggest that participants' flexible and appropriate use of reading strategies varies according to language proficiency and grade level, with the high‐proficiency group outperforming its lower‐proficiency counterpart and the high‐graders outnumbering the lower‐graders in terms of the number of strategies that they used. These differences were also exemplified with qualitative findings from case studies. Conclusions . The use of reading strategies differs according to proficiency levels, and the quality of pupils' strategy‐use patterns has more significant implications for understanding efficient reading among primary school pupils.