| 问题解决测验中过程数据的特征抽取与能力评估 |
| |
| 引用本文： | 韩雨婷,肖悦,刘红云. 问题解决测验中过程数据的特征抽取与能力评估[J]. 心理科学进展, 2022, 30(6): 1393-1409. DOI: 10.3724/SP.J.1042.2022.01393 |
| |
| 作者姓名： | 韩雨婷 肖悦 刘红云 |
| |
| 作者单位： | 1.北京大学医学部全国医学教育发展中心, 北京 100191;2.应用实验心理北京市重点实验室;3.北京师范大学心理学部, 北京 100875 |
| |
| 基金项目： | *国家自然科学基金项目(32071091);国家自然科学基金青年项目(72104006);北京大学引进人才计划与启动基金(BMU2021YJ010) |
| |
| 摘 要： | 基于计算机的问题解决测验可以实时记录被试探索环境和解决问题时的详细行动痕迹,并保存为过程数据。首先介绍了过程数据的分析流程,然后从问题解决测验入手,分别对过程数据的特征抽取和能力估计建模两方面的研究进行了梳理和评价。未来研究应注意：提高分析结果的可解释性;特征提取时纳入更多信息;实现更复杂问题情景下的能力评估;注重方法的实用性;以及融合与借鉴不同领域的分析方法。
|
| 关 键 词： | 计算机问题解决测验 过程数据 特征抽取 能力评估模型 |
| 收稿时间： | 2021-08-04 |
Feature extraction and ability estimation of process data in the problem-solving test |
| |
| HAN Yuting,XIAO Yue,LIU Hongyun. Feature extraction and ability estimation of process data in the problem-solving test[J]. Advances In Psychological Science, 2022, 30(6): 1393-1409. DOI: 10.3724/SP.J.1042.2022.01393 |
| |
| Authors: | HAN Yuting XIAO Yue LIU Hongyun |
| |
| Affiliation: | 1.National Center for Health Professions Education Development, Peking University Health Science Center, Beijing 100191, China;2.Beijing Key Laboratory of Applied Experimental Psychology;3.Faculty of Psychology, Beijing Normal University, Beijing 100875, China |
| |
| Abstract: | Computer-based problem-solving tests can record respondents’ response processes when they explore tasks and solve problems as process data, which is richer in information than traditional outcome data and can be used to estimate latent abilities more accurately. The analysis of process data in problem solving tests consists of two main steps: feature extraction and process information modeling. There are two main approaches to extracting information from process data: top-down and bottom-up method. The top-down method refers to developing rubrics by experts to extract meaningful behavioral indicators from process data. This approach extracts behavioral indicators that are closely related to the conceptual framework, have interpretable and clear scores, and can be analyzed directly using psychometric models, as is the case with items in traditional tests. However, such indicator construction methods are laborious and may miss unknown and previously unnoticed student thought processes, resulting in a loss of information. In contrast, the bottom-up method refers to the use of data-driven approaches to extract information directly from response sequences, which can be divided into the following three categories according to their processing ideas: (1) methods that analogize response sequences to character strings and construct indicators by natural language processing techniques; (2) methods that use dimensionality reduction algorithms to construct low-dimensional numerical feature vectors of response sequences; and (3) methods that use directed graphs to characterize response sequences and use network indicators to describe response features. Such methods partially address the task specificity in establishing scoring rules by experts, and the extracted features can be used to explore the behavioral patterns characteristic of different groups, as well as to predict respondents’ future performance. However, such methods may also lose information, and the relationship between the obtained features and the measured psychological traits is unclear. After behavioral indicators have been extracted from process data, probabilistic models that model the relationship between the indicators and the latent abilities can be constructed to enable the estimation of abilities. Depending on whether the model makes use of sequential relationships between indicators and whether continuously interpretable estimates of latent abilities can be obtained, current modeling methods can be divided into the following three categories: traditional psychometric models and their extensions, stochastic process models, and measurement models that incorporate the idea of stochastic processes. Psychometric models focus on estimates of latent abilities but are limited by their assumption of local independence and cannot include sequential information between indicators in the analysis. The stochastic process model focuses on modeling the response process, retaining information about response paths, but with weaker assumptions between indicators and underlying structure, and is unable to obtain continuous and stable estimates of ability. Finally, psychometric models that incorporate the idea of stochastic processes combine the advantages of both taking the sequence of actions as the object of analysis and having experts specify indicator coefficients or scoring methods that are consistent with the direction of abilities, thus allowing continuous interpretable estimates of abilities to be obtained while using more complete process information. However, such modeling methods are mostly suitable for simple tasks with a limited set of actions thus far. There are several aspects where research on feature extraction and capability evaluation modeling of process data could be improved: (1) improving the interpretability of analysis results; (2) incorporating more information in feature extraction; (3) enabling capability evaluation modeling in more complex problem scenarios; (4) focusing on the practicality of the methods; and (5) integrating and drawing on analytical methods from different fields. |
| |
| Keywords: | computer-based problem-solving test process data feature extraction capability evaluation model |
|
| 点击此处可从《心理科学进展》浏览原始摘要信息 |
|
点击此处可从《心理科学进展》下载全文 |
|
