When proportion consensus scoring works

Most objectively scored tests use items with easily identifiable correct answers. When such veridical scoring keys cannot be constructed, researchers sometimes use proportion consensus scoring (PCS) to identify the best answers. To determine if PCS identifies the best answers, we scored a test using both PCS and veridical scoring. Among 353 undergraduates, regular PCS, two-stage PCS, and expert PCS all had high correlations for easy items, but no PCS methods had high correlations for difficult items. Thus, PCS cannot reliability identify the best answers to individual items. However, PCS worked well for total scores. For easy items, total scores had correlations above .99 for all PCS methods. For difficult items, expert and two-stage PCS had correlations of .92 and .82 for the 60-item test. Thus, expert and two-stage PCS can be justified (even for difficult items) if the scoring key is based upon people who truly possess some degree of expertise and if scores are summed over many items.