Modelling Cognitive Bias in Crowdsourcing Systems

Crowdsourcing is the process of obtaining input or enlisting the services of a crowd of people with the aim of finding the best solution to a proposed problem. The method with which the collected input is aggregated affects the outcome of the crowdsourcing process. In this paper, we introduce a modelling framework through which we compare several aggregation methods for crowdsourcing systems. The work reveals a surprising result where confidence-related approaches lack in performance when compared to other approaches such as simple plurality voting or approaches which consider respondent competence. This inadequacy stems from a psychological phenomenon brought forth by David Dunning and Justin Kruger related to people’s bias in assessing their own cognitive abilities.