A comparison of stimulus ratings made online and in person: Gender and method effects

In Experiment 1, separate samples rated nouns on danger, using either an online survey or the same survey in person. In Experiment 2, a single sample rated words on familiarity, using both methods. Women’s in-person and online ratings correlated significantly better than men’s. In-person ratings correlated significantly better with existing norms in 4 of 8 instances. There were significant effects of condition on mean ratings and completion times. Ratings from participants who withdrew from the experiment correlated significantly less well with existing norms than did ratings from those who completed the whole experiment, in 12 of 16 instances. Analysis of existing data showed that a different statistical conclusion is reached depending on whether in-person or online ratings are used. Furthermore, the categorization of 17.9% (Experiment 1) and 5.3% (Experiment 2) of the items as high or low depends on which ratings are used. Ratings gathered in person and online cannot be freely substituted.