Do We Need Complex Incentives for Belief Elicitation Online?
Banko-Ferran, Burdea & Woon: "A Horserace of Methods for Eliciting Induced Beliefs Online" CRC Discussion Paper No. 562
In experimental economics, the standard practice for eliciting beliefs relies on complex, incentive-compatible mechanisms like the Binarized Scoring Rule (BSR) and the stochastic Becker-DeGroot-Marschak (BDM) mechanism. A recent study by Daniel Banko-Ferran (University of Pittsburgh), Valeria Burdea (LMU Munich, Project B01), and Jonathan Woon (University of Pittsburgh) conducts a “horserace” of these methods against simple, unincentivized introspection in an online environment. Their results suggest that the theoretical benefits of incentive-compatible methods do not always translate into better data quality when implemented in diverse online participant pools.
The researchers found that both BDM and BSR impose significantly higher cognitive costs on participants, who reported greater perceived difficulty and required more time to complete tasks compared to those using introspection. More importantly, these complex incentives did not improve the accuracy of reported beliefs on average relative to the unincentivized benchmark. In fact, the BSR method led to systematically larger errors in belief reporting than simple introspection. While BDM also resulted in higher errors than introspection, these deviations were less systematic than those found with BSR.
Individual differences in probabilistic reasoning, or numeracy, played an important role in how participants responded to these mechanisms. High numeracy skills helped mitigate errors under the BDM mechanism, but they provided no such benefit for the BSR method, which performed poorly across all skill levels. Interestingly, participants who found BDM more difficult often invested more cognitive effort, which actually improved their accuracy, a phenomenon not observed with the BSR method.
Ultimately, the authors argue that for online research, simpler approaches like introspection may offer comparable or even superior accuracy at a lower cost. These findings suggest that discussions on which belief elicitation method is theoretically correct may be less beneficial for researchers designing online experiments than considering which method produces the best data given the characteristics of the sample.
Link (pdf): A Horserace of Methods for Eliciting Induced Beliefs Online
