MTurk ‘Unscrubbed’: Exploring the Good, the ‘Super’, and the Unreliable on Amazon's Mechanical Turk
Jeanette Deetlefs, UNSW Australia Business School, School of Marketing; Mathew Chylinski, University of New South Wales (UNSW); Andreas Ortmann , UNSW Australia Business School, School of Economics
UNSW Business School Research Paper No. 2015-20A
October 12, 2015
Abstract:
"[T]wo commonly raised concerns remain: the presence of quasi-professional respondents, or “Super-Turkers”, and the presence of “Spammers”, those that compromise quality while optimising their pay rate. We isolate the influence on research results of experienced subjects (Super-Turkers), and of unreliable subjects (Spammers), jointly and separately. Jointly including these subjects produces very similar results to jointly excluding them, yet effect sizes decrease disproportionately to their sample representation. ...Hence removing only one of these types of respondents can be even more damaging to the reliability of results, than including both."
Number of Pages in PDF File: 45
Jeanette Deetlefs, UNSW Australia Business School, School of Marketing; Mathew Chylinski, University of New South Wales (UNSW); Andreas Ortmann , UNSW Australia Business School, School of Economics
UNSW Business School Research Paper No. 2015-20A
October 12, 2015
Abstract:
"[T]wo commonly raised concerns remain: the presence of quasi-professional respondents, or “Super-Turkers”, and the presence of “Spammers”, those that compromise quality while optimising their pay rate. We isolate the influence on research results of experienced subjects (Super-Turkers), and of unreliable subjects (Spammers), jointly and separately. Jointly including these subjects produces very similar results to jointly excluding them, yet effect sizes decrease disproportionately to their sample representation. ...Hence removing only one of these types of respondents can be even more damaging to the reliability of results, than including both."
Number of Pages in PDF File: 45