Gotta catch'em all: recognizing sloppy work in crowdsourcing tasks | PyData Amsterdam 2016
Crowdsourced work can be a solution to many problems from data labeling, to gathering subjective opinions, to producing transcripts etc. Turns out it can also work really well for functional software testing - but it’s not easy to get right.
One well-known problem with crowdsourcing is sloppy work - where people perform only the absolute minimum actions allowing them to get paid, without actually fulfilling the intended tasks. In many scenarios this can be counteracted by asking multiple workers to complete the same task, but that dramatically increases cost and can still be error-prone. Detecting lazy work is another way to increase quality of gathered data and we have found a way to do this reliably for quite a large variety of tasks.
In this talk, I describe how we have trained a machine learning model to discriminate between good and sloppy work.