It is challenging to come up with a new, original idea on what to say about an issue that’s been going on for years: unfair mass rejections.
It has been more than 14 days since requester AI Insights immediately rejected all of the HITs submitted to them without any feedback or communication. Assignments (submitted tasks) can be reversed within 30 days of submission. Currently, the requester has less than 16 days to overturn the rejections. We, the workers, can only hope these will be overturned while our approval ratings and ability to work on the platform hang in the balance.
This requester had an approval rating of 99% before posting these HITs. Using this information, workers felt safe completing the HITs, and the batch, made up of over 75,000 tasks, disappeared quickly. The HITs were simple choice tasks and seemed like a great batch until the rejections came in. The requester’s approval rating promptly fell to 5% the following day. [If you want to read some of the nightmare-ish reviews that workers have left, you can go to Turkopticon, Turkerview, or Reddit (1 / 2 / 3 / 4 / 5).]
Even worse, this is not an isolated case. There have been reports of workers’ submissions being rejected because, among other reasons, the requester had an error on their side (like Team Seminar & Fan Zeng), cancelled the study at the last minute (like SeunghyunShin), or gave vague feedback like “Wrong Answer” (Janine Chen, which has an approval rating of 2%) or “Doesn’t meet my requirements” (Pedro Amaro) only for the approval rating to disappear from the platform without any further explanation, or even took the responsibility to Amazon Mechanical Turk saying it is with them that we, the workers, have a contractual relationship and thus “they are the ones who should face the consequences” (Victor Ramos).
According to Amazon’s Developer Guide, it’s recommended that requesters “reject work when workers are clearly putting in no effort to submit an accurate response to your task. It’s inappropriate to penalize a worker for submitting data incorrectly because you provided unclear instructions or they simply made a mistake in interpreting what you wanted them to do. Most workers zealously guard their approval rating and avoid doing work for requesters that they believe are unfair in how they reject work”.
If you ever wondered what a big subsidiary of Amazon comes up with as a way of protecting workers, there you have it, it is an article. This is the same “solution” offered to the Turkopticon team during their last meeting with the Mturk leadership.
Amazon needs to address this issue seriously. Washing its hands of the matter while saying, “we are unable to mediate in disputes between Workers and Requesters” is not a solution, especially if workers and requesters are to believe that Mechanical Turk is an ethical and fair platform for everyone.
Other marketplaces have developed better mechanisms for handling complaints, appeals, and bad actors, so why is Amazon unable to do the same thing? Don’t they have enough resources to work on this?
Even if you were not affected by this issue with any of these requesters, it shows that something like this can happen to you and your account at any time and that you will not have protection or recourse.
Per Amazon’s suggestion, workers should report any questionable behavior on their platform, so we invite you to send a polite and respectful email to mturk-worker-support@amazon.com.
We are not “ghost workers”, our livelihoods and needs are real and affected by your inaction.
-The Turk