Amazon, we need some answers.
It was August 19th when Turkopticon first heard of the mass rejection dropped on turkers by AI Insights. Over the next thirty days, workers emailed the requester, workers emailed Amazon, and allies (some of them requesters) turned to Twitter to petition Amazon to overturn the rejections. The 30-day rejection reversal deadline came and went with no action. As turkers were trying to find a way to bounce back from the damage caused by this incident, a post showed up on a forum from a worker who claimed to have had their AI Insight rejections overturned.
Throughout the day, we watched more turkers chime in with their own stories. Great news, albeit a little strange. Turkers have always been told that rejections cannot be reversed after thirty days, yet there we were, more than 40 days in. The requester’s AMT approval rating went to 5% and then to 9%. Was it Amazon overturning the rejections, or was it the requester? We wouldn’t find out because the reversals suddenly stopped on the very same day they started. We posted a form that people affected by the mass rejection could fill out in the hope that we could identify a pattern or learn more. Instead, we ended up with more questions.
To date, 35 people have completed the form, totaling 3639 HITs submissions. Of those 35, 4 had rejections overturned for 121 out of 152 completed HITs. The calculations from our sample set put AI Insight’s approval rating increase on AMT at 3.3%, which is fairly close to the 4% increase shown on Mturk after they overturned the rejections. Some turkers chose to contact the requester and/or AMT, and some of them did not. There is no pattern of contact that led directly to the reversals.
Why is the amount of overturned rejections so small? Why were some workers paid and not others? When can we expect to see the rest of the rejections overturned now that we all know it can happen after the 30-day period?
Turkers: Please continue emailing AMT if you were affected by this issue and ask for your rejections to be overturned.
Requesters: Please stand with the turkers who complete your batches of work and your surveys by asking AMT to overturn these rejections.
Amazon: Many workers depend on the platform to supplement their income or even as full-time job. There are now people who have approval ratings too low to qualify for any work because of the rejections. Some people have lost incomes because you won’t step in and implement the turkers’ recommendation of limiting the number of rejections that impact a worker’s approval rating to ten rejections per day.