Fair payment

Fair payment

Crowdsourcing workers are a labor force. While we cannot speak for all crowd workers, many depend on income from crowdsourcing as a supplementary or primary income. Crowdsourcing workers are legally considered contractors and therefore are not protected by any minimum wage laws. When requesters pay a fair wage and treat workers like people, both sides receive positive results.

Posting to Turk is not just like collecting a survey

Posting an academic survey on Amazon Mechanical Turk is different than traditional forms of survey collection. Workers presume that they will be paid a fair wage and do not respect requesters who offer an extremely low rate, though what that means for different crowdworkers (India vs. US) may be different. Unfortunately some requesters may interpret this to mean they are allowed to pay extremely low rates and consider this to be “the norm of the market”. This is not acceptable to the majority of the established worker community. These requesters survive because of the constant influx of new workers who have not established themselves in the workplace and the large population of international workers who view these lower payments as acceptable.

What is ethical pay for Turkers in studies?

Consider the estimated duration and difficulty of your task when deciding about payment.

There isn’t one answer appropriate for all situations, but here are some points to consider when trying to decide what a fair and ethical rate of pay on your HITs would be for US-based workers:

  • Honest US-based turkers will generally be paying taxes on their MTurk earnings as self-employment.

  • There is a lot of unpaid overhead time involved in turking, including: looking for the next suitable HITs to do, taking uncompensated qualification tests to hopefully qualify for certain HITs, checking reviews for unfamiliar requesters to decide whether to work on their HITs, writing reviews of requesters, communicating with other workers on forums, dealing with some of Amazon’s security measures such as periodic Captchas and forced logouts that can interrupt workflow, dealing with occasional malfunctions of the worker’s ISP/browser/computer, communicating with requesters (or in about half the cases, apparently futilely sending messages into a void) about questions/problems/suggestions, keeping track of the work they’ve done and the payments and bonuses they have or haven’t received so far, checking their records of work they’ve done to see if it’s safe to take a survey that threatens rejections if you take it more than once, and more. All break time (even going to the restroom) is also uncompensated.

  • The more specialized knowledge/skills/characteristics, and/or the more stringent the qualifications (such as higher number of HITs approved, higher approval rate, scores on requesters’ custom quals, and/or Masters) that your HITs will expect or require, the higher the pay rate for it generally should be if you want to continue to be fair (a fair minimum pay rate logically would only be considered as fair for HITs with minimum requirements, just like more-qualified/experienced workers in the traditional workforce generally expect to receive higher pay than less-qualified/experienced workers).

  • Although self-employment work is not legally obligated to comply with minimum wage laws, they are commonly used as a benchmark in evaluating what pay would be fair and ethical. Points to consider regarding the minimum wage include:

  • The per-subject costs for other non-MTurk ways researchers can recruit survey participants reportedly tend to be much higher than you would be paying MTurk workers even at much more fair and ethical pay rates than is currently prevalent. Researchers who say they ‘don’t have the funding’ to pay better rates on MTurk should consider that the alternatives are often to pay even more for a participant pool that may be less diverse and in some cases less attentive than MTurk workers. Even in the case of unfunded student projects, please try to consider that the total difference between fair and unfair pay will usually be less than you might think, particularly compared to the other costs you’ve committed to in pursuing your education; even just the textbooks.

  • The availability of non-US workers on MTurk has apparently been gradually decreasing since Amazon stopped accepting registrations of new international worker accounts in late 2012. And with the exception of India (the only country besides the US that has ever been able to receive direct monetary payment), there were never a large percentage of workers from any other particular country. So even if you don’t specifically require workers to be in the US to accept your HITs, a large and re-growing proportion of them will be in the US, unless you specifically exclude US workers from your HITs.

Learning more about the demographics and other statistics of the turker workforce can help requesters make more informed decisions about how to structure and compensate their HITs. Many turkers are indeed casual participants performing a small number of HITs, but studies indicate the vast majority, perhaps 80%, of the HITs completed on MTurk are performed by turkers who are in the top 10% or so of productivity among active turkers, each completing hundreds or thousands of HITs per week; and many of those put forth that much effort because the money is very important to them to make ends meet, whether they have other significant sources of income or not.

Several dozen academic papers and blog posts, covering much of the above information and other related topics (many demographics, as well as work consistency, work distribution, etc), are listed with links and quotes of the relevant portions, at ‘Demographics of Mechanical Turk’ by turker ‘clickhapper’ at mTurk Grind [6].

What about my research sample?

No matter what guideline is used, it has always been up to the individual worker to decide how much their time is worth, but when large groups of workers are excluded from research because of poor payment, the results of the research cannot be considered a valid sampling of a population.

What if I want Turkers from different countries?

Since MTurk is a worldwide website, what may be an acceptable wage in Asia is not acceptable in many North American and European countries. If a requester would like to use the entire range of worldwide users, they should pay the same wage in India as they do in Indiana. If a requester would like to use only workers from emerging economies, it would be acceptable to break from the Western payment norms and price work according to fair wages within those countries.

What if my task takes longer than I thought, so the wages sink below what is fair?

If an amount of pay you expected to be a fair rate turns out not to be because you accidentally underestimated how long your survey would take for reasonably-efficient workers to complete, consider adjusting for this situation by sending bonuses to the workers to make up the difference (could base the bonus amount on the mean or median time to complete, in case a few workers are unusually inefficient).