📄 Collecting Image Annotations Using Amazon's Mechanical Turk

Summary

The authors wanted a corpus that pairs photographs with plain full-sentence descriptions of what they show, such as objects, their attributes, relations and actions. Such data would support work that combines computer vision and NLP, and neither standard vision datasets nor Flickr tags provide it. They gathered captions on Amazon's Mechanical Turk (MTurk) and compared ways of controlling the quality of free-text output, where no answer key exists.

Their first round had no restriction on who could write. It gave cheap but noisy captions, with many non-native errors, fragments and jokes. They then tried filtering this output with a second crowd task in which workers judged grammar and spelling. This worked badly: the judges were too strict and disagreed often with expert annotators. A second approach worked much better. Writers had to live in the US and pass a short English qualification test, and this produced far cleaner captions at low cost.

With this setup they built two corpora: one from PASCAL VOC2008 and one from Flickr. Together they hold over 40,000 captions for roughly 9,000 images. The work offers practical guidance for collecting crowdsourced linguistic data.

Key points

• The first corpus uses 1000 images from the PASCAL VOC2008 train/dev set, 50 chosen at random from each of the 20 object categories, with five captions per image.
• In the unrestricted first round, each task asked for 10 sentences and paid $0.10, so 5000 captions cost $50 plus Amazon's fee. Workers earned about $1.30/hour, 2.5% of the answers were empty, and expert raters fully accepted only about 60% of the sentences.
• In the crowd-based grammar/spelling filter, each HIT covered 50 captions, paid $0.08 and had three judges. The Turker majority matched the expert majority on only 65.1% of sentences. Among workers who answered every control item correctly, agreement fell further, to 56.7%.
• With the US-residence requirement and the qualification test, 82.0% of the captions received unanimous expert approval, compared with 60.2% for unqualified writers.
• An extended Flickr set of about 8000 images was drawn from groups such as dogs, action photography and wild-child. Annotating it took 284 hours and cost $812.

Interesting ideas

• I1. Screen workers before they write. A short qualification test gave a bigger quality gain than any cleanup step afterwards. Any crowdsourced text-production task, such as Ukrainian paraphrases or translations, could add a cheap language-proficiency gate at the start.
• I2. Spelling correction guided by the corpus. The authors first applied a hand-made correction list, then passed unknown words to aspell. Among aspell's suggestions, they picked the one that appeared most often in other captions of the same image, then the same topic, then the whole dataset. Using local context to rank candidates like this could help normalize noisy user text in other domains.
• I3. Check your control items against expert judgment. Workers who scored perfectly on the control items agreed less with experts, not more, so high control scores did not mean good judgment. Before trusting gold questions as a quality signal, confirm that they actually predict agreement with experts.
• I4. Keep the variety in multiple references. The authors avoided iterative crowd refinement, such as TurKit-style editing, partly because it could make the captions for one image too similar. When collecting several references per item, a design that keeps varied wording and viewpoints can be worth a few leftover errors.
• I5. Build the corpus on a standard benchmark. They captioned images from a dataset the vision community already uses, so the new annotations can be combined with existing labels. Adding a new annotation layer to a widely used resource makes it easier to adopt.