Elamur Research⌕
☰
home / papers / 2009 / Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk

Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk

Summary

The paper challenges the common view that human evaluation of machine translation is too slow and costly, a view that leaves researchers relying mostly on automatic metrics such as Bleu. Callison-Burch tests whether paid non-expert crowd workers on Amazon's Mechanical Turk can stand in for expert judges. He rebuilt the WMT08 German-English News ranking task, which covers 11 MT systems, inside Mechanical Turk. Each of the 200 expert-judged screens was redone by five Turkers, for a total cost of under $10. The individual rankings were merged using Schulze's preference-voting method, and votes were optionally weighted by each worker's reliability. When combined, the non-expert judgments match expert judgments about as well as experts match each other, and they correlate with expert system rankings better than Bleu does. The paper also shows that crowd workers can handle harder tasks: HTER computation, producing reference translations, and a full reading-comprehension evaluation that workers wrote, answered and graded. The result suggests that routine, cheap human evaluation of MT is practical.

Key points

  • Turkers earned $9.75 in total for close to 1,000 HITs. These reproduced the 200 WMT08 German-English News screens, each showing 5 system outputs plus a reference, with five Turkers per screen.
  • Agreement was counted over pairwise comparisons, where chance agreement is 1/3. WMT08 experts agreed with each other 58% of the time. One non-expert reached 41% agreement with experts, and five unweighted non-experts reached 53%. Five weighted non-experts reached the expert–expert level.
  • Workers varied widely in quality. The Turker who completed the most HITs was among the worst, scoring barely above chance.
  • Two weighting schemes worked about equally well:
    • weighting by agreement with experts on a worker's first 10 judgments;
    • weighting by agreement with the other Turkers across the whole dataset, which needs no gold data.
  • In the reading-comprehension study, Turkers wrote 90 questions for 10 articles, which were filtered to 47 (3–6 per article). 25 Turkers answered questions for each article, each seeing a randomly chosen system's translation, and other Turkers graded the answers.

Interesting ideas

  • I1. Recreate an existing gold-standard task first. Rebuilding a benchmark whose expert labels are already known gives a direct way to check crowd quality before trusting crowd labels on new data. This would suit validating crowdsourced annotation for Ukrainian NLP resources.
  • I2. Weight workers by peer agreement. Scoring each worker by how often they agree with other workers over the whole dataset filters out careless clickers without any gold data. This is useful when expert labels are scarce.
  • I3. Aggregate rankings with Schulze's method. For ranking or preference data, simple majority voting fails. A Condorcet-consistent method like Schulze's yields a full combined order, which fits any ranking or preference-style annotation.
  • I4. Run the whole evaluation through the crowd. Workers can write comprehension questions, answer them from MT output, and grade the answers. This measures whether a translation is actually usable, at very low cost and with little researcher effort.
  • I5. Randomize which system each worker sees. Assigning each worker a single, randomly chosen system translation prevents them from answering the same questions more than once and keeps comparisons fair. This is a simple design rule for any between-system human study.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 103 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{callisonburch2009fast,
  title = {{Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk}},
  author = {Callison-Burch, Chris},
  booktitle = {Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  url = {https://aclanthology.org/D09-1030/}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bogdan Babych and Anthony Hartley. 2004. Extending the Bleu MT evaluation method with frequency weightings. In Proceedings of ACL. find
  2. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In Proceedings of EACL. find
  3. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT08). find our card
  4. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT09), March. find our card
  5. David Chiang, Steve DeNeefe, Yee Seng Chan, and Hwee Tou Ng. 2008. Decomposability of translation metrics for improved evaluation and efficient algorithms. In Proceedings of EMNLP. find
  6. Douglas Jones, Wade Shen, Neil Granoien, Martha Herzog, and Clifford Weinstein. 2005. Measuring translation quality by testing English speakers with a new defense language proficiency test for Arabic. In Proceedings of the 2005 International Conference on Intelligence Analysis. find
  7. LDC. 2005. Linguistic data annotation specification: Assessment of fluency and adequacy in translations. Revision 1.5. find
  8. Evgeny Matusov, Nicola Ueffing, and Hermann Ney. 2006. Computing consensus translation for multiple machine translation systems using enhanced hypothesis alignment. In Proceedings of EACL. find
  9. NIST and LDC. 2007. Post editing guidelines for GALE machine translation evaluation. Guidelines developed by the National Institute of Standards and Technology (NIST), and the Linguistic Data Consortium (LDC). find
  10. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of ACL. find our card
  11. Michael Paul. 2006. Overview of the IWSLT 2006 evaluation campaign. In Proceedings of International Workshop on Spoken Language Translation. find
  12. Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz, and Bonnie Dorr. 2007. Combining outputs from multiple machine translation systems. In Proceedings of HLT/NAACL. find
  13. Markus Schulze. 2003. A new monotonic and clone-independent single-winner election method. Voting Matters, (17), October. find
  14. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of AMTA. find our card
  15. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP. find
  16. Omar F. Zaidan and Chris Callison-Burch. 2009. Feasibility of human-in-the-loop minimum error rate training. In Proceedings of EMNLP. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/burch-2009-fast-cheap-creative-evaluating-translation/