Elamur Research⌕
☰
home / papers / 2011 / Crowdsourcing Translation: Professional Quality from Non-Professionals

Crowdsourcing Translation: Professional Quality from Non-Professionals

Summary

The paper asks whether anonymous, non-expert workers on Amazon's Mechanical Turk can produce translations good enough to stand in for professional ones, at a much lower price. The authors re-translated the Urdu side of the NIST 2009 Urdu-to-English test set, which already has four professional references per sentence, so the crowd output could be compared directly against professionals. They gathered several translations per sentence, had US-based workers post-edit and rank them, and then built a scoring model to pick one output per sentence. The model combines features describing both the sentence and its author: residence country, years of English use, LM perplexity, edit distance to the other candidates, rank judgments, and optionally agreement with professional translations. Feature weights were tuned on a small gold-standard subset. With this selection, the chosen crowd translations achieve BLEU scores that fall inside the range achieved by professionals. Overall spending comes out more than ten times lower than paying professionals for the same work. This makes it practical to build parallel data for low-resource language pairs.

Key points

  • Data: 1,792 Urdu news and web sentences from the NIST 2009 evaluation set. Each has four LDC-supervised professional references, which show how much professional translators themselves vary.
  • Collection:
    • Payment was $0.10 per sentence, about $0.005 per word. For comparison, a cited estimate for professional translation is $0.36 per word.
    • The first pass collected one translation per sentence; the second pass collected three more from distinct workers.
    • Each sentence ended up with 4 raw translations and 10 post-edited versions, so selection chose among 14 candidates.
  • Anti-cheating changes in the second pass: documents were split into HITs of 10 sentences, the Urdu text was shown as images to block pasting into MT systems, and worker location was logged with a JavaScript plugin. The first-pass output was noticeably worse, likely because workers used MT.
  • Results (BLEU):
    • Combining all features gave 39.06.
    • The calibration feature alone, using 10% of the references, gave 37.82.
    • Dropping the editing stage entirely hurt performance.
    • Editing only the top-ranked translations still reached 38.67.
  • Recommended pipeline: collect several translations, collect rankings, then post-edit only the best-ranked candidates.

Interesting ideas

  • I1. Select instead of vote. When the output is a free-form sentence, majority voting does not apply. Scoring the candidates with a weighted feature model and keeping the top one works for any structured crowd output, such as summaries, paraphrases or transcriptions.
  • I2. Model the worker as well as the output. Signals about the person (location, years of English use, how well their work matches gold data) sit alongside sentence-level signals. This helps spot unreliable annotators early, in any annotation project.
  • I3. Use consensus among candidates as a quality signal. A translation's edit distance to the other submissions for the same source is a cheap indicator of quality that needs no reference.
  • I4. Order the stages by cost. Cheap ranking goes first, and the expensive post-editing is applied only to the promising candidates. This cuts cost with little quality loss, a useful pattern for multi-stage annotation budgets.
  • I5. Make cheating hard by design, and evaluate against a professional range. Showing the source as an image is a simple way to stop copy-pasting into MT. Scoring the crowd against the spread among several professional references gives a realistic target instead of a single "perfect" answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

90 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zaidan2011crowdsourcing,
  title = {{Crowdsourcing Translation: Professional Quality from Non-Professionals}},
  author = {Zaidan, Omar F. and Callison-Burch, Chris},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1122/}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yaser Al-Onaizan, Ulrich Germann, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Daniel Marcu, and Kenji Yamada. 2002. Translation with scarce bilingual resources. Machine Translation, 17(1), March. find
  2. Vamshi Ambati and Stephan Vogel. 2010. Can crowds build parallel corpora for machine translation systems? In Proceedings of the NAACL HLT Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk, pages 62–65. find
  3. Ben Bederson and Philip Resnik. 2010. Workshop on crowdsourcing and translation. http://www.cs.umd.edu/hcil/monotrans/workshop/. ↗
  4. Michael Bloodgood and Chris Callison-Burch. 2010. Using Mechanical Turk to build machine translation evaluation sets. In Proceedings of the NAACL HLT Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk, pages 208–211. find
  5. Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk, pages 1–12. find
  6. Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk. In Proceedings of EMNLP, pages 286– 295. find our card
  7. A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20–28. find
  8. AMTA 2010 workshop on collaborative translation: technology, crowdsourcing, and the translator perspective. http://bit.ly/gPnqR2. ↗
  9. Pascale Fung and Lo Yuen Yee. 1998. An ir approach for translating new words from nonparallel, comparable texts. In Proceedings of ACL/CoLing. find our card
  10. Ulrich Germann. 2001. Building a statistical machine translation system from scratch: How much bang for the buck can we expect? In ACL 2001 Workshop on Data-Driven Machine Translation, Toulouse, France. find
  11. Aria Haghighi, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proceedings of ACL/HLT. find our card
  12. Panos Ipeirotis. 2010. New demographics of Mechanical Turk. http://behind-the-enemy-lines.blogspot.com/2010/03/ new-demographics-of-mechanical-turk. html. ↗
  13. Ann Irvine and Alexandre Klementiev. 2010. Using Mechanical Turk to annotate lexicons for less commonly used languages. In Proceedings of the NAACL HLT Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk, pages 108–113. find
  14. Zhifei Li, Chris Callison-Burch, Chris Dyer, Juri Ganitkevitch, Ann Irvine, Sanjeev Khudanpur, Lane Schwartz, Wren Thornton, Ziyuan Wang, Jonathan Weese, and Omar Zaidan. 2010. Joshua 2.0: A toolkit for parsing-based machine translation with syntax, semirings, discriminative training and other goodi… find
  15. Dragos Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting comparable corpora. Computational Linguistics, 31(4):477– 504, December. find
  16. Sonja Niessen and Hermann Ney. 2004. Statistical machine translation with scarce resources using morpho-syntatic analysis. Computational Linguistics, 30(2):181–204. find
  17. Doug Oard, David Doermann, Bonnie Dorr, Daqing He, Phillip Resnik, William Byrne, Sanjeeve Khudanpur, David Yarowsky, Anton Leuski, Philipp Koehn, and Kevin Knight. 2003. Desperately seeking Cebuano. In Proceedings of HLT/NAACL. find
  18. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of ACL, pages 160–167. find
  19. Kishore Papineni, Salim Poukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL, pages 311–318. find our card
  20. Katharina Probst, Lori Levin, Erik Peterson, Alon Lavie, and Jamie Carbonell. 2002. MT for minority languages using elicitation-based learning of syntactic transfer rules. Machine Translation, 17(4). find
  21. Reinhard Rapp. 1995. Identifying word translations in non-parallel texts. In Proceedings of ACL. find our card
  22. Philip Resnik and Noah Smith. 2003. The web as a parallel corpus. Computational Linguistics, 29(3):349– 380, September. find our card
  23. Philip Resnik, Olivia Buzek, Chang Hu, Yakov Kronrod, Alex Quinn, and Benjamin Bederson. 2010. Improving translation via targeted paraphrasing. In Proceedings of EMNLP, pages 127–137. find
  24. Charles Schafer and David Yarowsky. 2002. Inducing translation lexicons via diverse similarity measures and bridge languages. In Conference on Natural Language Learning-2002, pages 146–152. find
  25. Jason R. Smith, Chris Quirk, and Kristina Toutanova. 2010. Extracting parallel sentences from comparable corpora using document level alignment. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 403–411, L… find our card
  26. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas (AMTA). find our card
  27. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast – but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP, pages 254–263. find our card
  28. Jakob Uszkoreit, Jay M. Ponte, Ashok C. Popat, and Moshe Dubiner. 2010. Large scale parallel document mining for machine translation. In Proc. of the International Conference on Computational Linguistics (COLING). find
  29. Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of NIPS, pages 2035–2043. find
  30. Omar F. Zaidan. 2009. Z-MERT: A fully configurable open source tool for minimum error rate training of machine translation systems. The Prague Bulletin of Mathematical Linguistics, 91:79–88. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaidan-2011-crowdsourcing-translation-professional-quality-non/