Elamur Research⌕
☰
home / papers / 2011 / Collecting Highly Parallel Data for Paraphrase Evaluation

Collecting Highly Parallel Data for Paraphrase Evaluation

Summary

Paraphrase research has trailed machine translation largely because it lacks shared corpora and an automatic, BLEU-like way to score system output. Chen and Dolan address both gaps. For data, they built a crowdsourcing task on Amazon's Mechanical Turk. Workers watched a short YouTube clip, usually under ten seconds, and wrote one sentence about its main action. Each worker describes the scene without seeing anyone else's wording, so every clip collects many independent sentences with the same meaning. Over two months this produced about 85K English descriptions of roughly 2K videos, released as the Microsoft Research Video Description Corpus, along with descriptions in other languages. For evaluation, BLEU computed against the many references measures meaning preservation. A new metric, PINC (Paraphrase In N-gram Changes), rewards candidates whose n-grams depart from those of the source sentence. Both measures track human ratings well, and BLEU's agreement improves as more references are added. Together, the corpus and metrics give paraphrasing a reproducible, automated train-and-test loop like the one used in SMT.

Key points

  • Corpus: about 85K English sentences for about 2K video segments, collected over two months and released as the Microsoft Research Video Description Corpus. The authors call it one of the largest public paraphrase resources.
  • Task speed and quality control: a task took about 80 seconds on average, and experienced workers needed only 20–25 seconds. Quality was managed with two pay tiers. Tier-1 paid 1 cent and was open to everyone. Tier-2 paid 5 cents and was limited to manually qualified workers.
  • References: BLEU's correlation with human judgments grows with the number of reference descriptions. Adding the source sentence as a reference helps when judging meaning alone. It hurts when judging overall paraphrase quality, because it favors minimal edits.
  • Combining BLEU and PINC: the two scores were merged through a logistic function rather than a linear one. The experiment picked the function with an oracle. In practice, a small set of human ratings would be needed to tune it.
  • Video vs. direct paraphrasing: for comparison, 1,000 sentences each received two direct paraphrases.
    • Direct paraphrases scored a mean PINC of 70.08, against 78.75 for independent video descriptions.
    • In a survey, 92% of workers found video description more enjoyable and 75% found it easier.
    • Half preferred the video task, while 16% preferred paraphrasing.

Interesting ideas

  • I1. Elicit through a non-text stimulus: showing a video instead of a sentence keeps annotators from copying the original's words and word order. This trick could help build paraphrase or NLI-style corpora for languages like Ukrainian, where existing parallel text is scarce.
  • I2. Let annotators use any language: this yields aligned descriptions across languages without needing bilingual workers. It is a cheap route to parallel MT data for lower-resource language pairs.
  • I3. Score meaning and novelty separately: adequacy (BLEU against references) and dissimilarity (PINC against the source) are measured independently, then joined with a logistic curve. Because meaning preservation acts almost like a pass/fail gate, this design fits any generation task where correctness must hold before diversity matters.
  • I4. Match references to the goal: whether the source sentence counts as a reference should depend on what is being measured. Including it suits pure equivalence checks. Excluding it suits overall quality ratings, since otherwise the metric rewards timid outputs.
  • I5. Use tiers for quality and filtering: a cheap open tier plus a better-paid tier for vetted workers keeps quality up at low cost. The authors also suggest using the trusted Tier-2 output as positive examples to train a filter that recovers usable sentences from the noisier data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

196 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 183 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chen2011collecting,
  title = {{Collecting Highly Parallel Data for Paraphrase Evaluation}},
  author = {Chen, David and Dolan, William},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1020/}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vamshi Ambati and Stephan Vogel. 2010. Can crowds build parallel corpora for machine translation systems? In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05). find our card
  3. Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In Proceedings of Human Language Technology Conference / North American Association for Computational Linguistics Annual Meeting (HLT-NAACL-2003). ↗ our card
  4. Regina Barzilay and Kathleen McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (ACL-2001). find our card
  5. Michael Bloodgood and Chris Callison-Burch. 2010a. Bucking the trend: Large-scale cost-focused active learning for statistical machine translation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL-2010). find
  6. Michael Bloodgood and Chris Callison-Burch. 2010b. Using Mechanical Turk to build machine translation evaluation sets. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  7. Olivia Buzek, Philip Resnik, and Benjamin B. Bederson. 2010. Error driven paraphrase annotation using Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  8. Chris Callison-Burch, Philipp Koehn, and Miles Osborne. 2006. Improved statistical machine translation using paraphrases. In Proceedings of Human Language Technology Conference / North American Chapter of the Association for Computational Linguistics Annual Meeting (HLT-NAACL-06). find
  9. Chris Callison-Burch, Trevor Cohn, and Mirella Lapata. 2008. Parametric: An automatic evaluation metric for paraphrasing. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING-2008). find
  10. Chris Callison-Burch. 2008. Syntactic constraints on paraphrases extracted from parallel corpora. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP-2008). find our card
  11. Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP-2009). find our card
  12. Wallace L. Chafe. 1997. The Pear Stories: Cognitive, Cultural and Linguistic Aspects of Narrative Production. Ablex, Norwood, NJ. find
  13. Trevor Cohn, Chris Callison-Burch, and Mirella Lapata. 2008. Constructing corpora for the development and evaluation of paraphrase systems. Computational Linguistics, 34:597–614, December. ↗ our card
  14. Michael Denkowski and Alon Lavie. 2010. Exploring normalization techniques for human judgments of machine translation adequacy collected using Amazon Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  15. Michael Denkowski, Hassan Al-Haj, and Alon Lavie. 2010. Turker-assisted paraphrasing for English-Arabic machine translation. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  16. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th International Conference on Computational Linguistics (COLING-2004). find our card
  17. Li Fei-Fei, Asha Iyer, Christof Koch, and Pietro Perona. 2007. What do we perceive in a glance of a real-world scene? Journal of Vision, 7(1):1–29. ↗
  18. Ali Ibrahim, Boris Katz, and Jimmy Lin. 2003. Extracting structural paraphrases from aligned monolingual corpora. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL-03). ↗
  19. Ann Irvine and Alexandre Klementiev. 2010. Using Mechanical Turk to annotate lexicons for less commonly used languages. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  20. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL-07). find
  21. Stanley Kok and Chris Brockett. 2010. Hitting the right paraphrases in good time. In Proceedings of Human Language Technologies: The Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT-2010). find
  22. Dekang Lin and Patrick Pantel. 2001. DIRT-discovery of inference rules from text. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2001). find
  23. Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. 2010. PEM: A paraphrase evaluation metric exploiting parallel texts. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP-2010). find
  24. Xiaojuan Ma and Perry R. Cook. 2009. How well do visual verbs work in daily communication for young and old adults. In Proceedings of ACM CHI 2009 Conference on Human Factors in Computing Systems. find
  25. Bo Pang, Kevin Knight, and Daniel Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In Proceedings of Human Language Technology Conference / North American Association for Computational Linguistics Annual Meeting (HLT-NAACL-2003). find our card
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-2002), pages 311–318, Philadelphia, PA, July. find our card
  27. Chris Quirk, Chris Brockett, and William Dolan. 2004. Monolingual machine translation for paraphrase generation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP-2004). find our card
  28. Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. 2010. Collecting image annotations using Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  29. Yusuke Shinyama, Satoshi Sekine, Kiyoshi Sudo, and Ralph Grishman. 2002. Automatic paraphrase acquisition from news articles. In Proceedings of Human Language Technology Conference. find
  30. Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of ACM CHI 2004 Conference on Human Factors in Computing Systems. find
  31. Omar F. Zaidan and Chris Callison-Burch. 2009. Feasibility of human-in-the-loop minimum error rate training. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP-2009). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/chen-2011-collecting-highly-parallel-data-paraphrase/