Elamur Research⌕
☰
home / papers / 2008 / Constructing Corpora for the Development and Evaluation of Paraphrase Systems

Constructing Corpora for the Development and Evaluation of Paraphrase Systems

Summary

Paraphrase acquisition research has had no shared benchmark. Systems were usually judged by people rating sample outputs, which is slow, hard to reproduce and says almost nothing about recall. Cohn, Callison-Burch and Lapata address this gap by building a monolingual parallel corpus in which paraphrases are made explicit through manual word and phrase alignments between sentences with equivalent meaning.

To speed up annotation, Giza++ pre-aligned the sentence pairs and trained linguists then corrected the links, marking each as sure or possible. Because Kappa is designed for nominal categories, the authors propose a new chance-corrected agreement measure suited to structured outputs such as alignments. Overlap metrics show that annotators agree reliably.

The corpus can serve as a gold standard for automatically computing precision, recall and F1 of paraphrase extractors, and as training material for syntax-aware paraphrase models. A pilot comparison suggested that alignment-based extraction outperforms a co-training approach.

Key points

  • The corpus contains 900 English sentence pairs, 300 from each of three sources: the Multiple-Translation Chinese (MTC) corpus, two translations of Verne's Twenty Thousand Leagues Under the Sea (Leagues), and the MSR paraphrase corpus. Of these, 300 pairs were annotated by two coders.
  • Automatic pre-alignment used Giza++ trained on all translator pairings of the 993-sentence MTC part 1, supplemented with a word-identity lexicon. The two alignment directions were symmetrized by intersection.
  • Annotators aligned spans that could substitute for each other in context. They labeled links as sure (S) or possible (P), with S ⊆ P, and reserved block alignments for idioms and non-compositional phrases.
  • Inter-annotator agreement on sure phrase pairs reached about 67% precision and 66% recall, which the authors treat as an upper bound for systems.
  • Against the gold standard, Giza++ scored 55% precision and 49% recall, while Barzilay and McKeown's co-training extractor scored 30% and 16%. A manual check of 48 supposed errors confirmed 63% as genuinely wrong.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

33 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 27 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.08-003-r1-07-044

@article{cohn2008constructing,
  title = {{Constructing Corpora for the Development and Evaluation of Paraphrase Systems}},
  author = {Cohn, Trevor and Callison-Burch, Chris and Lapata, Mirella},
  journal = {Computational Linguistics},
  year = {2008},
  doi = {10.1162/coli.08-003-r1-07-044},
  url = {https://doi.org/10.1162/coli.08-003-r1-07-044}
}

References

37 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aho, A. V. and J. D. Ullman. 1969. Syntax directed translations and the pushdown assembler. Journal of Compute System Sciences, 3(1):37–56. ↗
  2. Artstein, Ron and Massimo Poesio. 2008. Inter-coder agreement for Computational Linguistics. Computational Linguistics. ↗
  3. Ayan, Necip Fazil and Bonnie J. Dorr. 2006. Going beyond AER: An extensive analysis of word alignments and their impact on MT. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 9–16, Sydney. find
  4. Bannard, Colin and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 597–604, Ann Arbor, MI. find our card
  5. Bartko, John J. and William T. Carpenter. 1976. On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163(5):307–317. ↗
  6. Barzilay, Regina. 2003. Information Fusion for Multi-Document Summarization: Paraphrasing and Generation. Ph.D. thesis, Columbia University, New York, NY. find
  7. Barzilay, Regina and Noemie Elhadad. 2003. Sentence alignment for monolingual comparable corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 25–32, Sapporo. find
  8. Barzilay, Regina and Lillian Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In Proceedings of the Human Language Technology Conference and the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 16–23, Edmonton. ↗ our card
  9. Barzilay, Regina and Kathy McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 50–57, Toulouse. find our card
  10. Bikel, Daniel. 2002. Design of a multi-lingual, parallel-processing statistical parsing engine. In Proceedings of the Human Language Technology Conference, pages 24–27, San Diego, CA. find
  11. Brown, Peter F., Stephen A. Della-Pietra, Vincent J. Della-Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  12. Callison-Burch, Chris. 2007. Paraphrasing and Translation. Ph.D. thesis, University of Edinburgh, Edinburgh, Scotland. find
  13. Callison-Burch, Chris, Philipp Koehn, and Miles Osborne. 2006. Improved statistical machine translation using paraphrases. In Proceedings of the Human Language Technology Conference and Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 17–24, New York, NY. find
  14. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37–46. find
  15. Cohn, Trevor and Mirella Lapata. 2007. Large margin synchronous generation and its application to sentence compression. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and on Computational Natural Language Learning, pages 73–82, Prague. find
  16. Cowan, Brooke, Ivona Kuc̆erová, and Michael Collins. 2006. A discriminative model for tree-to-tree translation. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 232–241, Sydney. find
  17. Daumé III, Hal and Daniel Marcu. 2004. A phrase-based HMM approach to document/abstract alignment. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 119–126, Barcelona. find
  18. Di Eugenio, Barbara and Michael Glass. 2004. The kappa statistic: A second look. Computational Linguistics, 30(1):95–101. ↗
  19. Dolan, William, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th International Conference on Computational Linguistics, pages 350–356, Geneva. find our card
  20. Duboue, Pablo and Jennifer Chu-Carroll. 2006. Answering the question you wish they had asked: The impact of paraphrasing for question answering. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 33–36, New York, NY. find
  21. Fleiss, Joseph L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. ↗
  22. Fraser, Alexander and Daniel Marcu. 2007. Measuring word alignment quality for statistical machine translation. Computational Linguistics, 33(3):293–303. ↗
  23. Koehn, Philipp, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Human Language Technology Conference and Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 48–54, Edmonton. ↗
  24. Kupper, Lawrence L. and Kerry B. Hafner. 1989. On assessing interrater agreement for multiple attribute responses. Biometrics, 45(3):957–967. ↗
  25. Lin, Dekang and Patrick Pantel. 2001. Discovery of inference rules for question answering. Natural Language Engineering, 7(4):342–360. ↗
  26. Martin, Joel, Rada Mihalcea, and Ted Pedersen. 2005. Word alignment for languages with scarce resources. In Proceedings of the ACL Workshop on Building and Using Parallel Texts, pages 67–74, Ann Arbor, MI. find
  27. Melamed, I. Dan. 1998. Manual annotation of translational equivalence: The Blinker project. IRCS Technical Report #98-07, University of Pennsylvania, Philadelphia, PA. find
  28. Mihalcea, Rada and Ted Pedersen. 2003. An evaluation exercise for word alignment. In Proceedings of the HLT-NAACL Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 1–6, Edmonton. find our card
  29. Och, Franz Josef and Hermann Ney. 2000a. A comparison of alignment models for statistical machine translation. In Proceedings of the 18th International Conference on Computational Linguistics, pages 1086–1090, Saarbrücken. find our card
  30. Och, Franz Josef and Hermann Ney. 2000b. Improved statistical alignment models. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 440–447, Hong Kong. find our card
  31. Och, Franz Josef and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–52. ↗
  32. Och, Franz Josef, Christoph Tillmann, and Hermann Ney. 1999. Improved alignment models for statistical machine translation. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 20–28, College Park, MD. find our card
  33. Pang, Bo, Kevin Knight, and Daniel Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In Proceedings of the Human Language Technology Conference and the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 181–188, Edmonton. find our card
  34. Quirk, Chris, Chris Brockett, and William Dolan. 2004. Monolingual machine translation for paraphrase generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 142–149, Barcelona. find our card
  35. Scott, William A. 1955. Reliability of content analysis: The case of nominal scale. Public Opinion Quarterly, 19:127–141. find
  36. Siegel, Sidney and N. John Castellan. 1988. Non Parametric Statistics for the Behavioral Sciences. McGraw-Hill, New York. find
  37. Zhou, Liang, Chin-Yew Lin, Dragos Stefan Munteanu, and Eduard Hovy. 2006. Paraeval: Using paraphrases to evaluate summaries automatically. In Proceedings of the Human Language Technology Conference, pages 447–454, New York, NY. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cohn-2008-constructing-corpora-development-evaluation-paraphrase/