Elamur Research⌕
☰
home / papers / 2008 / Syntactic Constraints on Paraphrases Extracted from Parallel Corpora

Syntactic Constraints on Paraphrases Extracted from Parallel Corpora

Summary

This work addresses a weakness of the pivot-based paraphrasing approach of Bannard and Callison-Burch (2005). In that approach, two English phrases count as paraphrases when they share a foreign translation in a word-aligned parallel corpus. Because phrase extraction heuristics allow unaligned words at phrase edges, many extracted "paraphrases" are just longer or shorter versions of the original phrase. Swapping these into a sentence usually breaks grammar.

The author parses the English side of the parallel corpora and changes the extraction algorithm (called SBP) so that each bilingual phrase pair also carries a syntactic label. Paraphrase probabilities are then computed only within the same syntactic category. To keep coverage for word sequences that are not constituents, the method adds complex syntactic labels. Constraints are applied during extraction and, optionally, when substituting paraphrases into parsed test sentences.

A manual evaluation on out-of-domain news commentary sentences shows about a 19% absolute gain in paraphrases judged correct in both meaning and grammar. This makes the resource more usable for tasks such as generation, where output must be grammatical.

Key points

  • The baseline, trained on Europarl, produced 3.7 million paraphrases for 400,000 of 900,000 test phrases. 34% of these were sub- or super-strings of the source phrase, and such strings were the top-ranked paraphrase 73% of the time.
  • Under the strict criterion (meaning and grammar both correct), the baseline scored .30. The syntactically constrained variants scored up to .55, and grammaticality rose from .35 to as much as .68.
  • Adding a language model made output 7–9% more grammatical. However, it cut correct meaning by 12% for the baseline, against only 1–2% for the constrained versions.
  • Substitution-time constraints reduced coverage to 876 of 1194 sentences (73%). Extraction-only constraints kept wider coverage with nearly the same meaning scores.
  • The manual evaluation covered eight conditions:
    • Exact agreement on the 5-point scale gave Kappa 0.33.
    • Thresholded (correct/incorrect) agreement gave Kappa 0.61.
  • The author released the following at cs.jhu.edu~ccb/:
    • the paraphrase sets (3.7M baseline, 3.5M constrained);
    • the code;
    • 10 word-aligned parallel corpora with English parses;
    • the human judgments.

Interesting ideas

  • I1. Conditioning on a syntactic label. Making the pivot probabilities depend on the label splits the paraphrase space by category. This cheaply removes cross-category noise (nouns paraphrasing adjectives, for instance) and could carry over to any pivot- or alignment-based lexicon induction.
  • I2. Parsing only one side. Only the English side of the parallel data needs a parser, so the foreign side requires no syntactic tools. This is useful when only one language in a pair has a good parser.
  • I3. Labels for non-constituents. Building composite syntactic labels for arbitrary word spans keeps the coverage of purely phrase-based methods while still adding syntax. This compromise is relevant wherever strict constituency would throw away too much data.
  • I4. Diagnosing errors by counting substrings. Measuring how often outputs are sub- or super-strings of the input is a quick, automatic check for boundary artifacts caused by alignment. It could serve as a sanity metric for other extraction pipelines.
  • I5. Reporting agreement at two granularities. Giving agreement both on the full scale and on the binary thresholds actually used for scoring is a more honest picture of annotation reliability. It suits any graded human evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

92 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{callisonburch2008syntactic,
  title = {{Syntactic Constraints on Paraphrases Extracted from Parallel Corpora}},
  author = {Callison-Burch, Chris},
  booktitle = {Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  url = {https://aclanthology.org/D08-1021/}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of ACL. find our card
  2. Regina Barzilay and Kathleen McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of ACL. find our card
  3. Dan Bikel. 2002. Design of a multi-lingual, parallel-processing statistical parsing engine. In Proceedings of HLT. find
  4. Chris Callison-Burch, Philipp Koehn, and Miles Osborne. 2006. Improved statistical machine translation using paraphrases. In Proceedings of HLT/NAACL. find
  5. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation. find our card
  6. Jean Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254. ↗ our card
  7. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of ACL. find our card
  8. Heidi J. Fox. 2002. Phrasal cohesion and statistical machine translation. In Proceedings of EMNLP. find
  9. Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In Proceedings of HLT/NAACL. ↗ our card
  10. Bryant Huang and Kevin Knight. 2006. Relabeling syntax trees to improve syntax-based machine translation quality. In Proceedings of HLT/NAACL. find
  11. Ali Ibrahim, Boris Katz, and Jimmy Lin. 2003. Extracting structural paraphrases from aligned monolingual corpora. In Proceedings of the Second International Workshop on Paraphrasing (ACL 2003). ↗
  12. David Kauchak and Regina Barzilay. 2006. Paraphrasing for automatic evaluation. In Proceedings of EMNLP. find our card
  13. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of HLT/NAACL. ↗
  14. Philipp Koehn. 2005. A parallel corpus for statistical machine translation. In Proceedings of MT-Summit, Phuket, Thailand. find
  15. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174. find
  16. Dekang Lin and Colin Cherry. 2002. Word alignment with cohesion constraint. In Proceedings of HLT/NAACL. find
  17. Dekang Lin and Patrick Pantel. 2001. Discovery of inference rules from text. Natural Language Engineering, 7(3):343–360. find
  18. Nitin Madnani, Necip Fazil Ayan, Philip Resnik, and Bonnie Dorr. 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceedings of the ACL Workshop on Statistical Machine Translation. find
  19. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. ↗
  20. Franz Josef Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics, 30(4):417–449. ↗
  21. Karolina Owczarzak, Declan Groves, Josef Van Genabith, and Andy Way. 2006. Contextual bitext-derived paraphrases in automatic MT evaluation. In Proceedings of the SMT Workshop at HLT-NAACL. find
  22. Bo Pang, Kevin Knight, and Daniel Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In Proceedings of HLT/NAACL. find our card
  23. Chris Quirk, Arul Menezes, and Colin Cherry. 2005. Dependency treelet translation: Syntactically informed phrasal smt. In Proceedings of ACL. find our card
  24. Stefan Riezler, Alexander Vasserman, Ioannis Tsochantaridis, Vibhu Mittal, and Yi Liu. 2007. Statistical machine translation for query expansion in answer retrieval. In Proceedings of ACL. find
  25. Mark Steedman. 1999. Alternative quantier scope in ccg. In Proceedings of ACL. find
  26. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, Denver, Colorado, September. find
  27. Idan Szpektor, Eyal Shnarch, and Ido Dagan. 2007. Instance-based evaluation of entailment rule acquisition. In Proceedings of ACL. find
  28. Dekai Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational Linguistics, 23(3). find our card
  29. Shiqi Zhao, Cheng Niu, Ming Zhou, Ting Liu, and Sheng Li. 2008a. Combining multiple resources to improve SMT-based paraphrasing model. In Proceedings of ACL/HLT. find
  30. Shiqi Zhao, Haifeng Wang, Ting Liu, and Sheng Li. 2008b. Pivot approach for extracting paraphrase patterns from bilingual corpora. In Proceedings of ACL/HLT. find
  31. Liang Zhou, Chin-Yew Lin, and Eduard Hovy. 2006. Re-evaluating machine translation results with paraphrase support. In Proceedings of EMNLP. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/burch-2008-syntactic-constraints-paraphrases-extracted-parallel/