Paraphrasing with Bilingual Parallel Corpora

What it is about

The paper shows that paraphrases can be extracted from a bilingual parallel corpus, which is far more common than monolingual parallel text. Two English phrases that align to the same foreign phrase are taken as paraphrases. The paraphrase probability is the sum over foreign phrases of the probability of the English paraphrase given the foreign phrase, times the probability of that foreign phrase given the first English phrase, both taken from phrase alignments of statistical MT. The paper also controls the word sense by requiring the same foreign phrase in a test sentence, and adds language-model re-ranking. The evaluation substitutes the top paraphrase into 289 sentences and judges correctness. With manual alignments, 74.9% of the substituted sentences are judged correct and 84.7% keep the meaning. With automatic alignments the accuracy is 48.9%, rising to 57% with word-sense control and 61.9% with language-model re-ranking; several European corpora together reach 55%. The gap between manual and automatic alignment is the main source of error.

Ideas

I1. A count form of the translator's lexicon: the source chunk (Hebrew) plays the role of the foreign phrase, the English chunks aligned to it are the candidate paraphrases, and the probability is a product of two counts. No neural model is involved, and each merge carries its counts.
I2. Context control for merges: accept a merge in a verse only if both words were aligned to the same source chunk in that verse. This is the paper's word-sense trick applied to the reader; measure how many merges survive it.
I3. Report accuracy split by the source of the alignment, as the paper does. For Mova the Strong's-number alignment (open problem 7) is the gold side and IBM Model 1 is the automatic side; the size of the gap says how much the alignment work is worth.

For Mova

This paper is the origin of the pivot idea that the translator's lexicon (open problem 2) needs, and its main lesson is that alignment noise, not the pivot, limits accuracy. The tools are tools/translate/ibm.rs and hebrew.rs. Summary only, as the licence is non-commercial.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P05-1074.pdf

Summary

The paper addresses the shortage of data for learning paraphrases. Earlier methods relied on monolingual parallel corpora, such as several English translations of the same novel. These are rare and cover few genres. Bannard and Callison-Burch instead use bilingual parallel text, which is far more plentiful. They borrow phrase alignment from phrase-based statistical machine translation and treat a foreign phrase as a pivot. Two English phrases that both align to the same German phrase are taken as candidate paraphrases. Candidates are ranked by a paraphrase probability that combines two translation probabilities and sums over all pivot phrases. A language model can then re-rank candidates for a given sentence. The experiments use German-English Europarl with both Giza++ alignments and a hand-corrected gold standard. Native speakers judged each paraphrase inside real sentences. Paraphrase quality rises when alignments improve and when candidates are limited to those sharing the original phrase's meaning. The approach matters because it scales with available parallel data and naturally handles multi-word units.

Key points

Data and alignment: The corpus is the German-English section of Europarl (version 2). Automatic word alignments come from Giza++, and phrases are extracted with the Och and Ney (2003) heuristic.
Probability formula: The paraphrase probability is p(e2|e1) = Σ_f p(f|e1)·p(e2|f). The translation probabilities are estimated by maximum likelihood from alignment counts.
Test phrases: 46 multi-word test phrases were randomly drawn from WordNet. Each had to occur several times in the first 50,000 corpus sentences.
Evaluation setup: Evaluation used 289 substitution sets containing 1366 unique sentences. Two native English speakers judged each sentence. A paraphrase counted as correct only if it kept both the meaning and the grammaticality.
Results with automatic alignments: Accuracy was 48.9% at baseline. Restricting candidates to the same foreign pivot (a proxy for word sense) raised it to 57%. Adding language-model re-ranking raised it further to 61.9%.

Interesting ideas

Pivoting through another language: Two strings are treated as near-synonyms when they share a translation. This turns any parallel corpus into a source of paraphrases, which could help low-resource settings, for example pivoting Ukrainian through English or Polish.
Ranked, probabilistic output: The method returns a scored list rather than one fixed paraphrase. A downstream step, such as a language model, can then pick the candidate that fits the actual sentence.
Separating method quality from preprocessing: The authors built gold-standard alignments only for the phrases under study. This lets them measure the method itself apart from alignment errors, a cheap design that works whenever a pipeline depends on a noisy upstream component.
Substitution-based evaluation: Each candidate is placed into several real sentences containing the original phrase, not judged in isolation. A paraphrase must preserve both meaning and grammar, which gives a stricter, context-aware test for any lexical substitution or synonym resource.
Foreign phrase as a sense label: Sense is approximated by requiring candidates to share the same foreign pivot phrase. This shows how translation splits can stand in for a sense disambiguation system when parallel data is at hand.

Topics: Word sense disambiguation · Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1219840.1219914

@inproceedings{bannard2005paraphrasing,
  title = {{Paraphrasing with Bilingual Parallel Corpora}},
  author = {Bannard, Colin and Callison-Burch, Chris},
  booktitle = {Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)},
  year = {2005},
  doi = {10.3115/1219840.1219914},
  url = {https://doi.org/10.3115/1219840.1219914}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In Proceedings of HLT/NAACL. find our card
Regina Barzilay and Kathleen McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of ACL. find our card
Peter Brown, Stephen Della Pietra, Vincent Della Pietra, and Robert Mercer. 1993. The mathematics of machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311, June. find
Chris Callison-Burch, David Talbot, and Miles Osborne. 2004. Statistical machine translation with word- and sentence-aligned parallel corpora. In Proceedings of ACL. find
Mona Diab and Philip Resnik. 2002. An unsupervised method for word sense tagging using parallel corpora. In Proceedings of ACL. find
Ali Ibrahim, Boris Katz, and Jimmy Lin. 2003. Extracting structural paraphrases from aligned monolingual corpora. In Proceedings of the Second International Workshop on Paraphrasing (ACL 2003). find
Lidija Iordanskaja, Richard Kittredge, and Alain Polgére. 1991. Lexical selection and paraphrase in a meaning-text generation model. In Cécile L. Paris, William R. Swartout, and William C. Mann, editors, Natural Language Generation in Artificial Intelligence and Computational Linguistics. Kluwer Academic. find
Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of HLT/NAACL. find
Philipp Koehn. 2002. Europarl: A multilingual corpus for evaluation of machine translation. Unpublished Draft. find
Philipp Koehn. 2004. Pharaoh: A beam search decoder for phrase-based statistical machine translation models. In Proceedings of AMTA. find
Dekang Lin and Patrick Pantel. 2001. DIRT - discovery of inference rules from text. In Proceedings of ACM SIGKDD Conference on Knowledge Discovery and Data Mining. find
Daniel Marcu and William Wong. 2002. A phrase-based, joint probability model for statistical machine translation. In Proceedings of EMNLP. find
Kathleen R. McKeown, Regina Barzilay, David Evans, Vasileios Hatzivassiloglou, Judith L. Klavans, Ani Nenkova, Carl Sable, Barry Schiffman, and Sergey Sigelman. 2002. Tracking and summarizing news on a daily basis with Columbia’s Newsblaster. In Proceedings of the Human Language Technology Conference. find
Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51, March. find
Bo Pang, Kevin Knight, and Daniel Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In Proceedings of HLT/NAACL. find our card
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of ACL. find our card
Christoph Tillmann. 2003. A projection extension algorithm for statistical machine translation. In Proceedings of EMNLP. find
Stephan Vogel, Ying Zhang, Fei Huang, Alicia Tribble, Ashish Venugopal, Bing Zhao, and Alex Waibel. 2003. The CMU statistical machine translation system. In Proceedings of MT Summit 9. find