Elamur Research⌕
☰
home / papers / 2006 / Paraphrasing for Automatic Evaluation

Paraphrasing for Automatic Evaluation

Summary

Metrics like BLEU score machine output by counting n-gram overlap with a human reference. They therefore penalize correct translations that use different wording. Kauchak and Barzilay show that such variation is common. In the NIST 2004 MT evaluation data, two independent human translations of the same sentence almost never match exactly.

Their remedy is to rewrite each reference so it borrows vocabulary from the system output while keeping its meaning. The method works in three steps:

  1. Find unmatched word pairs that are WordNet synonyms.
  2. Check each candidate with a per-word classifier trained on automatically gathered Gigaword sentences, using word sense disambiguation-style context features.
  3. Substitute the words that pass, producing a synthetic reference that replaces the original in BLEU.

The context-filtered variant (ContextWN) brought BLEU closest to human judgments. Human-rated paraphrase accuracy across four resources (WordNet, ContextWN, LSA, Brown) ranked them in the same order as their benefit to evaluation. The work matters because it gives cheap semantic flexibility to overlap metrics. It also makes MT evaluation a task-based test of paraphrase quality.

Key points

  • Of 10,728 human reference pairs in NIST 2004 MT evaluation data, only 21 (under 0.2%) were identical, and 60% differed in at least 11 words.

  • Candidate substitutions are limited to reference words absent from the output, paired with output words absent from the reference, and kept only if WordNet lists them as synonyms. Words occurring fewer than 10,000 times in Gigaword are never substituted.

  • Context features are n-grams and position-tagged local collocations (n = 1…4, window i−2 to i+2). Training data is built automatically: sentences containing the target word are positives, and random sentences at random positions are negatives.

  • Two judges rated paraphrase accuracy on 200 samples per method, with Kappa around 0.7:

    • ContextWN: about 75%
    • WordNet: about 63%
    • Brown: 56%
    • LSA: about 30%

    LSA made by far the most substitutions yet hurt BLEU's agreement with humans.

  • Filtering discarded roughly a quarter of WordNet's proposals. Of 73 incorrect WordNet substitutions, 40 were removed, at the cost of wrongly dropping 27 correct ones. The net effect on evaluation was still positive.

Interesting ideas

  • I1. Paraphrase toward the output. Rather than producing arbitrary paraphrases of a reference, rewrite it only toward words the hypothesis actually uses. This targeted normalization could help any overlap-based metric, including evaluation of Ukrainian generation where morphology and synonymy inflate surface mismatch.
  • I2. Self-supervised substitution classifiers. Each candidate word gets its own classifier, trained on raw-corpus sentences containing it versus random sentences. This needs no labeled data, so it transfers readily to lower-resource languages with large monolingual corpora.
  • I3. Filter dictionary synonyms by context. Lexical resources overgenerate, as when "place" inside "take place" gets matched to "home". A discriminative filter of this kind could improve thesaurus-based query expansion or data augmentation.
  • I4. Accuracy over coverage. The most prolific rewriter was the least useful, which is a good caution when choosing lexical resources for augmentation or matching.
  • I5. Downstream use as an extrinsic test. Ranking paraphrase sources by their effect on a metric's agreement with humans mirrored the manual accuracy ranking. Such a downstream check can serve as a cheaper proxy for intrinsic human annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Word sense disambiguation · Machine translation

Cited by

103 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 90 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kauchak2006paraphrasing,
  title = {{Paraphrasing for Automatic Evaluation}},
  author = {Kauchak, David and Barzilay, Regina},
  booktitle = {Proceedings of the Human Language Technology Conference of the NAACL, Main Conference},
  year = {2006},
  doi = {10.3115/1220835.1220893},
  url = {https://doi.org/10.3115/1220835.1220893}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. Babych, A. Hartley. 2004. Extending the BLEU evaluation method with frequency weightings. In Proceedings of the ACL, 621–628. find
  2. S. Banerjee, A. Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, 65–72. find our card
  3. R. Barzilay, L. Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In Proceedings of NAACL-HLT, 16–23. ↗ our card
  4. P. F. Brown, P. V. deSouza, R. L. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18:467–479. find
  5. I. Dagan, O. Glickman, B. Magnini, eds. 2005. The PAS-CAL recognizing textual entailment challenge, 2005. find
  6. P. Edmonds, G. Hirst. 2002. Near synonymy and lexical choice. Computational Linguistics, 28(2):105–144. ↗
  7. A. Haghighi, A. Ng, C. Manning. 2005. Robust textual inference via graph matching. In Proceedings of NAACL-HLT, 387–394. find
  8. S. Harabagiu, D. Moldovan, M. Pasca, R. Mihalcea, M. Surdeanu, R. Bunescu, R. Girju, V. Rus, P. Morarescu. 2001. The role of lexico-semantic feedback in open-domain textual question-answering. In Proceedings of ACL, 274–291. find
  9. P. Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP, 388–395. find
  10. J. R. Landis, G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174. ↗
  11. C. Lin, F. Och. 2004. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In Proceedings of COLING, 501–507. find our card
  12. I. D. Melamed, R. Green, J. P. Turian. 2003. Precision and recall of machine translation. In Proceedings of NAACL-HLT, 61–63. find
  13. S. Miller, J. Guinness, A. Zamanian. 2004. Name tagging with word clusters and discriminative training. In Proceedings of HLT-NAACL, 337–342. find our card
  14. NIST. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics, 2002. find
  15. B. Pang, K. Knight, D. Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In Proceedings of NAACL-HLT, 102–209. find our card
  16. K. Papineni, S. Roukos, T. Ward, W. Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the ACL, 311–318. find our card
  17. C. Quirk, C. Brockett, W. Dolan. 2004. Monolingual machine translation for paraphrase generation. In Proceedings of EMNLP, 142–149. find our card
  18. R. E. Schapire, Y. Singer. 2000. Boostexter: A boosting-based system for text categorization. Machine Learning, 39(2/3):135–168. ↗
  19. R. Serafin, B. D. Eugenio. 2004. FLSA: Extending latent semantic analysis with features for dialogue act classification. In Proceedings of the ACL, 692–699. find our card
  20. A. Stent, M. Marge, M. Singhai. 2005. Evaluating evaluation methods for generation in the presense of variation. In Proceedings of CICLING, 341–351. find

↑ back to top

This page in Markdown →