Elamur Research⌕
☰
home / papers / 2012 / Re-examining Machine Translation Metrics for Paraphrase Identification

Re-examining Machine Translation Metrics for Paraphrase Identification

Summary

The paper revisits an older idea: since an MT evaluation metric essentially judges whether a system translation says the same thing as a reference, such metrics might work as paraphrase detectors. Earlier attempts relied on BLEU, NIST, WER and PER. The authors test whether the newer, more linguistically informed metrics developed in recent MetricsMATR and WMT competitions do better. They compute eight metrics (BLEU, NIST, TER, TERp, METEOR, SEPIA, BADGER, MAXSIM) as the only features for a meta-classifier, which averages the probability outputs of three base learners. They evaluate on two corpora built in very different ways. The first is the Microsoft Research Paraphrase corpus (MSRP), mined from related news stories. The second is a sentence-level set they derived from the PAN 2010 plagiarism collection, where humans deliberately rewrote text. Using only metric features, the system beats all earlier approaches on MSRP and performs strongly on PAN. A manual error analysis identifies the main failure types. The work shows that off-the-shelf evaluation metrics make a strong, cheap paraphrase baseline, and it contributes a new, more realistic benchmark.

Key points

  • Features: Eight MT metrics serve as features, some expanded into several values: BLEU1–4, NIST1–5, TER, TERp, METEOR, SEPIA, BADGER and MAXSIM. TESLA was dropped because it matched MAXSIM in cross-validation but ran more slowly.

  • Classifier: The meta-classifier, built in WEKA, takes an unweighted average of the probabilities from three learners:

    • logistic regression,
    • an SMO support vector machine,
    • an instance-based nearest-neighbour learner.

    These three were selected as the best performers in 5-fold cross-validation.

  • Data:

    • MSRP uses its standard split of 4,076 training and 1,725 test pairs, with an annotator kappa of 0.62.
    • The PAN 2010 collection contains 41,233 Gutenberg documents with 94,202 inserted plagiarism cases. Only the human (Turker) rewrites were used.

  • Released resources: a 13,000-pair PAN sentence dataset, plus error-analysis annotations for 100 MSRP pairs and 100 PAN pairs.

  • Findings:

    • TERp alone already outperforms many feature-rich supervised systems.
    • The only error type producing false positives was misleading lexical overlap.

Interesting ideas

  • I1.

    Repurpose evaluation metrics as features. Similarity scores built for one task, here MT evaluation, can be fed straight into a classifier for a related task. This gives a strong, cheap baseline without new feature engineering, and could be tried for Ukrainian paraphrase or plagiarism detection.

  • I2.

    Test on corpora built in different ways. Evaluating on both naturally occurring paraphrases (news) and deliberately crafted ones (plagiarism) shows whether a method only exploits accidental topical overlap. This is useful for checking robustness in any similarity task.

  • I3.

    Use a crowd vote as a graded label. Instead of a single yes/no, ask many annotators (20 or more) for a binary judgment and use the share of "yes" votes as the degree of equivalence. This yields graded labels without asking people to use rating scales.

  • I4.

    Classify errors to find preprocessing fixes. The error taxonomy points to concrete pipeline modules:

    • normalising dates and numbers,
    • named-entity recognition,
    • anaphora resolution,
    • spelling correction.

    Doing this kind of error analysis before adding model complexity is a good general habit.

  • I5.

    Weight words by importance, not just overlap. Proper names and the reported content of speech should count more than other words. This idea could guide weighted similarity measures elsewhere.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

87 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{madnani2012re,
  title = {{Re-examining Machine Translation Metrics for Paraphrase Identification}},
  author = {Madnani, Nitin and Tetreault, Joel and Chodorow, Martin},
  booktitle = {Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2012},
  url = {https://aclanthology.org/N12-1019/}
}

References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. W. Aha, D. Kibler, and M. K. Albert. 1991. Instance-based learning algorithms. Mach. Learn., 6:37–66. ↗ our card
  2. C. Callison-Burch, P. Koehn, C. Monz, K. Peterson, and O. Zaidan, editors. 2010. Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. find
  3. Y. S. Chan and H. T. Ng. 2008. MAXSIM: A maximum similarity metric for machine translation evaluation. In Proceedings of ACL-HLT, pages 55–62. find
  4. D. Das and N.A. Smith. 2009. Paraphrase Identification as Probabilistic Quasi-synchronous Recognition. In Proceedings of ACL-IJCNLP, pages 468–476. find our card
  5. M. Denkowski and M. Lavie. 2010. Extending the METEOR Machine Translation Metric to the Phrase Level. In Proceedings of NAACL. find
  6. G. Doddington. 2002. Automatic Evaluation of Machine Translation Quality using N-gram Co-occurrence Statistics. In Proceedings of HLT, pages 138–145. find
  7. B. Dolan, C. Quirk, and C. Brockett. 2004. Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources. In Proceedings of COLING, pages 350–356, Geneva, Switzerland. find our card
  8. S. Fernando and M. Stevenson. 2008. A Semantic Similarity Approach to Paraphrase Detection. In Proceedings of the Computational Linguistics UK (CLUK) 11th Annual Research Colloquium. find
  9. A. Finch, Y.S. Hwang, and E. Sumita. 2005. Using Machine Translation Evaluation Techniques to Determine Sentence-level Semantic Equivalence. In Proceedings of the Third International Workshop on Paraphrasing, pages 17–24. find
  10. N. Habash and A. El Kholy. 2008. SEPIA: Surface Span Extension to Syntactic Dependency Precision-based MT Evaluation. In Proceedings of the Workshop on Metrics for Machine Translation at AMTA. find
  11. M. Hall, E. Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11. ↗
  12. A. Islam and D. Inkpen. 2007. Semantic Similarity of Short Texts. In Proceedings of RANLP, pages 291– 297. find
  13. J. J. Jiang and D. W. Conrath. 1997. Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy. CoRR, cmp-lg/9709008. ↗
  14. S. S. Keerthi, S. K. Shevade, C. Bhattacharyya, and K. R. K. Murthy. 2001. Improvements to Platt’s SMO Algorithm for SVM Classifier Design. Neural Comput., 13(3):637–649. ↗
  15. Z. Kozareva and A. Montoyo. 2006. Paraphrase Identification on the Basis of Supervised Machine Learning Techniques. In Proceedings of FinTAL, pages 524– 233. find
  16. N. Madnani, J. Tetreault, M. Chodorow, and A. Rozovskaya. 2011. They Can Help: Using Crowdsourcing to Improve the Evaluation of Grammatical Error Detection Systems. In Proceedings of ACL (Short Papers). find
  17. R. Mihalcea, C. Corley, and C. Strapparava. 2006. Corpus-based and Knowledge-based Measures Of Text Semantic Similarity. In Proceedings of AAAI, pages 775–780. find
  18. NIST. 2008. NIST MetricsMATR Challenge. Information Access Division. http://www.itl.nist.gov/iad/mig/tests/metricsmatr/. ↗
  19. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of ACL. find our card
  20. S. Parker. 2008. BADGER: A New Machine Translation Metric. In Proceedings of the Workshop on Metrics for Machine Translation at AMTA. find
  21. John C. Platt. 1999. Advances in kernel methods. chapter Fast Training of Support Vector Machines using Sequential Minimal Optimization, pages 185–208. MIT Press. find
  22. M. Potthast, B. Stein, A. Barrón-Cedeño, and P. Rosso. 2010. An Evaluation Framework for Plagiarism Detection. In Proceedings of COLING, pages 997–1005. find
  23. L. Qiu, M. Y. Kan, and T. S. Chua. 2006. Paraphrase Recognition via Dissimilarity Significance Classification. In Proceedings of the EMNLP, pages 18–26. find
  24. V. Rus, P.M. McCarthy, M.C. Lintean, D.S. McNamara, and A.C. Graesser. 2008. Paraphrase Identification with Lexico-Syntactic Graph Subsumption. In Proceedings of FLAIRS, pages 201–206. find
  25. M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of AMTA. find our card
  26. M. Snover, N. Madnani, B. Dorr, and R. Schwartz. 2009. TER-Plus: Paraphrase, Semantic, and Alignment Enhancements to Translation Edit Rate. Machine Translation, 23(2–3):117–127. ↗
  27. R. Socher, E.H. Huang, J. Pennington, A.Y. Ng, and C.D. Manning. 2011. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In Advances in Neural Information Processing Systems 24 (NIPS). find
  28. S. Wan, R. Dras, M. Dale, and C. Paris. 2006. Using Dependency-based Features to Take the ”para-farce” Out of Paraphrase. In Proceedings of the Australasian Language Technology Workshop (ALTW), pages 131– 138. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/madnani-2012-re-examining-machine-translation-metrics/