Elamur Research
☰
home / papers / 2018 / BLEU is Not Suitable for the Evaluation of Text Simplification

BLEU is Not Suitable for the Evaluation of Text Simplification

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1081.pdf

Summary

The authors ask whether BLEU, the dominant automatic metric for Text Simplification (TS), can reliably judge sentence splitting, which is the central structural simplification operation and the focus of the Split-and-Rephrase task. They built HSplit, a fully manual gold-standard splitting corpus. Four annotators rewrote the 359 complex test sentences of Xu et al. (2016) under two guideline sets: one asked them to split as much as possible, the other to split only when it simplifies. Next, the authors measured Spearman correlations of automatic metrics against human judgments on four dimensions: how grammatical the output is (G), how well it keeps the original meaning (M), how simple it is (S), and how simple its structure is (StS). The metrics were BLEU, iBLEU, SARI, FK and negative Levenshtein distance to the source (-LD_SC). They used two settings: the standard Xu et al. references, and HSplit itself as references. BLEU tended to correlate negatively with simplicity in both settings. When splitting was involved, BLEU tracked G and M no better than, or worse than, plain similarity to the input. The study argues against using BLEU for TS evaluation and calls for structure-aware alternatives.

Key points

  • HSplit corpus. It has four versions (HSplit1–4), each with 359 sentences.
    • On average, the versions produce 2.02 output sentences per input and split 72% of the inputs.
    • Differences between annotators outweighed differences caused by the two guideline sets.
  • Scores for the unchanged input vs. HSplit. These figures cover the first 70 sentences.
    • BLEU-8ref gives the unchanged input (Identity) 94.63 and HSplit 73.03.
    • Human ratings barely differ: G is 4.80 vs. 4.43 and M is 5.00 vs. 4.70.
  • BLEU as a conservatism measure. BLEU correlates strongly with -LD_SC. For example, BLEU-1ref reaches 0.86 when outputs have no splits.
  • HSplit as references. System-level correlations of BLEU with human ratings are:
    • G: 0.57
    • M: 0.11
    • S: −0.70
    • StS: −0.60
  • SARI. It correlates positively with S (0.52 without splits) but not with StS in either setting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

111 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 106 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1081

@inproceedings{sulem2018bleu,
  title = {{BLEU is Not Suitable for the Evaluation of Text Simplification}},
  author = {Sulem, Elior and Abend, Omri and Rappoport, Ari},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1081},
  url = {https://doi.org/10.18653/v1/d18-1081}
}

References

33 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. text simplification. ArXiv:1710.02318 [cs.CL]. https://arxiv.org/pdf/1710.02318. pdf. ↗
  2. Roee Aharoni and Yoav Goldberg. 2018. Split and rephrase: Better evaluation and a stronger baseline. In Proc. of ACL’18, Short papers, pages 719– 724. http://aclweb.org/anthology/ P18-2114. ↗
  3. Jana M. Mason and Janet R. Kendall. 1979. Facilitating reading comprehension through text structure manipulation. Alberta Journal of Medical Psychology, 24:68–76. find
  4. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of BLEU in machine translation. In Proc. of EACL’06, pages 249–256. http://www.aclweb.org/ anthology/E06-1032. ↗
  5. Kshitij Mishra, Ankush Soni, Rahul Sharma, and Dipti Misra Sharma. 2014. Exploring the effects of sentence simplification on Hindi to English Machine Translation systems. In Proc. of the Workshop on Automatic Text Simplification: Methods and Applications in the Multilingual Society, pages 21– 29. http://www.aclweb.org/anthology/ W14-5603. ↗
  6. Raman Chandrasekar, Christine Doran, and Bangalore Srinivas. 1996. Motivations and methods for sentence simplification. In Proc. of COL-ING’96, pages 1041–1044. http://aclweb.org/anthology/C/C96/C96-2183.pdf. ↗
  7. Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proc. of ACL14, pages 435– 445. http://aclweb.org/anthology/P/ P14/P14-1041.pdf. ↗
  8. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. find
  9. Yvette Graham. 2015. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proc. of EMNLP’15, pages 128–137. http:// www.aclweb.org/anthology/D15-1013. ↗ our card
  10. Shashi Narayan and Claire Gardent. 2016. Unsupervised sentence simplification using deep semantics. In Proc. of INLG’16, pages 111– 120. http://aclweb.org/anthology/W/ W16/W16-6620.pdf. ↗ our card
  11. J. Peter Kincaid, Robert P. Fishburne Jr., Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and Flesch reading ease formula) for Navy enlisted personnel. Technical report, Defense Technical Information Center (DTIC) Document. find
  12. Shashi Narayan, Claire Gardent, Shay B. Cohen, and Anastasia Shimorina. 2017. Split and rephrase. In Proc. of EMNLP’17, pages 617–627. http:// aclweb.org/anthology/D17-1064. ↗
  13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Buch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: open source toolkit for statistical machine translation. In Proc. of ACL’07 on interactive poster and demonstration sessions, pages 177– 180. http://aclweb.org/anthology/P/ P07/P07-2045.pdf. ↗
  14. Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P. Dinu. 2017. Exploring neural text simplification models. In Proc. of ACL’17 (Short paper), pages 85–91. http://www.aclweb.org/anthology/P17-2014. ↗
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL’02, pages 311–318. http://aclweb.org/anthology/P/P02/P02-1040.pdf. ↗ our card
  16. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proc. of the Workshop on Statistical Machine Translation. http://www.aclweb.org/anthology/W06-3114. ↗
  17. Y. Albert Park and Roger Levy. 2011. Automated whole sentence grammar correction using a noisy channel model. In Proc.of ACL-HLT’11, pages 934– 944. http://aclweb.org/anthology/P/ P11/P11-1094.pdf. ↗
  18. Junyi Jessi Li and Ani Nenkova. 2015. Detecting content-heavy sentences: A cross-language case study. In Proc. of EMNLP’15, pages 1271–1281. http://www.aclweb.org/anthology/ D15-1148. ↗
  19. Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. TACL, 4:61–74. http://www.aclweb.org/ anthology/Q16-1005. ↗ our card
  20. Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. In Proc. of EMNLP’02, pages 63–70. http://www.aclweb.org/ anthology/W/W02/W02-0109.pdf. ↗
  21. Advaith Siddharthan. 2006. Syntactic simplification and text cohesion. Research on Language and Computation, 4:77–109. find
  22. Shuming Ma and Xu Sun. 2017. A semantic relevance based neural network for text summarization and Advaith Siddharthan and M. A. Angrosh. 2014. Hybrid text simplification using synchronous dependency Yaoyuan Zhang, Zhenxu Ye, Dongyan Zhao, and Rui Yan. 2017. A constrained sequence-to-sequence neural model for sentence simplification. ArXiv:1704.02312 [cs.CL]. https:// arxiv.org/pdf/1704.02312.pdf. grammars with hand-written and automatically harvested rules. In Proc. of EACL’14, pages 722– 731. http://aclweb.org/anthology/E/ E14/E14-1076.pdf. ↗
  23. Sanja Štajner, Hannah Bechara, and Horacio Saggion. 2015. A deeper exploration of the standard PB-SMT approach to text simplification and its evaluation. In Proc. of ACL’15, Short papers, pages 823– 828. http://aclweb.org/anthology/P/ P15/P15-2135.pdf. ↗
  24. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proc. of COL-ING’10, pages 1353–1361. http://aclweb.org/anthology/C/C10/C10-1152.pdf. ↗ our card
  25. Sanja Štajner, Ruslan Mitkov, and Horacio Saggion. 2014. One step closer to automatic evaluation of text simplification systems. Proc. of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 1–10. http:// www.aclweb.org/anthology/W14-1201. ↗
  26. Elior Sulem, Omri Abend, and Ari Rappoport. 2018a. Semantic structural evaluation for text simplification. In Proc. of NAACL’18, pages 685– 696. http://aclweb.org/anthology/ N18-1063. ↗ our card
  27. Elior Sulem, Omri Abend, and Ari Rappoport. 2018b. Simple and effective text simplification using semantic and neural methods. In Proc. of ACL’18, pages 162–173. http://aclweb.org/anthology/P18-1016. ↗ our card
  28. Hong Sun and Ming Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In Proc. of ACL’12, pages 38– 42. http://aclweb.org/anthology/P/ P12/P12-2008.pdf. ↗
  29. Sandra Williams, Ehud Reiter, and Liesl Osman. 2003. Experiments with discourse-level choices and readability. In Proc. of the European Natural Language Workshop (ENLG). find
  30. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proc. of EMNLP’11, pages 409–420. http://aclweb.org/anthology/D/D11/D11-1038.pdf. ↗ our card
  31. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proc. of ACL’12, pages 1015–1024. http://aclweb.org/ anthology/P/P12/P12-1107.pdf. ↗ our card
  32. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. TACL, 4:401–415. http://aclweb.org/ anthology/Q/Q16/Q16-1029.pdf. ↗ our card
  33. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proc. of EMNLP’17, pages 595–605. http:// aclweb.org/anthology/D17-1062. ↗ our card