Elamur Research
☰
home / papers / 2021 / The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification

The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.cl-4.28.pdf

Summary

Sentence simplification systems usually rewrite text through several operations at once, including lexical substitution, deletion and sentence splitting. Yet their outputs are typically scored with metrics built around a single operation: SARI targets lexical paraphrasing and SAMSA targets splitting. This paper asks how well such metrics reflect human perception of overall simplicity. The authors crowdsourced a new resource, Simplicity-DA, using the Direct Assessment methodology. It contains continuous ratings of fluency, meaning preservation and simplicity for system outputs on TurkCorpus sentences. They used it alongside two older datasets (Simplicity Gain and Structural Simplicity) in what they present as the first meta-evaluation of automatic metrics for Text Simplification. Correlations with human scores were analysed across three conditions: perceived simplicity level, type of system, and the reference set used (ASSET, TurkCorpus, HSplit). All three factors changed how well metrics agreed with humans, and operation-specific metrics showed clear weaknesses on multi-operation outputs. The work matters because it shows that a single overall correlation figure can mislead. It also offers practical guidance for evaluating simplification systems.

Key points

  • Simplicity-DA dataset: 600 automatic simplifications from six systems, three of them neural sequence-to-sequence models. Each output received 15 ratings, compared with 5 in Simplicity Gain and 3 in Structural Simplicity. Data and code are released at github.com/feralvam/metaeval-simplification.
  • Effect of output quality and system type: On Simplicity-DA, metrics score low-quality simplifications more reliably than good ones. Most metrics also correlate better with humans on outputs from neural seq2seq systems.
  • Effect of references: Pooling ASSET, TurkCorpus and HSplit gives no significant improvement over using ASSET alone. Early tests that chose references per sentence, based on the operations performed, looked promising.
  • Recommendations: During development, first use BERTScore with ASSET references to check output quality, then use SARI and/or SAMSA to estimate simplicity gains. Final comparisons should rely on human evaluation. A high BERTScore does not by itself mean the output is simpler, because references vary in how simple they are.
  • Proposals for future resources: Collect better-controlled and more diverse ratings, and design metrics that look at both the source sentence and the output. Annotate references with their simplicity level, and evaluate new metrics beyond absolute overall correlation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

92 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00418

@article{alvamanchego2021un,
  title = {{The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification}},
  author = {Alva-Manchego, Fernando and Scarton, Carolina and Specia, Lucia},
  journal = {Computational Linguistics},
  year = {2021},
  doi = {10.1162/coli\_a\_00418},
  url = {https://doi.org/10.1162/coli_a_00418}
}

References

64 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abend, Omri and Ari Rappoport. 2013. Universal conceptual cognitive annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238, Sofia. find our card
  2. Aluı́sio, Sandra M., Lucia Specia, Thiago A. S. Pardo, Erick G. Maziero, Helena M. Caseli, and Renata P. M. Fortes. 2008. A corpus analysis of simple account texts and the proposal of simplification strategies: First steps towards text simplification systems. In Proceedings of the 26th Annual ACM International Conference on Design of Communication, SIGDOC ’08, pages 15–22, Lisbon. https://doi.org/10.1145/1456536.145654 ↗
  3. Alva-Manchego, Fernando. 2020. Automatic Sentence Simplification with Multiple Rewriting Transformations. Ph.D. thesis, University of Sheffield, Sheffield, UK. find
  4. Alva-Manchego, Fernando, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017. Learning how to simplify from explicit labeling of complex-simplified text pairs. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 295–305, Taipei. find our card
  5. Alva-Manchego, Fernando, Louis Martin, Antoine Bordes, Carolina Scarton, Benoı̂t Sagot, and Lucia Specia. 2020. ASSET: A data set for tuning and evaluation of sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4668–4679, Online. https://doi.org/10.18653/v1/2020.acl-main.424 ↗ our card
  6. Alva-Manchego, Fernando, Louis Martin, Carolina Scarton, and Lucia Specia. 2019. EASSE: Easier automatic sentence simplification evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 49–54, Hong Kong. https://doi.org/10.18653/v1/D19-3009 ↗
  7. Alva-Manchego, Fernando, Carolina Scarton, and Lucia Specia. 2020. Data-driven sentence simplification: Survey and benchmark. Computational Linguistics, 46(1):135–187. https://doi.org/10 .1162/coli a 00370 ↗ our card
  8. Amidei, Jacopo, Paul Piwek, and Alistair Willis. 2019. Agreement is overrated: A plea for correlation to assess human evaluation reliability. In Proceedings of the 12th International Conference on Natural Language Generation, pages 344–354, Tokyo. https://doi.org/10.18653/v1/W19-8642 ↗
  9. Barrault, Loı̈c, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence. https://doi.org/10.18653/v1/W19-5301 ↗ our card
  10. Bojar, Ondřej, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Brussels. https://doi.org/10.18653/v1/W18-6401 ↗ our card
  11. Bojar, Ondřej, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, pages 489–513, Copenhagen. https://doi.org/10.18653/v1/W17 -4755 ↗ our card
  12. Bojar, Ondřej, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 199–231, Berlin. https://doi.org/10.18653/v1/W16-2302 ↗ our card
  13. Bott, Stefan and Horacio Saggion. 2011. Spanish text simplification: An exploratory study. Procesamiento del Lenguaje Natural, 47:87–95. find
  14. Cooper, Michael and Matthew Shardlow. 2020. CombiNMT: An exploration into neural text simplification models. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5588–5594, Marseille, France. find
  15. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, MN. find
  16. Dong, Yue, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3393–3402, Florence. https://doi.org/10.18653/v1/P19-1331 ↗ our card
  17. Feblowitz, Dan and David Kauchak. 2013. Sentence simplification as tree transduction. In Proceedings of the Second Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 1–10, Sofia. find
  18. Fomicheva, Marina and Lucia Specia. 2019. Taking MT evaluation metrics to extremes: Beyond correlation with human judgments. Computational Linguistics, 45(3):515–558. https://doi.org/10 .1162/coli a 00356 ↗
  19. Freitag, Markus, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71, Online. https://doi.org/10.18653/v1/2020.emnlp-main.5 ↗ our card
  20. Ganitkevitch, Juri, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of NAACL-HLT, pages 758–764, Atlanta, GA. find our card
  21. Graham, Yvette, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia. find
  22. Graham, Yvette, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, 23(1):3–30. https://doi.org/10.1017/S1351324915000339 ↗
  23. Guo, Han, Ramakanth Pasunuru, and Mohit Bansal. 2018. Dynamic multi-level multi-task learning for sentence simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 462–476, Santa Fe, NM. find
  24. Jalili Sabet, Masoud, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1627–1643, Online. https://doi.org/10.18653/v1/2020.findings-emnlp.147 ↗
  25. Jiang, Chao, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural CRF model for sentence alignment in text simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7943–7960, Online. https://doi.org/10.18653/v1/2020.acl-main.709 ↗ our card
  26. Kincaid, J. P., R. P. Fishburne, R. L. Rogers, and B. S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical Report 8–75, Chief of Naval Technical Training: Naval Air Station Memphis. https://doi.org/10 .21236/ADA006655 ↗
  27. Koehn, Philipp, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, ACL ’07, pages 177–180, Stroudsburg, PA. find
  28. Kriz, Reno, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Callison-Burch. 2019. Complexity-weighted loss and diverse reranking for sentence simplification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3137–3147, Minneapolis, MN. https://doi.org/10.18653/v1/N19-1317 ↗
  29. Kumar, Dhruv, Lili Mou, Lukasz Golab, and Olga Vechtomova. 2020. Iterative edit-based unsupervised sentence simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7918–7928, Online. https://doi.org/10.18653/v1/2020.acl-main.707 ↗
  30. Landis, J. Richard and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. https://doi.org/10 .2307/2529310, PubMed: 843571 ↗
  31. Ma, Qingsong, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671–688, Brussels. https://doi.org/10.18653/v1/W18-6450 ↗ our card
  32. Ma, Qingsong, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence. https://doi.org/10.18653/v1/W19-5302 ↗ our card
  33. Maddela, Mounica, Fernando Alva-Manchego, and Wei Xu. 2021. find
  34. Controllable text simplification with explicit paraphrasing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3536–3553, Online. find our card
  35. Martin, Louis, Éric de la Clergerie, Benoı̂t Sagot, and Antoine Bordes. 2020. Controllable sentence simplification. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4689–4698, Marseille, France. https://doi.org/10 .18653/v1/2021.naacl-main.277 ↗ our card
  36. Mathur, Nitika, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. https://doi.org/10.18653/v1/2020.acl -main.448 ↗ our card
  37. Mathur, Nitika, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. find our card
  38. Narayan, Shashi and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 435–445, Baltimore, MD. https://doi.org/10 .3115/v1/P14-1041 ↗
  39. Narayan, Shashi and Claire Gardent. 2016. Unsupervised sentence simplification using deep semantics. In Proceedings of the 9th International Natural Language Generation Conference, pages 111–120, Edinburgh, UK. https://doi.org/10 .18653/v1/W16-6620 ↗ our card
  40. Nisioi, Sergiu, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P. Dinu. 2017. Exploring neural text simplification models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 85–91, Vancouver. https://doi.org/10.18653/v1/P17-2014 ↗
  41. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL ’02, pages 311–318, ACL, Philadelphia, PA. https://doi.org/10.3115/1073083 .1073135 ↗ our card
  42. Pavlick, Ellie and Chris Callison-Burch. 2016. Simple PPDB: A paraphrase database for simplification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 143–148, Berlin. find
  43. Petersen, Sarah E. 2007. Natural Language Processing Tools for Reading Level Assessment and Text Simplification for Bilingual Education. Ph.D. thesis. University of Washington, Seattle, WA, USA. AAI3275902. find
  44. Post, Matt. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels. https://doi.org/10.18653/v1/W18 -6319 ↗ our card
  45. Reiter, Ehud. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393–401. find our card
  46. Rosenthal, James A. 1996. Qualitative descriptors of strength of association and effect size. Journal of Social Service Research, 21(4):37–59. https://doi.org/10 .1300/J079v21n04 02 ↗
  47. Shrout, Patrick E. and Joseph L. Fleiss. 1979. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2):420–428. https://doi.org/10.1037/0033-2909.86.2.420, PubMed: 18839484 ↗
  48. Sulem, Elior, Omri Abend, and Ari Rappoport. 2018a. BLEU is not suitable for the evaluation of text simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 738–744, Brussels. https://doi.org/10.18653/v1/D18-1081 ↗ our card
  49. Sulem, Elior, Omri Abend, and Ari Rappoport. 2018b. Semantic structural evaluation for text simplification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 685–696, New Orleans, LA. https://doi.org/10.18653/v1/N18-1063 ↗ our card
  50. Sulem, Elior, Omri Abend, and Ari Rappoport. 2018c. Simple and effective text simplification using semantic and neural methods. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 162–173, Melbourne. https://doi.org/10.18653/v1/P18 -1016 ↗ our card
  51. Sun, Hong and Ming Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2, ACL ’12, pages 38–42, Stroudsburg, PA. https://doi.org/10 .21105/joss.01026 ↗
  52. Štajner, Sanja, Ruslan Mitkov, and Horacio Saggion. 2014. One step closer to automatic evaluation of text simplification systems. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), pages 1–10, Gothenburg. https://doi.org/10.3115/v1/W14-1201 ↗
  53. Vallat, Raphael. 2018. Pingouin: Statistics in Python. Journal of Open Source Software, 3(31):1026. find
  54. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30. Curran Associates, Inc., pages 5998–6008. find
  55. Vu, Tu, Baotian Hu, Tsendsuren Munkhdalai, and Hong Yu. 2018. Sentence simplification with memory-augmented neural networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 79–85, New Orleans, LA. https://doi.org/10.18653/v1/N18-2013 ↗
  56. Williams, Evan James. 1959. Regression Analysis, volume 14. Wiley, New York. find
  57. Woodsend, Kristian and Mirella Lapata. 2011. Learning to Simplify Sentences with Quasi-Synchronous Grammar and Integer Programming. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 409–420, Edinburgh. find our card
  58. Wubben, Sander, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1, ACL ’12, pages 1015–1024, Stroudsburg, PA. find our card
  59. Xu, Wei, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. https://doi.org/10.1162/tacl_a_00139 ↗ our card
  60. Xu, Wei, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. https://doi.org/10.1162/tacl_a_00107 ↗ our card
  61. Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. International Conference on Learning Representations. 43 pages. find
  62. Zhang, Xingxing and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 595–605, Copenhagen. https://doi.org/10.18653/v1/D17-1062 ↗ our card
  63. Zhao, Sanqiang, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. 2018. Integrating transformer and paraphrase rules for sentence simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3164–3173, Brussels. https://doi.org/10.18653/v1/D18 -1355 ↗
  64. Zhu, Zhemin, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd International Conference on Computational Linguistics, COLING ’10, pages 1353–1361, Stroudsburg, PA. find our card