Elamur Research⌕
☰
home / papers / 2019 / EASSE: Easier Automatic Sentence Simplification Evaluation

EASSE: Easier Automatic Sentence Simplification Evaluation

Summary

Evaluating sentence simplification (SS) systems has been fragmented. The standard metrics are scattered across separate repositories, sometimes written in different languages (corpus-level SARI existed only in Java), and some, such as SAMSA, are poorly documented or depend on chains of scripts with hard-coded paths. This hurts reproducibility and makes systems hard to compare. The authors built EASSE, a Python package with a command-line interface that combines several resources:

  • the common corpus-level metrics (BLEU, SARI, SAMSA, FKGL);
  • word-level analysis of which edit operations a system performs well;
  • reference-free Quality Estimation features;
  • direct access to standard test sets such as TurkCorpus, HSplit and PWKP.

It also produces an HTML report that supports both numeric and qualitative inspection of a system's output. Several metrics were re-implemented or cleaned up along the way, fixing bugs and removing awkward dependencies. The authors report that these functionalities allow SS systems to be compared and understood better. The package matters because it gives the field one standardised, reproducible evaluation setup.

Key points

  • BLEU is computed through SacreBLEU, which takes detokenised output plus a test-set name and applies identical preprocessing to hypotheses and references.
  • Metric re-implementations:
    • SARI: the corpus-level version was ported from Java to Python. Like the Java code, it uses F1 for all three operations (add, delete, keep), even though the original paper suggested using only precision for deletion.
    • FKGL: ported to Python 3, with fixes for edge cases such as newlines being counted as words and memoization bugs.
  • SAMSA was refactored into a single function call. It now invokes the TUPA parser internally and uses a Python 3 version of the Sultan et al. monolingual aligner built on Stanford CoreNLP.
  • Word-level transformation labels (delete, move, replace, copy) come from MASSAlign's annotation algorithms. Against four human annotators on 100 sentence pairs, these labels reached a micro-averaged F1 of 0.61.
  • The QE features come from Martin et al. (2018):
    • compression ratio and Levenshtein similarity;
    • average number of splits;
    • share of unchanged sentences;
    • proportions of added and deleted words;
    • lexical complexity.

Interesting ideas

  • I1. Silver labels for operation-level scoring. The same automatic annotator is applied to original→reference pairs and to original→system pairs, and the reference annotations then serve as gold. This makes it possible to score individual edit types without manually annotated data, which could suit any rewriting task that lacks fine-grained labels.
  • I2. Best match across references. With several references, each sentence keeps its highest per-reference score before corpus averaging. This avoids penalising a system for matching one valid rewrite rather than another, and works for any multi-reference evaluation.
  • I3. Reference-free diagnostics alongside reference-based metrics. Simple features such as compression ratio or the share of untouched inputs expose behaviours that SARI or BLEU hide, for example a system that mostly copies its input. These features are cheap, so they also transfer to settings with few or no references.
  • I4. Wrapping canonical tools rather than reimplementing them. Delegating BLEU to SacreBLEU and bundling the test sets by name removes preprocessing differences as a source of irreproducible scores. A shared evaluation library for any NLP task could follow the same pattern.
  • I5. One report for numbers and examples. A single generated report that pairs aggregate scores with concrete output-versus-reference comparisons encourages qualitative checks that are often skipped. It is a useful design choice for internal benchmarking tools.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

95 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-3009

@inproceedings{alvamanchego2019easse,
  title = {{EASSE: Easier Automatic Sentence Simplification Evaluation}},
  author = {Alva-Manchego, Fernando and Martin, Louis and Scarton, Carolina and Specia, Lucia},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations},
  year = {2019},
  doi = {10.18653/v1/d19-3009},
  url = {https://doi.org/10.18653/v1/d19-3009}
}

References

18 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fernando Alva-Manchego, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017. Learning how to simplify from explicit labeling of Elior Sulem, Omri Abend, and Ari Rappoport. 2018b. Semantic structural evaluation for text simplification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 685–696, New Orleans, Louisiana. ACL. complex-simplified text pairs. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 295–305, Taipei, Taiwan. AFNLP. find our card
  2. Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2017. A transition-based directed acyclic graph parser for ucca. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1127– 1138, Vancouver, Canada. ACL. find
  3. Md Sultan, Steven Bethard, and Tamara Sumner. 2014. Back to basics for monolingual alignment: Exploiting word similarity and contextual evidence. Transactions of the Association for Computational Linguistics, 2:219–230. ↗
  4. J.P. Kincaid, R.P. Fishburne, R.L. Rogers, and B.S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical Report 8-75, Chief of Naval Technical Training: Naval Air Station Memphis. 49 p. ↗
  5. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
  6. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland. ACL. find our card
  7. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1015–1024, Jeju Island, Korea. ACL. find our card
  8. Louis Martin, Samuel Humeau, Pierre-Emmanuel Mazaré, Éric de La Clergerie, Antoine Bordes, and Benoı̂t Sagot. 2018. Reference-less quality estimation of text simplification systems. In Proceedings of the 1st Workshop on Automatic Text Adaptation (ATA), pages 29–38, Tilburg, the Netherlands. ACL. ↗
  9. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. ↗ our card
  10. Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 435–445, Baltimore, Maryland. ACL. find
  11. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 595–605, Copenhagen, Denmark. ACL. find our card
  12. Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. 2018. Integrating transformer and paraphrase rules for sentence simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3164–3173, Brussels, Belgium. ACL. find our card
  13. Gustavo H. Paetzold, Fernando Alva-Manchego, and Lucia Specia. 2017. Massalign: Alignment and annotation of comparable documents. In Proceedings of the IJCNLP 2017, System Demonstrations, pages 1–4, Tapei, Taiwan. ACL. ↗
  14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Philadelphia, Pennsylvania. ACL. find our card
  15. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd International Conference on Computational Linguistics, COLING ’10, pages 1353–1361, Stroudsburg, PA, USA. ACL. find our card
  16. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191. Association for Computational Linguistics. find our card
  17. Sanja Štajner, Maja Popović, Horacio Saggion, Lucia Specia, and Mark Fishel. 2016. Shared task on quality assessment for text simplification. In Proceeding of the Workshop on Quality Assessment for Text Simplification - LREC 2016, QATS 2016, pages 22–31, Portorož, Slovenia. ELRA. find
  18. Elior Sulem, Omri Abend, and Ari Rappoport. 2018a. Bleu is not suitable for the evaluation of text simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 738–744. ACL. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/manchego-2019-easse-easier-automatic-sentence-simplification/