Elamur Research⌕
☰
home / papers / 2017 / Exploring Neural Text Simplification Models

Exploring Neural Text Simplification Models

Summary

The paper investigates whether neural sequence-to-sequence models, borrowed from neural machine translation, can handle automated text simplification (ATS). Earlier systems typically targeted either word-level substitution or used phrase- or syntax-based SMT. The authors built Neural Text Simplification (NTS) models with OpenNMT: an LSTM encoder-decoder with global attention and input feeding. A second variant, NTS-w2v, adds concatenated pre-trained and locally trained word2vec embeddings. Training used about 280K aligned sentence pairs from English Wikipedia and Simple English Wikipedia. Beam outputs were then reranked with BLEU or SARI. In a human evaluation against PBSMT-R, SBMT (SARI+PPDB) and the unsupervised LightLS, every NTS configuration had a higher share of correct changes and produced simpler output. Grammaticality and meaning preservation stayed close to perfect. The neural models also learned to drop irrelevant content while simplifying vocabulary. The authors present this as the first ATS approach to do both jointly. The configurations, data and trained models are publicly released.

Key points

  • Architecture and training: OpenNMT with two LSTM layers (size 500) and 0.3 dropout. The vocabulary is 50,000 words. Training ran 15 epochs with plain SGD, and the checkpoint with the best dev-set perplexity was kept.
  • Training data: the EW–SEW alignment dataset of Hwang et al. (2015), roughly 150K full and 130K partial matches (similarity above 0.45). Including partial matches is what allows the model to learn sentence shortening.
  • Tuning and test data: the Xu et al. (2016) set, with 2000 tuning and 359 test sentences, each paired with eight crowd-sourced simplifications. Human evaluation covered the first 70 test sentences.
  • Correct changes: the best NTS variants reached 68–77% correct changes, versus 26.6–41.0% for the baselines.
    • NTS-w2v with SARI reranking made 110 changes, 68.1% correct, with a simplicity score of +0.63.
    • NTS with BLEU reranking scored highest on simplicity (+0.92).
    • Wilcoxon test: p < 0.001.
  • Grammaticality and meaning: the default NTS scored 4.92 for grammaticality and 4.31 for meaning preservation on a 1–5 scale. Inter-annotator agreement (quadratic kappa) was 0.78 for G, 0.63 for M and 0.66 for simplicity.

Interesting ideas

  • I1. Metric-driven reranking of beam outputs: the authors generate the top two hypotheses for beam sizes 5–12, then choose by SARI or BLEU. Each choice steers output toward a different goal: SARI favours more edits, BLEU favours accuracy, and the default favours fluency. This is a cheap way to tune a generator's behaviour without retraining.
  • I2. Global plus local embeddings, separated by side: large generic vectors (Google News) are concatenated with vectors trained on in-domain text. The encoder gets embeddings from the original-side text and the decoder from the simplified side. This could help any low-resource or domain-specific seq2seq task.
  • I3. Recovering rare words through attention: instead of character models, each UNK in the output is replaced with the source word that has the highest attention weight. This is useful for entity-heavy corpora where names fall outside the vocabulary.
  • I4. Keeping partial alignments in parallel data: the authors deliberately keep imperfect sentence matches rather than only clean ones. This lets the model learn deletion and compression, which is worth considering when building monolingual parallel corpora.
  • I5. Two-level human evaluation that values precision: each individual edit is judged, with native speakers checking meaning and grammar and non-native speakers checking ease. Separately, whole sentences are rated. The paper argues that wrong edits do more harm than missing ones, which is a sensible framing for any text-rewriting system.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

159 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 145 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2014

@inproceedings{nisioi2017exploring,
  title = {{Exploring Neural Text Simplification Models}},
  author = {Nisioi, Sergiu and Štajner, Sanja and Ponzetto, Simone Paolo and Dinu, Liviu P.},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2014},
  url = {https://doi.org/10.18653/v1/p17-2014}
}

References

31 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marcelo Adriano Amancio and Lucia Specia. 2014. An Analysis of Crowdsourced Text Simplifications . In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR). pages 123–130. find
  2. Orhan Firat, KyungHyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. CoRR abs/1601.01073. ↗ our card
  3. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In Proceedings of NAACL-HLT. pages 758–764. find our card
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473. http://arxiv.org/abs/1409.0473. ↗ our card
  5. Justin Garten, Kenji Sagae, Volkan Ustun, and Morteza Dehghani. 2015. Combining distributed vector representations for words. In Proceedings of NAACL-HLT. pages 95–101. find
  6. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”. find
  7. Goran Glavaš and Sanja Štajner. 2015. Simplifying Lexical Simplification: Do We Need Simplified Corpora? In Proceedings of the ACL&IJCNLP 2015 (Volume 2: Short Papers). pages 63–68. find our card
  8. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 . ↗
  9. Alex Graves. 2012. Supervised sequence labelling with recurrent neural networks, volume 385. Springer. find
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–1780. ↗
  11. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2015. Building end-to-end dialogue systems using generative hierarchical neural network models. arXiv preprint arXiv:1507.04808 . ↗ our card
  12. William Hwang, Hannaneh Hajishirzi, Mari Ostendorf, and Wei Wu. 2015. Aligning Sentences from Standard Wikipedia to Simple Wikipedia. In Proceedings of NAACL&HLT. pages 211–217. find
  13. Lucia Specia. 2010. Translating from complex to simplified sentences. In Proceedings of the 9th international conference on Computational Processing of the Portuguese Language (PROPOR). Springer Berlin Heidelberg, volume 6001 of Lecture Notes in Computer Science, pages 30–39. find
  14. David Kauchak. 2013. Improving text simplification language modeling using unsimplified text data. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, pages 1537–1546. find our card
  15. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(1):1929–1958. find
  16. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. 2017. OpenNMT: Open-Source Toolkit for Neural Machine Translation. ArXiv e-prints . find our card
  17. Philipp Koehn. 2010. Statistical machine translation. ↗
  18. Sanja Štajner, Hannah Bechara, and Horacio Saggion. 2015. A Deeper Exploration of the Standard PB-SMT Approach to Text Simplification and its Evaluation. In Proceedings of ACL&IJCNLP (Volume 2: Short Papers). pages 823–828. find
  19. Minh-Thang Luong and Christopher D Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. arXiv preprint arXiv:1604.00788 . ↗ our card
  20. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP. The Association for Computational Linguistics, pages 1412– 1421. ↗
  21. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 . ↗
  22. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of Workshop at International Conference on Learning Representations. ↗
  23. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL): Long Papers - Volume 1. Association for Computational Linguistics, pages 1015–1024. find our card
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. pages 3111–3119. ↗
  25. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in Current Text Simplification Research: New Data Can Help. Transactions of the Association for Computational Linguistics (TACL) 3:283–297. ↗ our card
  26. Gustavo Henrique Paetzold and Lucia Specia. 2013. Text simplification as tree transduction. In Proceedings of the 9th Brazilian Symposium in Information and Human Language Technology. pages 116–125. find
  27. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics 4:401–415. ↗ our card
  28. Gustavo Henrique Paetzold and Lucia Specia. 2016. Unsupervised lexical simplification for non-native speakers. In Proceedings of the 30th AAAI. find
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL. find our card
  30. Z. Zhu, D. Berndard, and I. Gurevych. 2010. A Monolingual Tree-based Translation Model for Sentence Simplification. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010). pages 1353–1361. find our card
  31. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, pages 45–50. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/nisioi-2017-exploring-neural-text-simplification-models/