Elamur Research
☰
home / papers / 2019 / EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing

EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1331.pdf

Summary

The paper addresses a weakness of sentence simplification systems built on machine translation. These seq2seq models learn simplifying changes only indirectly from complex-simple pairs, so they often copy the input unchanged. The authors propose EditNTS, an end-to-end Neural Programmer-Interpreter model. It works through the source sentence token by token and decides explicitly whether to KEEP a word, DELETE it, ADD a new word, or STOP. At each step, a programmer chooses the operation and an interpreter applies it. The interpreter also tracks the partial output, which lets it act as a language model that discourages ungrammatical edits. Gold edit sequences are derived with Levenshtein-style alignment, so no external aligner or rule base is needed.

On WikiLarge, WikiSmall and Newsela, EditNTS beats earlier neural systems that use no external knowledge, as measured by SARI. Human judges also rated its output simpler and better overall. Because the edit traces are visible, the system is easier to interpret. Its behaviour can also be steered, for example toward more compression or more rewording, by reweighting the loss.

Key points

  • The operation set is ADD(W), KEEP, DELETE and STOP, so the programmer chooses among V + 3 candidates. Unknown words are kept by predicting KEEP, which replaces a separate copy mechanism.
  • SARI improves over the previous best neural models by +0.95 on WikiLarge, +1.89 on WikiSmall and +1.41 on Newsela.
  • On WikiLarge, EditNTS performs comparably to PBMT-R. SBMT-SARI and DMASS+DCSS get higher SARI, but they rely on PPDB, and human raters still preferred EditNTS.
  • Raising one edit type's loss weight tenfold on Newsela changes output length: about 25.2 tokens when ADD is favoured versus about 15.8 when DELETE is favoured.
  • In the ablation on Newsela, removing the interpreter costs the most, with SARI falling from 31.41 to 30.13. Removing POS tags or attention context gives smaller drops. Code and data are available at github.com/yuedongP/EditNTS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

103 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 97 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1331

@inproceedings{dong2019editnts,
  title = {{EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing}},
  author = {Dong, Yue and Li, Zichao and Rezagholizadeh, Mehdi and Cheung, Jackie Chi Kit},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1331},
  url = {https://doi.org/10.18653/v1/p19-1331}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fernando Alva-Manchego, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017. Learning how to simplify from explicit labeling of complex-simplified text pairs. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 295–305. find our card
  2. Alexandre Bérard, Laurent Besacier, and Olivier Pietquin. 2017. Lig-cristal submission for the wmt 2017 automatic post-editing task. In Proceedings of the Second Conference on Machine Translation, pages 623–629. find
  3. John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying text for language-impaired readers. In Ninth Conference of the European Chapter of the Association for Computational Linguistics. find
  4. Richard Evans, Constantin Orasan, and Iustin Dornescu. 2014. An evaluation of syntactic simplification rules for people with autism. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), pages 131–140. find
  5. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393. ↗ our card
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. find
  7. David Kauchak. 2013. Improving text simplification language modeling using unsimplified text data. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (volume 1: Long papers), volume 1, pages 1537–1546. find
  8. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. ↗
  9. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  10. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 158–167. find
  11. Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 435–445. find
  12. Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P Dinu. 2017. Exploring neural text simplification models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 85–91. find
  13. Gustavo Paetzold, Fernando Alva-Manchego, and Lucia Specia. 2017. Massalign: Alignment and annotation of comparable documents. Proceedings of the IJCNLP 2017, System Demonstrations, pages 1–4. find
  14. Gustavo Paetzold and Lucia Specia. 2017. Lexical simplification with neural ranking. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 34–40. find
  15. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310–1318. find
  16. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  17. Scott Reed and Nando de Freitas. 2016. Neural programmer-interpreters. In Proceedings of International Conference on Learning Representations (ICLR). find
  18. Luz Rello, Clara Bayarri, Azuki Górriz, Ricardo Baeza-Yates, Saurabh Gupta, Gaurang Kanvinde, Horacio Saggion, Stefan Bott, Roberto Carlini, and Vasile Topac. 2013. Dyswebxia 2.0!: more accessible text for people with dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, page 25. Citeseer. find
  19. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  20. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. find our card
  21. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584–594. find our card
  22. Elior Sulem, Omri Abend, and Ari Rappoport. 2018a. Bleu is not suitable for the evaluation of text simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 738–744. find our card
  23. Yaoyuan Zhang, Zhenxu Ye, Yansong Feng, Dongyan Zhao, and Rui Yan. 2017. A constrained sequence-to-sequence neural model for sentence simplification. arXiv preprint arXiv:1704.02312. ↗
  24. Elior Sulem, Omri Abend, and Ari Rappoport. 2018b. Simple and effective text simplification using semantic and neural methods. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 162–173. find our card
  25. Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. 2018. Integrating transformer and paraphrase rules for sentence simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3164–3173. find
  26. Thuy-Trang Vu and Gholamreza Haffari. 2018. Automatic post-editing of machine translation: A neural programmer-interpreter approach. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3048–3053. find
  27. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1353–1361. Association for Computational Linguistics. find our card
  28. Tu Vu, Baotian Hu, Tsendsuren Munkhdalai, and Hong Yu. 2018. Sentence simplification with memory-augmented neural networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 79–85. find
  29. Willian Massami Watanabe, Arnaldo Candido Junior, Vinı́cius Rodriguez Uzêda, Renata Pontin de Mattos Fortes, Thiago Alexandre Salgueiro Pardo, and Sandra Maria Aluı́sio. 2009. Facilita: reading assistance for low-literacy readers. In Proceedings of the 27th ACM International Conference on Design of Communication, pages 29–36. ACM. find
  30. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 409–420. Association for Computational Linguistics. find our card
  31. Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 1015–1024. Association for Computational Linguistics. find our card
  32. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. find our card