Elamur Research⌕
☰
home / papers / 2018 / Integrating Transformer and Paraphrase Rules for Sentence Simplification

Integrating Transformer and Paraphrase Rules for Sentence Simplification

Summary

The paper addresses sentence simplification, which means rewriting a sentence so it is easier to read without changing its meaning. Neural encoder-decoder systems trained on aligned normal–simple sentence pairs had trailed statistical systems on this task. The authors see two causes: training corpora are too small, and neural models absorb frequent rewrites while dismissing rare ones as noise.

They make two changes. First, they replace RNN/LSTM with a Transformer (multi-layer, multi-head attention). Second, they inject knowledge from Simple PPDB, a large set of weighted normal-to-simple paraphrase rules, in two ways:

  • DCSS adds a critic-style loss. Whenever a rule applies, it raises the probability of the simpler phrase and lowers that of the original, scaled by the rule's weight.
  • DMASS adds an external memory that keeps each rule's context and output separately, so rare rules are not overwritten in shared parameters.

The combined DMASS+DCSS achieves the best SARI among the compared systems, including SBMT-SARI and DRESS. It also applies rules with both good recall and good precision. This shows that curated lexical resources can be fused effectively into neural text generation.

Key points

  • Simple PPDB, derived from PPDB, holds about 4.5 million rules. Each rule has a normal phrase, a simpler phrase, a syntactic type (e.g. [NN], [VP]) and a simplification weight.
  • The baselines include PBMT-R, Hybrid, SBMT-SARI, DRESS and DRESS-LS. Evaluation uses SARI (with separate add/keep/delete F1), FKGL, average word length (WLen), sentence length (SLen) and rule-utilization precision/recall.
  • DRESS-style models simplify mainly by shortening sentences. SBMT-SARI and the proposed models simplify mainly by choosing shorter words.
  • SBMT-SARI simplifies aggressively, which gives high recall but weaker precision. DMASS beats DCSS, and DMASS+DCSS is best overall.
  • Larger beam sizes correlate strongly with rule utilization but only loosely with SARI. Code and models are released at github.com/Sanqiang/text_simplification.

Interesting ideas

  • I1. Contrastive critic loss. The loss rewards the target-side replacement and penalizes the source-side word, weighted by an external confidence score. This is a cheap way to bias any seq2seq model toward a lexicon, for example for terminology or dialect normalization.
  • I2. External per-rule memory. Storing each rule in its own memory slot, rather than only in shared weights, protects long-tail knowledge from being treated as noise. This is useful wherever data is scarce and rare mappings matter, as in low-resource languages.
  • I3. Combining internal and external memory. The authors find that the two storage strategies complement each other rather than compete, so it is worth trying both together.
  • I4. Rule utilization as a metric. Scoring how often applicable rules are used (recall) and how often the applied ones are correct (precision) gives a more interpretable check than a single overall score.
  • I5. Decomposing readability scores. Splitting FKGL into word length and sentence length reveals how a system simplifies. It also flags cases where a good score may come from dropping information.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

73 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 64 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1355

@inproceedings{zhao2018integrating,
  title = {{Integrating Transformer and Paraphrase Rules for Sentence Simplification}},
  author = {Zhao, Sanqiang and Meng, Rui and He, Daqing and Saptono, Andi and Parmanto, Bambang},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1355},
  url = {https://doi.org/10.18653/v1/d18-1355}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John A Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying text for language-impaired readers. In EACL, pages 269–270. find our card
  2. Michał Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel. 2017. Frustratingly short attention spans in neural language modeling. arXiv preprint arXiv:1702.04521. ↗
  3. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159. find
  4. Yang Feng, Shiyue Zhang, Andi Zhang, Dong Wang, and Andrew Abel. 2017. Memory-augmented neural machine translation. arXiv preprint arXiv:1708.02005. ↗
  5. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In HLT-NAACL, pages 758–764. find our card
  6. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426. ↗
  7. David Kauchak. 2013. Improving text simplification language modeling using unsimplified text data. In ACL (1), pages 1537–1546. find our card
  8. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch. ↗
  9. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126. ↗
  10. Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In the 52nd Annual Meeting of the Association for Computational Linguistics, pages 435–445. find
  11. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310–1318. ↗
  12. Ellie Pavlick and Chris Callison-Burch. 2016. Simple ppdb: A paraphrase database for simplification. In The 54th Annual Meeting of the Association for Computational Linguistics, page 143. find
  13. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  14. Luz Rello, Clara Bayarri, Azuki Gòrriz, Ricardo Baeza-Yates, Saurabh Gupta, Gaurang Kanvinde, Horacio Saggion, Stefan Bott, Roberto Carlini, and Vasile Topac. 2013. Dyswebxia 2.0!: more accessible text for people with dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, page 25. ACM. find
  15. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929–1958. find
  16. Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2017. Learning to remember translation history with a continuous cache. arXiv preprint arXiv:1711.09367. ↗
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. ↗
  18. Willian Massami Watanabe, Arnaldo Candido Junior, Vinı́cius Rodriguez Uzêda, Renata Pontin de Mattos Fortes, Thiago Alexandre Salgueiro Pardo, and Sandra Maria Aluı́sio. 2009. Facilita: reading assistance for low-literacy readers. In Proceedings of the 27th ACM international conference on Design of communication, pages 29–36. ACM. find
  19. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of the conference on empirical methods in natural language processing, pages 409–420. Association for Computational Linguistics. find our card
  20. Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 1015–1024. Association for Computational Linguistics. find our card
  21. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015a. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057. ↗ our card
  22. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015b. Problems in current text simplification research: New data can help. Transactions of the Association of Computational Linguistics, 3(1):283– 297. ↗ our card
  23. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. ↗ our card
  24. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. arXiv preprint arXiv:1703.10931. ↗ our card
  25. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd international conference on computational linguistics, pages 1353–1361. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zhao-2018-integrating-transformer-paraphrase-rules-sentence/