Elamur Research
☰
home / papers / 2020 / Neural CRF Model for Sentence Alignment in Text Simplification

Neural CRF Model for Sentence Alignment in Text Simplification

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.709.pdf

Summary

Modern text simplification systems learn from pairs of complex and simplified sentences, and those pairs are only as good as the alignment between parallel articles. Earlier corpora such as Newsela and WikiLarge matched sentences with shallow lexical overlap, such as Jaccard or TF-IDF. That approach misses paraphrases and ignores context, so models trained on the data mostly delete words. The authors first built gold-standard alignment data by hand: NEWSELA-MANUAL and WIKI-MANUAL. They then designed an aligner based on a linear-chain CRF. It scores each sentence pair with a fine-tuned BERT similarity model and adds learned transition scores, which reward the fact that parallel documents tend to keep content in a similar order. Decoding uses Viterbi, after a paragraph-alignment step that narrows the search. The aligner beats prior monolingual methods by more than 5 F1 points. Running it on full collections produced NEWSELA-AUTO and WIKI-AUTO. A BERT-initialized Transformer trained on these new corpora reached state-of-the-art simplification quality in both automatic metrics and human ratings. It also rewrote sentences more actively.

Key points

  • Manual datasets: NEWSELA-MANUAL covers 50 article sets, and WIKI-MANUAL covers 500 article pairs. Both are annotated for training and evaluating sentence alignment.
  • Aligner design and accuracy: The neural CRF aligner outperforms CATS (Štajner et al., 2018), Paetzold et al. (2017) and Xu et al. (2015) by over 5 F1. Its transition features capture label distance and unaligned sentences.
  • Automatic corpora: NEWSELA-AUTO contains 666,645 sentence pairs, drawn from 1,882 article sets. WIKI-AUTO contains 488,332 pairs, drawn from 138,095 article pairs. These are roughly 4.7× and 1.6× larger than the earlier datasets for each domain.
  • Simplification results: The BERT-initialized Transformer improves on the previous state of the art by 3.4% SARI. On the Wikipedia TURK test set it scores 36.6 SARI, compared with 35.3 when trained on WikiLarge. Its outputs contain about 25% more rephrasing.
  • Availability: Code and WIKI-AUTO data are available at github.com/chaojiang06/wiki-auto. Newsela data must be requested from Newsela.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

124 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 122 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.709

@inproceedings{jiang2020neural,
  title = {{Neural CRF Model for Sentence Alignment in Text Simplification}},
  author = {Jiang, Chao and Maddela, Mounica and Lan, Wuwei and Zhong, Yang and Xu, Wei},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.709},
  url = {https://doi.org/10.18653/v1/2020.acl-main.709}
}

References

55 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmad Aghaebrahimian. 2018. Deep neural networks at the service of multilingual parallel sentence extraction. In Proceedings of the 27th International Conference on Computational Linguistics. find
  2. Fernando Alva-Manchego, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017. Learning how to simplify from explicit labeling of complex-simplified text pairs. In Proceedings of the Eighth International Joint Conference on Natural Language Processing. find our card
  3. Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics. find
  4. Regina Barzilay and Noemie Elhadad. 2003. Sentence alignment for monolingual comparable corpora. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing. find
  5. Ohio Supercomputer Center. 2012. Oakley supercomputer. http://osc.edu/ark:/19495/ hpc0cvqn . ↗
  6. R. Chandrasekar, Christine Doran, and B. Srinivas. 1996. Motivations and methods for text simplification. In The 16th International Conference on Computational Linguistics. find
  7. Han-Bin Chen, Hen-Hsen Huang, Hsin-Hsi Chen, and Ching-Ting Tan. 2012. A simplification-translation-restoration framework for cross-domain SMT applications. In Proceedings of the 24th International Conference on Computational Linguistics. find
  8. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. find
  9. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find our card
  10. William Coster and David Kauchak. 2011. Simple English Wikipedia: A new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. find
  12. Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find our card
  13. Noemie Elhadad and Komal Sutaria. 2007. Mining a lexicon of technical terms and lay equivalents. In Biological, translational, and clinical language processing. find
  14. Simon Gottschalk and Elena Demidova. 2017. Mul-tiwiki: interlingual text passage alignment in wikipedia. ACM Transactions on the Web. find
  15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation. find
  16. William Hwang, Hannaneh Hajishirzi, Mari Ostendorf, and Wei Wu. 2015. Aligning sentences from standard Wikipedia to simple Wikipedia. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics. find
  17. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. Montreal neural machine translation systems for WMT’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation. find
  18. Tomoyuki Kajiwara, Hiroshi Matsumoto, and Kazuhide Yamamoto. 2013. Selecting proper lexical paraphrase for children. In Proceedings of the 25th Conference on Computational Linguistics and Speech Processing. find
  19. Robert P. Jr.; Rogers Richard L.; Kincaid, J. Peter; Fishburne and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. research branch report. find
  20. Reno Kriz, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Callison-Burch. 2019. Complexity-weighted loss and diverse reranking for sentence simplification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. find
  21. Makoto Miwa, Rune Sætre, Yusuke Miyao, and Jun’ichi Tsujii. 2010. Entity-focused sentence simplification for relation extraction. In Proceedings of the 23rd International Conference on Computational Linguistics. find
  22. Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. find
  23. Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh. 2019. Transforming complex sentences into a semantic hierarchy. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  24. Sergiu Nisioi, Sanja Štajner, Simone Paolo Ponzetto, and Liviu P. Dinu. 2017. Exploring neural text simplification models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. find
  25. Gustavo Paetzold, Fernando Alva-Manchego, and Lucia Specia. 2017. MASSAlign: Alignment and annotation of comparable documents. In Proceedings of the IJCNLP 2017, System Demonstrations. find
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. find our card
  27. David Pellow and Maxine Eskenazi. 2014. An open corpus of everyday documents for simplification tasks. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations. find
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing. find
  29. Sarah E Petersen and Mari Ostendorf. 2007. Text simplification for language learners: A corpus analysis. In Proceedings of Workshop on Speech and Language Technology for Education. find
  30. Scott E. Reed and Nando de Freitas. 2016. Neural programmer-interpreters. In 4th International Conference on Learning Representations. find
  31. Luz Rello, Ricardo Baeza-Yates, and Horacio Saggion. 2013. The impact of lexical simplification by verbal paraphrases for people with and without dyslexia. In Proceedings of the 14th International Conference on Computational Linguistics and Intelligent Text Processing. find
  32. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics. find our card
  33. Jason R. Smith, Chris Quirk, and Kristina Toutanova. 2010. Extracting parallel sentences from comparable corpora using document level alignment. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  34. Sanja Štajner, Hannah Béchara, and Horacio Saggion. 2015. A deeper exploration of the standard PB-SMT approach to text simplification and its evaluation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. find
  35. Sanja Štajner, Marc Franco-Salvador, Paolo Rosso, and Simone Paolo Ponzetto. 2018. CATS: A tool for customized alignment of text simplification corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation. find
  36. Sanja Štajner and Maja Popovic. 2016. Can text simplification help machine translation? In Proceedings of the 19th Annual Conference of the European Association for Machine Translation. find
  37. Brian Thompson and Philipp Koehn. 2019. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. find
  38. Chen-Tse Tsai and Dan Roth. 2016. Cross-lingual wikification using multilingual embeddings. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics. find
  39. Dan Tufis , , Radu Ion, S , tefan Dumitrescu, and Dan S , tefănescu. 2013. Wikipedia as an SMT training corpus. In Proceedings of the International Conference Recent Advances in Natural Language Processing. find
  40. Lucy Vanderwende, Hisami Suzuki, Chris Brockett, and Ani Nenkova. 2007. Beyond sumbasic: Task-focused summarization with sentence simplification and lexical expansion. Inf. Process. Manage. find
  41. Horacio Saggion. 2017. Automatic text simplification. Synthesis Lectures on Human Language Technologies. find
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. find
  43. Advaith Siddharthan. 2006. Syntactic simplification and text cohesion. Research on Language and Computation. find
  44. David Vickrey and Daphne Koller. 2008. Sentence simplification for semantic role labeling. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics. find
  45. Advaith Siddharthan and Napoleon Katsos. 2010. Reformulating discourse connectives for non-expert readers. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
  46. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗ our card
  47. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. find our card
  48. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. find our card
  49. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics. find our card
  50. Wei Xu and Ralph Grishman. 2009. A parse-and-trim approach with information significance for Chinese sentence compression. In Proceedings of the 2009 Workshop on Language Generation and Summarisation. find
  51. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics. find our card
  52. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. find
  53. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find our card
  54. Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. 2018. Integrating transformer and paraphrase rules for sentence simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
  55. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd International Conference on Computational Linguistics. find our card