Elamur Research⌕
☰
home / papers / 2013 / Improving Text Simplification Language Modeling Using Unsimplified Text Data

Improving Text Simplification Language Modeling Using Unsimplified Text Data

Summary

This paper asks whether a language model for simplified English can benefit from ordinary, unsimplified English. Simplification is a monolingual translation task, so data from the input side is also English, but simple text is scarce. The author built a document-aligned corpus of 60K paired articles, each matching a Simple English Wikipedia entry with its regular English Wikipedia counterpart, and trained trigram models with interpolated Kneser-Ney smoothing in SRILM. The models used simple-only data, normal-only data, or mixtures of the two, combined either by concatenating the corpora or by linear interpolation. They were evaluated by perplexity on held-out simple text and on the SemEval 2012 lexical simplification task. Adding normal data improved results over a model trained on all available simple data: perplexity dropped by 23% and lexical simplification improved by 24%. Further analysis showed that normal text mainly helps by covering rare and unseen n-grams. It remains out-of-domain, however, so simple data is worth much more per sentence. This matters for any low-resource monolingual generation task where in-domain target text is limited.

Key points

  • Corpus: 60K aligned article pairs, with 2K pairs held out for testing. The simple side has 385K sentences, 7.15M words and a 78K vocabulary. The normal side has 2,540K sentences, 64.7M words and a 307K vocabulary.
  • The test set has 7,799 simple sentences (179K words). All models share a closed vocabulary taken from the simple training side.
  • In a 137K sentence-aligned set, 96% of simple unigrams appear in normal text. For 5-grams the overlap is only about 55% (simple in normal) and 46% (normal in simple).
  • Perplexity fell from 129 (simple-only) to 100 (all simple + normal). The normal-only model slightly beat the simple-only model, but only by using about ten times more data.
  • Simple data is more valuable per sentence. To match 250K simple sentences, a model needs 200K simple plus 300K normal, or 100K simple plus 850K normal. The best interpolation weight on simple was 0.5 for perplexity and 0.98 for lexical simplification.

Interesting ideas

  • I1. Train the LM on input-side text too. In monolingual tasks such as compression, simplification or dialect normalization, abundant source-side text can act as a backoff or smoothing resource for a small target-style corpus.
  • I2. Measure n-gram overlap in both directions. Asymmetric overlap between two text varieties is a cheap diagnostic. It shows whether one variety is more lexically diverse and whether extra data is likely to fill coverage gaps.
  • I3. Use a fixed shared vocabulary. Fixing one closed vocabulary across all compared models keeps perplexity comparisons fair when training corpora differ in size and source.
  • I4. Tune the mixing weight per task. The best interpolation weight differed sharply between intrinsic perplexity and the downstream task. Tuning it on the target task avoids misleading conclusions drawn from perplexity alone.
  • I5. Separate content effects from style effects. Training on content-matched, sentence-aligned pairs shows that the out-of-domain penalty comes from different language use, not topic. The same control could isolate register effects in other corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

99 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kauchak2013improving,
  title = {{Improving Text Simplification Language Modeling Using Unsimplified Text Data}},
  author = {Kauchak, David},
  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2013},
  url = {https://aclanthology.org/P13-1151/}
}

References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michiel Bacchiani and Brian Roark. 2003. Unsupervised language model adaptation. In Proceedings of ICASSP. find
  2. Michele Banko, Vibhu Mittal, and Michael Witbrock. 2000. Headline generation based on statistical translation. In Proceedings of ACL. find
  3. Jerome R. Bellegarda. 2004. Statistical language model adaptation: Review and perspectives. Speech Communication. ↗
  4. Or Biran, Samuel Brody, and Noeḿie Elhadad. 2011. Putting it simply: A context-aware approach to lexical simplification. In Proceedings of ACL. ↗ our card
  5. Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language models in machine translation. In Proceedings of EMNLP. find our card
  6. Raman Chandrasekar and Bangalore Srinivas. 1997. Automatic induction of rules for text simplification. Knowledge Based Systems. ↗
  7. Stanley Chen, Douglas Beeferman, and Ronald Rosenfeld. 1998. Evaluation metrics for language models. In DARPA Broadcast News Transcription and Understanding Workshop. find
  8. Trevor Cohn and Mirella Lapata. 2009. Sentence compression as tree transduction. Journal of Artificial Intelligence Research. ↗
  9. William Coster and David Kauchak. 2011a. Learning to simplify sentences using Wikipedia. In Proceedings of Text-To-Text Generation. find our card
  10. William Coster and David Kauchak. 2011b. Simple English Wikipedia: A new text simplification task. In Proceedings of ACL. find our card
  11. Hal Daume and Daniel Marcu. 2002. A noisy-channel model for document compression. In Proceedings of ACL. ↗
  12. Christopher Cieri David Graff. 2003. English gigaword. http://www.ldc.upenn.edu/Catalog/CatalogEntry. jsp?catalogId=LDC2003T05. ↗
  13. Carsten Eickhoff, Pavel Serdyukov, and Arjen P. de Vries. 2010. Web page classification on child suitability. In Proceedings of CIKM. find
  14. Michel Galley and Kathleen McKeown. 2007. Lexicalized Markov grammars for sentence compression. In Proceedings of HLT-NAACL. ↗
  15. Le Quan Ha, E. I. Sicilia-Garcia, Ji Ming, and F. J. Smith. 2003. Extension of Zipf’s law to word and character n-grams for English and Chinese. Computational Linguistics and Chinese Language Processing. find
  16. Bo-June Hsu. 2007. Generalized linear interpolation of language models. In IEEE Workshop on ASRU. find
  17. Frederick Jelinek and Robert Mercer. 1980. Interpolated estimation of markov source parameters from sparse data. In Proceedings of the Workshop on Patter Recognition in Practice. find
  18. Kevin Knight and Daniel Marcu. 2002. Summarization beyond sentence extraction: a probabilistic approach to sentence compression. Artificial Intelligence. ↗
  19. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics. ↗
  20. Gondy Leroy, James E. Endicott, Obay Mouradi, David Kauchak, and Melissa Just. 2012. Improving perceived and actual text difficulty for health information consumers using semi-automated methods. In American Medical Informatics Association (AMIA) Fall Symposium. find
  21. Courtney Napoles and Mark Dredze. 2010. Learning simple Wikipedia: A cogitation in ascertaining abecedarian language. In Proceedings of HLT/NAACL Workshop on Computation Linguistics and Writing. find
  22. Tadashi Nomoto. 2009. A comparison of model free versus model intensive approaches to sentence compression. In Proceedings of EMNLP. find
  23. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. ↗
  24. Ronald Rosenfeld. 1996. A maximum entropy approach to adaptive statistical language modeling. Computer, Speech and Language. find
  25. Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea. 2012. Semeval-2012 task 1: English lexical simplification. In Joint Conference on Lexical and Computerational Semantics (*SEM). find our card
  26. Lucia Specia. 2010. Translating from complex to simplified sentences. In Proceedings of Computational Processing of the Portuguese Language. find
  27. Andreas Stolcke. 2002. SRILM - An extensible language modeling toolkit. In Proceedings of ICSLP. find
  28. Hisami Suzuki and Jianfeng Gao. 2005. A comparative study on language model adaptation techniques. In Proceedings of EMNLP. find
  29. Jenine Turner and Eugene Charniak. 2005. Supervised and unsupervised learning for sentence compression. In Proceedings of ACL. find
  30. Ken Urano. 2000. Lexical simplification and elaboration: Sentence comprehension and incidental vocabulary acquisition. Master’s thesis, University of Hawaii. find
  31. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of EMNLP. find our card
  32. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of ACL. find our card
  33. Mark Yatskar, Bo Pang, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2010. For the sake of simplicity: Unsupervised extraction of lexical simplifications from Wikipedia. In Proceedings of NAACL. ↗
  34. Bing Zhao, Matthias Eck, and Stephan Vogel. 2004. Language model adaptation for statistical machine translation with structured query models. In Proceedings of COLING. ↗
  35. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of ICCL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/kauchak-2013-improving-text-simplification-language-modeling/