Elamur Research
☰
home / papers / 2015 / Problems in Current Text Simplification Research: New Data Can Help

Problems in Current Text Simplification Research: New Data Can Help

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q15-1021.pdf

Summary

The authors contend that text simplification research has relied too heavily on Simple English Wikipedia, especially the Parallel Wikipedia Simplification (PWKP) corpus of Zhu et al. (2010), which became the default benchmark. In this opinion paper, they combine manual inspection, statistical corpus analysis and evidence gathered from earlier publications. They show that PWKP has three main weaknesses: its automatic sentence alignment is unreliable, many of its pairs are not real simplifications, and it transfers poorly to other genres. The authors attribute these weaknesses to how Simple Wikipedia was produced, with volunteers writing without clear guidelines or a defined audience. As an alternative, they introduce the Newsela corpus, in which professional editors rewrote news articles for specific school grade levels. Comparing Newsela with PWKP exposes the flaws of the Wikipedia data more clearly. The authors recommend retiring PWKP as the standard benchmark. They also propose better evaluation practices: designing for specific reader groups and assessing splitting, deletion and paraphrasing as separate sub-tasks. These changes matter because both training data and evaluation methods determine how much real progress simplification systems can show.

Key points

  • PWKP contains about 108,000 automatically aligned Normal–Simple Wikipedia sentence pairs. A 100-sentence test set drawn from it served as the usual benchmark for comparing systems.
  • In a manual check of 200 sampled PWKP pairs:
    • 17% were misaligned.
    • 33% kept the meaning but were not simpler.
    • Only 50% were genuine simplifications: 21% by deletion only, 17% by paraphrase only, and 12% by both.
  • The simple side of PWKP still has a large vocabulary of 78,009 unique words, and 6,669 of these do not appear on the normal side. Even a state-of-the-art aligner (Hwang et al., 2015) reaches only 0.712 max F1.
  • The Newsela corpus contains 1,130 news articles, each rewritten four times (Simp-1 to Simp-4). The rewrites are grounded in Lexile readability scores and the US Common Core Standards, and the data is available on request via newsela.com/data.
  • On evaluation, the authors argue that 5-point Likert ratings of grammaticality, simplicity and adequacy are insufficient. They call for audience-specific evaluation and separate metrics for each sub-task, partly because adequacy ratings tend to unfairly penalize deletion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

290 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 270 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00139

@article{xu2015problems,
  title = {{Problems in Current Text Simplification Research: New Data Can Help}},
  author = {Xu, Wei and Callison-Burch, Chris and Napoles, Courtney},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.1162/tacl\_a\_00139},
  url = {https://doi.org/10.1162/tacl_a_00139}
}

References

60 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allen, D. (2009). A study of the role of relative clauses in the simplification of news texts for learners of English. System, 37(4):585–599. find
  2. Angrosh, M., Nomoto, T., and Siddharthan, A. (2014). Lexico-syntactic text simplification and compression with typed dependencies. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL). find
  3. Bach, N., Gao, Q., Vogel, S., and Waibel, A. (2011). Tris: A statistical sentence simplifier with log-linear models and margin-based discriminative training. In Proceedings of 5th International Joint Conference on Natural Language Processing (IJCNLP). find
  4. Barzilay, R. and Lapata, M. (2008). Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34. find our card
  5. Callison-Burch, C., Cohn, T., and Lapata, M. (2008). ParaMetric: An automatic evaluation metric for paraphrasing. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING). find
  6. Canning, Y., Tait, J., Archibald, J., and Crawley, R. (2000). Cohesive generation of syntactically simplified newspaper text. In Proceedings of the Third International Workshop on Text, Speech and Dialogue (TSD). find
  7. Carroll, J., Minnen, G., Pearce, D., Canning, Y., Devlin, S., and Tait, J. (1999). Simplifying text for language-impaired readers. In Proceedings of the 14th Conference of the 9th European Conference for Computational Linguistics (EACL). find
  8. Chandrasekar, R., Doran, C., and Srinivas, B. (1996). Motivations and methods for text simplification. In Proceedings of the 16th Conference on Computational linguistics (COLING). find
  9. Chen, D. L. and Dolan, W. B. (2011). Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL). find
  10. Chen, H.-B., Huang, H.-H., Chen, H.-H., and Tan, C.-T. (2012). A simplification-translation-restoration framework for cross-domain smt applications. In Proceedings of the 24th International Conference on Computational Linguistics (COLING). find
  11. Collins, M. (2003). Head-driven statistical models for natural language parsing. Computational linguistics, 29(4):589–637. find our card
  12. Coster, W. and Kauchak, D. (2011). Simple English Wikipedia: A new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). find
  13. De Belder, J. and Moens, M.-F. (2010). Text simplification for children. In Prroceedings of the SIGIR Workshop on Accessible Search Systems. find
  14. Eisenstein, J. (2013). What to do about bad language on the Internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
  15. Elhadad, N. and Sutaria, K. (2007). Mining a lexicon of technical terms and lay equivalents. In Proceedings of the Workshop on BioNLP 2007: Biological, Translational, and Clinical Language Processing. find
  16. Feng, L. (2008). Text simplification: A survey. Technical report, The City University of New York. find
  17. Filippova, K. and Strube, M. (2008). Dependency tree based sentence compression. In Proceedings of the 5th International Natural Language Generation Conference (INLG). find
  18. Ganitkevitch, J., Callison-Burch, C., Napoles, C., and Van Durme, B. (2011). Learning sentential paraphrases from bilingual parallel corpora for text-to-text generation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  19. Gasperin, C., Maziero, E., Specia, L., Pardo, T., and Aluisio, S. M. (2009). Natural language processing for social inclusion: A text simplification architecture for different literacy levels. In Proceedings of SEMISH-XXXVI Seminário Integrado de Software e Hardware. find
  20. Gerber, L. and Hovy, E. (1998). Improving translation quality by manipulating sentence length. In Machine Translation and the Information Soup, pages 448–460. Springer. find
  21. Hwang, W., Hajishirzi, H., Ostendorf, M., and Wu, W. (2015). Aligning sentences from Standard Wikipedia to Simple Wikipedia. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  22. Inui, K., Fujita, A., Takahashi, T., Iida, R., and Iwakura, T. (2003). Text simplification for reading assistance: A project note. In Proceedings of the 2nd International Workshop on Paraphrasing. find
  23. Jaccard, P. (1912). The distribution of the flora in the alpine zone. New Phytologist, 11(2):37–50. find
  24. Jurafsky, D., Chahuneau, V., Routledge, B. R., and Smith, N. A. (2014). Narrative framing of consumer sentiment in online restaurant reviews. First Monday, 19(4). find
  25. Kauchak, D. (2013). Improving text simplification language modeling using unsimplified text data. In Proceedings of the 2013 Conference of the Association for Computational Linguistics (ACL). find
  26. Kincaid, J. P., Fishburne Jr, R. P., Rogers, R. L., and Chissom, B. S. (1975). Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, DTIC Document. find
  27. Klebanov, B. B., Knight, K., and Marcu, D. (2004). Text simplification for information-seeking applications. In On the Move to Meaningful Internet Systems 2004: CoopIS, DOA, and ODBASE, pages 735–747. Springer. find
  28. Klein, D. and Manning, C. D. (2002). Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems. find
  29. Louis, A. and Nenkova, A. (2013). What makes writing great? First experiments on article quality prediction in the science journalism domain. Transactions of the Association for Computational Linguistics (TACL), 1:341–352. find
  30. Miwa, M., Saetre, R., Miyao, Y., and Tsujii, J. (2010). Entity-focused sentence simplification for relation extraction. In Proceedings of the 24th International Conference on Computational Linguistics (COLING). find
  31. Monroe, B. L., Colaresi, M. P., and Quinn, K. M. (2008). Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403. find
  32. Moraes, P., McCoy, K., and Carberry, S. (2014). Adapting graph summaries to the users’ reading levels. In Proceedings of the 8th International Natural Language Generation Conference (INLG). find
  33. Napoles, C., Callison-Burch, C., and Van Durme, B. (2011). Evaluating sentence compression: Pitfalls and suggested remedies. In Proceedings of the Workshop on Monolingual Text-To-Text Generation. find
  34. Narayan, S. and Gardent, C. (2014). Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL). find
  35. Petersen, S. and Ostendorf, M. (2007). Text simplification for language learners: A corpus analysis. In Proceedings of the Workshop on Speech and Language Technology for Education. find
  36. Petersen, S. E. and Ostendorf, M. (2009). A machine learning approach to reading level assessment. Computer Speech & Language, 23(1):89– 106. find
  37. Pitler, E. and Nenkova, A. (2008). Revisiting readability: A unified framework for predicting text quality. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  38. Porter, A., McMaken, J., Hwang, J., and Yang, R. (2011). Common Core Standards the new US intended curriculum. Educational Researcher, 40(3):103–116. find
  39. Post, M. (2011). Judging grammaticality with tree substitution grammar derivations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). find
  40. Riezler, S., King, T. H., Crouch, R., and Zaenen, A. (2003). Statistical sentence condensation using ambiguity packing and stochastic disambiguation methods for lexical-functional grammar. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technology (NAACL-HLT). find
  41. Scarton, C., De Oliveira, M., Candido Jr, A., Gasperin, C., and Aluísio, S. M. (2010). Simplifica: A tool for authoring simplified texts in brazilian portuguese guided by readability assessments. In Proceedings of the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
  42. Siddharthan, A. (2003). Preserving discourse structure when simplifying text. In Proceedings of European Workshop on Natural Language Generation (ENLG). find
  43. Siddharthan, A. (2006). Syntactic simplification and text cohesion. Research on Language and Computation, 4(1):77–109. find
  44. Siddharthan, A. (2014). A survey of research on text simplification. Special issue of International Journal of Applied Linguistics, 165(2). find
  45. Siddharthan, A. and Angrosh, M. (2014). Hybrid text simplification using synchronous dependency grammars with hand-written and automatically harvested rules. In Proceedings of the 25th International Conference on Computational Linguistics (COLING). find our card
  46. Siddharthan, A. and Copestake, A. (2002). Generating anaphora for simplifying text. In Proceedings of the 4th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC). find
  47. Siddharthan, A. and Katsos, N. (2010). Reformulating discourse connectives for non-expert readers. In Proceedings of the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
  48. Siddharthan, A., Nenkova, A., and McKeown, K. (2004). Syntactic simplification for improving content selection in multi-document summarization. In Proceedings of the 20th International Conference on Computational Linguistics (COL-ING). find
  49. Vanderwende, L., Suzuki, H., Brockett, C., and Nenkova, A. (2007). Beyond SumBasic: Task-focused summarization with sentence simplification and lexical expansion. Information Processing & Management, 43(6):1606–1618. find
  50. Vickrey, D. and Koller, D. (2008). Sentence simplification for semantic role labeling. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). find
  51. Watanabe, W. M., Junior, A. C., Uzêda, V. R., Fortes, R. P. d. M., Pardo, T. A. S., and Aluísio, S. M. (2009). Facilita: reading assistance for low-literacy readers. In Proceedings of the 27th ACM International Conference on Design of Communication. find
  52. Weese, J., Ganitkevitch, J., and Callison-Burch, C. (2014). PARADIGM: Paraphrase diagnostics through grammar matching. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL). find
  53. Williams, S., Reiter, E., and Osman, L. (2003). Experiments with discourse-level choices and readability. In Proceedings of the European Natural Language Generation Workshop (ENLG). find
  54. Woodsend, K. and Lapata, M. (2011a). Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  55. Woodsend, K. and Lapata, M. (2011b). WikiSimple: Automatic simplification of Wikipedia articles. In Proceedings of the 25th Conference on Artificial Intelligence (AAAI). find
  56. Wubben, S., van den Bosch, A., and Krahmer, E. (2012). Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  57. Xu, W. and Grishman, R. (2009). A parse-and-trim approach with information significance for chinese sentence compression. In Proceedings of the 2009 Workshop on Language Generation and Summarisation. find
  58. Xu, W., Ritter, A., Dolan, B., Grishman, R., and Cherry, C. (2012). Paraphrasing for style. In Proceedings of the 24th International Conference on Computational Linguistics (COLING). find our card
  59. Xu, W., Ritter, A., and Grishman, R. (2013). Gathering and generating paraphrases from twitter with application to normalization. In Proceedings of the Sixth Workshop on Building and Using Comparable Corpora (BUCC). find
  60. Zhu, Z., Bernhard, D., and Gurevych, I. (2010). A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING). find our card