Elamur Research⌕
☰
home / papers / 2018 / OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification

OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification

Summary

Research on automatic readability assessment (ARA) and automatic text simplification (ATS) lacks open corpora in which the same content appears at several difficulty levels. Most ARA corpora group unrelated texts by grade. ATS work depends heavily on Wikipedia–Simple Wikipedia sentence pairs, which makes it hard to tell whether models generalize. The authors release the OneStopEnglish corpus, built from the weekly news lessons of onestopenglish.com. On that site, teachers rewrite Guardian articles for adult ESL learners at elementary, intermediate and advanced levels. The material was collected over 2013–2016 and converted from PDF. One author then checked every file by hand for missing content. Sentence pairs between levels were derived with cosine similarity. In demonstration experiments, a three-class text classifier reached about 78% accuracy with combined linguistic features, only slightly above character n-grams alone. A phrase-based MT simplifier reached a BLEU of 54.45. The resource is free and parallel at both text and sentence level. It targets L2 readers and offers a second benchmark for simplification beyond Wikipedia.

Key points

  • The corpus has 189 articles, each in three versions (567 texts). It is released under CC BY-SA 4.0 via Zenodo, with permission from onestopenglish.com and The Guardian.
  • Mean text length rises from about 533 words (elementary) to 677 (intermediate) and 820 (advanced). Standard deviations are large, so length alone does not determine level. FKGL drops from 9.5 (ADV) to 6.4 (ELE).
  • Sentence alignment kept pairs with cosine similarity between 0.7 and 0.95. This produced 1674 ELE-INT, 2166 ELE-ADV and 3154 INT-ADV pairs.
  • Classification used an SMO classifier with a linear kernel, with features extracted in LightSide; the random baseline is 33%.
    • Character n-grams scored 77.25%.
    • All 155 complexity features together scored 78.13%.
    • Dependency relations scored only 27.16%, below the random baseline.
  • For simplification, Moses PBMT was trained on the ADV-ELE pairs (1000 train / 500 dev / 666 test). It scored BLEU 54.45 and METEOR 46.

Interesting ideas

  • I1. Upper similarity bound for alignment. Capping similarity at 0.95 removes near-identical pairs, such as those differing only in punctuation. Any parallel-corpus builder can use this to avoid filling data with trivial "simplifications."
  • I2. Character n-grams for readability. In parallel texts, word overlap across levels weakens word n-grams. Sub-word patterns such as affixes still seem to separate the levels, making character n-grams a cheap and strong baseline worth trying for morphologically rich languages.
  • I3. Controlling for topic. Because the three versions share content, a classifier must rely on style rather than topic. This gives a harder but more honest test of whether features really capture difficulty.
  • I4. Small parallel sets as generalization tests. A corpus too small to train a strong simplifier can still check whether models trained elsewhere transfer, or it can be added to larger training data.
  • I5. From local to global judgments. The authors suggest ranking sentence or paragraph pairs by difficulty and aggregating these judgments into a text-level score. This could yield finer-grained difficulty feedback than a single label per document.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

83 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-0535

@inproceedings{vajjala2018onestopenglish,
  title = {{OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification}},
  author = {Vajjala, Sowmya and Lučić, Ivana},
  booktitle = {Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2018},
  doi = {10.18653/v1/w18-0535},
  url = {https://doi.org/10.18653/v1/w18-0535}
}

References

48 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Scott A. Crossley, Hae Sung Yang, and Danielle S. Mc-Namara. 2014. What’s so simple about simplified texts? a computational and psycholinguistic investigation of text comprehension and text processing. Reading in a Foreign Language, 26(1):92–113. ↗
  2. David Allen. 2009. A study of the role of relative clauses in the simplification of news texts for learners of English. System, 37(4):58–599. ↗
  3. Bharat Ram Ambati, Siva Reddy, and Mark Steedman. 2016. Assessing relative sentence complexity using an incremental ccg parser. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1051– 1057, San Diego, California. Association for Computational Linguistics. find
  4. Felice Dell’Orletta, Simonetta Montemagni, and Giulia Venturi. 2011. Read-it: Assessing readability of Italian texts with a view to text simplification. In Proceedings of the 2nd Workshop on Speech and Language Processing for Assistive Technologies, pages 73–83. find our card
  5. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376–380. find our card
  6. Karin Berendes, Sowmya Vajjala, Detmar Meurers, Doreen Bryant, Wolfgang Wagner, Maria Chinkina, and Ulrich Trautwein. 2017. Reading demands in secondary school: Does the linguistic complexity of textbooks increase with grade level and the academic orientation of the school track? Journal of Educational Psychology. ↗
  7. Lijun Feng. 2010. Automatic Readability Assessment. Ph.D. thesis, City University of New York (CUNY). find
  8. Itziar Gonzalez-Dios, Marı́a Jesús Aranzabe, Arantza Dı́az de Ilarraza, and Haritz Salaberri. 2014. Simple or complex? assessing the readability of basque texts. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 334–344, Dublin, Ireland. Dublin City University and Association for Computational Linguistics. find
  9. S Bott and H Saggion. 2011. An unsupervised alignment algorithm for text simplification corpus construction. In ACL Workshop on Monolingual Text-to-Text Generation. find
  10. Tim Vor der Brück, Hermann Helbig, and Johannes Leveling. 2008. The readability checker delite. Technical Report Technical Report 345-5/2008, Fakultät für Mathematik und Informatik, FernUni-versität in Hagen. find
  11. Arthur C. Graesser, Danielle S. McNamara, Zhiqang Cai, Mark Conley, Haiying Li, and James Pennebaker. 2014. Coh-metrix measures text characteristics at multiple levels of language and discourse. The Elementary School Journal, 115(2):pp. 210– 229. ↗
  12. Dominique Brunato, Andrea Cimino, Felice Dell’Orletta, and Giulia Venturi. 2016. Paccss-it: A parallel corpus of complex-simple sentences for automatic text simplification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 351–361. find
  13. Julia Hancke, Detmar Meurers, and Sowmya Vajjala. 2012. Readability classification for German using lexical, syntactic, and morphological features. In Proceedings of the 24th International Conference on Computational Linguistics (COLING), pages 1063– 1080, Mumbay, India. find
  14. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar. Association for Computational Linguistics. find our card
  15. Michael Heilman, Kevyn Collins-Thompson, and Maxine Eskenazi. 2008. An analysis of statistical models and features for reading difficulty prediction. In Proceedings of the 3rd Workshop on Innovative Use of NLP for Building Educational Applications at ACL-08, Columbus, Ohio. find
  16. Orphée De Clercq, Véronique Hoste, Bart Desmet, Philip Van Oosten, Martine De Cock, and Lieve Macken. 2014. Using the crowd for readability prediction. Natural Language Engineering, -:1–33. find
  17. Hieu Hoang, Alexandra Birch, Chris Callison-burch, Richard Zens, Rwth Aachen, Alexandra Constantin, Marcello Federico, Nicola Bertoldi, Chris Dyer, Brooke Cowan, Wade Shen, Christine Moran, and Ondřej Bojar. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45nd Annual Meeting of the Association for Computational Linguistics (ACL ’07). find
  18. Kevyn Collins-Thompson. 2014. Computational assessment of text readability: A survey of past, present, and future research. International Journal of Applied Linguistics, 165(2):97–135. find
  19. William Coster and David Kauchak. 2011. Simple English wikipedia: A new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 665–669, Portland, Oregon, USA. Association for Computational Linguistics. find our card
  20. David M Howcroft and Vera Demberg. 2017. Psycholinguistic models of sentence processing improve sentence readability ranking. In Proceedings of the 15th Conference of the European Chapter of J. Nelson, C. Perfetti, D. Liben, and M. Liben. 2012. Measures of text difficulty: Testing their predictive value for grade levels and student performance. Technical report, The Council of Chief State School Officers. the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pages 958–968. ↗
  21. William Hwang, Hannaneh Hajishirzi, Mari Ostendorf, and Wei Wu. 2015. Aligning sentences from standard wikipedia to simple wikipedia. In Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 211–217. find
  22. Philip van Oosten and Véronique Hoste. 2011. Readability annotation: replacing the expert by the crowd. In Proceedings of the 6th Workshop on Innovative Use of NLP for Building Educational Applications, IUNLPBEA ’11, pages 120–129, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  23. Zahurul Islam, Alexander Mehler, and Rashedur Rahman. 2012. Text readability classification of textbooks of a low-resource language. In 26th Pacific Asia Conference on Language,Information and Computation pages. find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: A method for automatic evaluation of machine translation. Technical report, IBM Research Division, Thomas J. Watson Research Center. find our card
  25. J. P. Kincaid, R. P. Jr. Fishburne, R. L. Rogers, and B. S Chissom. 1975. Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease formula) for Navy enlisted personnel. Research Branch Report 8-75, Naval Technical Training Command, Millington, TN. ↗
  26. Sarah E. Petersen. 2007. Natural Language Processing Tools for Reading Level Assessment and Text Simplification for Bilingual Education. Ph.D. thesis, University of Washington. find
  27. David Klaper, Sarah Ebling, and Martin Volk. 2013. Building a german/simple German parallel corpus for automatic text simplification. In Proceedings of the Second Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR). find
  28. Sarah E. Petersen and Mari Ostendorf. 2009. A machine learning approach to reading level assessment. Computer Speech and Language, 23:86–106. ↗
  29. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 13–16. Association for Computational Linguistics. find our card
  30. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Meeting of the Association for Computational Linguistics (ACL 2003), pages 423–430, Sapporo, Japan. find our card
  31. Advaith Siddharthan. 2014. A survey of research on text simplification. International Journal of Applied Linguistics: Special issue on Recent Advances in Automatic Readability Assessment and Text Simplification, 165:2:259–298. ↗
  32. Sigrid Klerke and Anders Søgaard. 2012. Danish parallel corpus for text simplification. In In Proceedings of Language Resources and Evaluation Conference (LREC), 2012. find
  33. Alon Lavie. 2011. Evaluating the output of machine translation systems. MT Summit Tutorial, page 86. find
  34. K. Toutanova, D. Klein, C. Manning, and Y. Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In HLT-NAACL, pages 252–259, Edmonton, Canada. find
  35. Yi Ma, Eric Fosler-Lussier, and Robert Lofthus. 2012. Ranking-based readability assessment for early primary children’s literature. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT ’12, pages 548–552, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  36. Sowmya Vajjala. 2015. Analyzing Text Complexity and Text Simplification: Connecting Linguistics, Processing and Educational Applications. Ph.D. thesis, University of Tübingen. ↗
  37. Sowmya Vajjala and Detmar Meurers. 2012. On improving the accuracy of readability classification using insights from second language acquisition. In In Proceedings of the 7th Workshop on Innovative Use of NLP for Building Educational Applications, pages 163–173, Montréal, Canada. Association for Computational Linguistics. find
  38. Elijah Mayfield and Carolyn Penstein Rosé. 2013. Open source machine learning for text. Handbook of automated essay evaluation: Current applications and new directions. find
  39. Courtney Napoles and Mark Dredze. 2010. Learning simple wikipedia: a cogitation in ascertaining abecedarian language. In Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics and Writing: Writing Processes and Authoring Aids, CL&W ’10, pages 42–50, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  40. Sowmya Vajjala and Detmar Meurers. 2013. On the applicability of readability models to web texts. In Proceedings of the Second Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 59–68. find
  41. Sowmya Vajjala and Detmar Meurers. 2014. Readability assessment for text simplification: From analyzing documents to identifying sentential simplifications. International Journal of Applied Linguistics, Special Issue on Current Research in Readability and Text Simplification, 165(2):142–222. find our card
  42. Sowmya Vajjala and Detmar Meurers. 2016. Readability-based sentence ranking for evaluating text simplification. arXiv preprint arXiv:1603.06009. ↗
  43. Sowmya Vajjala, Detmar Meurers, Alexander Eitel, and Katharina Scheiter. 2016. Towards grounding computational linguistic approaches to readability: Modeling reader-text interaction for easy and difficult texts. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), pages 38–48, Osaka, Japan. find
  44. The COL-ING 2016 Organizing Committee. find
  45. Mabel Vogel and Carleton Washburne. 1928. An objective method of determining grade placement of children’s reading material. Elementary School Journal, 38:58–66. ↗
  46. Carleton Washburne and Mabel Vogel. 1926. Winnetka graded book list, volume 1. American Library Association. find
  47. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584–594. find our card
  48. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of The 23rd International Conference on Computational Linguistics (COLING), August 2010. Beijing, China, pages 1353–1361. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/vajjala-2018-onestopenglish-corpus-new-corpus-automatic/