Elamur Research⌕
☰
home / papers / 2016 / Text Readability Assessment for Second Language Learners

Text Readability Assessment for Second Language Learners

Summary

The paper addresses automatic readability grading for learners of English as a second language. The main obstacle is that almost all level-annotated corpora target native-speaking children, and their levels reflect native criteria. The authors build a small CEFR-graded set of reading passages from Cambridge English Exams. They also clean the larger native WeeBit corpus and design a rich feature set: surface measures, lexico-semantic cues, EVP-based vocabulary features, GR-based syntactic complexity and language-model scores. Using linear SVMs, they then test how native data can support the scarce L2 data. They compare classification with pairwise ranking, learn a mapping from ranking scores to CEFR levels, apply EasyAdapt domain adaptation, and use self-training that treats native texts as unlabelled data. The best L2 result comes from self-training (ACC 0.797, PCC 0.938). A ranking model trained on native texts also transfers to unseen CEFR data much better than a native classifier. This matters for picking reading material for learners when graded learner data is scarce.

Key points

  • Cambridge Exams dataset (released publicly). It contains reading passages from KET, PET, FCE, CAE and CPE, mapped to CEFR A2–C2, with 60–71 documents per level.
  • Cleaned WeeBit corpus. The authors re-extracted WeeBit (Weekly Reader + BBC-Bitesize, five age-based levels) from raw HTML. They removed boilerplate such as copyright notices and links, because it correlated with the labels. The cleaned corpus keeps 529–1288 texts per level.
  • In-domain results with a linear SVM. On native data the model reaches ACC 0.803 / PCC 0.900. On L2 data alone it reaches ACC 0.785 / PCC 0.924.
  • Cross-domain transfer from native to L2 data.
    • Applying the native classifier directly to L2 texts gives only ACC 0.233 / PCC 0.730.
    • A native ranking model followed by a learned mapping to CEFR levels reaches ACC 0.622 / PCC 0.864.
    • EasyAdapt with ranking improves this to ACC 0.707 / PCC 0.899, but still stays below training on L2 data only.
  • Self-training. The best setting adds the 10 most confident native texts per round for 9 rounds (270 texts in total). The added texts go only to the three lowest L2 levels. This gives ACC 0.797 / PCC 0.938, a significant gain over L2-only training.

Interesting ideas

  • I1. Check scraped corpora for label-leaking boilerplate. Page leftovers such as footers and links can track the source and therefore the label, which inflates scores. Re-extracting clean text is a cheap safeguard for any web-derived benchmark.
  • I2. Rank first, then calibrate. Learning relative difficulty and then fitting a separate score-to-level mapping lets a model move between incompatible level scales. This could work wherever label schemes differ across sources, for example native vs. learner grades, or ratings from different annotators.
  • I3. Build ranking pairs only within one domain. When pooling data with mismatched scales for domain adaptation, the authors form preference pairs only inside each domain. This avoids meaningless cross-scale comparisons while still sharing features through EasyAdapt.
  • I4. Restrict self-training to the label range the unlabelled data covers. Adding pseudo-labelled native texts to all levels hurt performance. Limiting them to the lower levels, where children's texts plausibly overlap with learner levels, helped. The general lesson is to filter pseudo-labels by where the auxiliary data actually fits.
  • I5. Treat graded language-exam materials as labelled data. Past exam papers are already professionally levelled. This makes them a ready-made source of difficulty labels for low-resource settings, such as building learner-oriented corpora for other languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-0502

@inproceedings{xia2016text,
  title = {{Text Readability Assessment for Second Language Learners}},
  author = {Xia, Menglin and Kochmar, Ekaterina and Briscoe, Ted},
  booktitle = {Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2016},
  doi = {10.18653/v1/w16-0502},
  url = {https://doi.org/10.18653/v1/w16-0502}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34. António Branco, Jose Rodrigues, Francois Costa, Jaime Silva, and Richard Vaz. 2014. Assessing automatic text classification for interactive language learning. In Info… find our card
  2. Ted Briscoe, John Carroll, and Rebecca Watson. 2006. The second release of the RASP system. In Proceedings of the COLING/ACL on Interactive presentation sessions, pages 77–80. find our card
  3. Annette Capel. 2012. Completing the English Vocabulary Profile: C1 and C2 vocabulary. English Profile Journal, 3(e1). find
  4. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27. find
  5. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283–284. find
  6. Kevyn Collins-Thompson and James Callan. 2004. A Language Modeling Approach to Predicting Reading Difficulty. In Proceedings of North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 193–200. find
  7. Kevyn Collins-Thompson. 2014. Computational assessment of text readability: A survey of current and future research. ITL - International Journal of Applied Linguistics, 165(2):97–135. Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. find
  8. Cambridge University Press. Averil Coxhead. 2000. A new academic word list. TESOL Quarterly, pages 213–238. find
  9. Edgar Dale and Jeanne S. Chall. 1949. The concept of readability. Elementary English, 26(1):19–26. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256– 263. find
  10. Micha Eisner and Eugene Charniak. 2011. Extending the entity grid with entity-specific features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 125–129. Lijun Feng, Noémie Elhadad, and Matt Huener… find
  11. Michel Galley and Kathleen McKeown. 2003. Improving word sense disambiguation in lexical chaining. In International Joint Conference on Artificial Intelligence, volume 3, pages 1486–1488. find
  12. Michael Heilman, Kevyn Collins-Thompson, Jamie Callan, and Maxine Eskenazi. 2007. Combining lexical and grammatical features to improve readability measures for first and second language texts. In Proceedings of North American Chapter of the Association for Computational Linguistics: Human Language… find
  13. Michael Heilman, Kevyn Collins-Thompson, and Maxine Eskenazi. 2008. An analysis of statistical models and features for reading difficulty prediction. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications, pages 71– 79. find
  14. Rohit J. Kate, Xiaoqiang Luo, Siddharth Patwardhan, Martin Franz, Radu Florian, Raymond J. Mooney, Salim Roukos, and Chris Welty. 2010. Learning to predict readability using diverse linguistic features. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 546–554. find
  15. J. Peter Kincaid, Robert P. Fishburne Jr., Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Technical report, DTIC Document. find
  16. Xiaofei Lu. 2011. A Corpus-Based Evaluation of Syntactic Complexity Measures as Indices of College-Level ESL Writers’ Language Development. TESOL Quarterly, pages 36–62. find
  17. George A Miller. 1995. WordNet: a lexical database for English. Communications of the ACM, 38(11):39–41. find our card
  18. Peter Phandi, Kian Ming A. Chai, and Hwee Tou Ng. 2015. Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 431– 439. Ildikó Pilán, Sowmya Vajjala, and Elena Volo… find
  19. Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 186– 195. find our card
  20. Sarah E. Schwarm and Mari Ostendorf. 2005. Reading level assessment using support vector machines and statistical language models. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 523–530. find our card
  21. Wade Shen, Jennifer Williams, Tamas Marius, and Elizabeth Salesky. 2013. A Language-Independent Approach to Automatic Text Difficulty Assessment for Second-Language Learners. In Proceedings of the 2nd Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 30–38. find
  22. Luo Si and Jamie Callan. 2001. A statistical model for scientific readability. In Proceedings of the tenth international conference on Information and knowledge management, pages 574–576. find
  23. Andreas Stolcke. 2002. SRILM-an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, pages 901–904. find
  24. Sowmya Vajjala and Detmar Meurers. 2012. On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 163–173. find our card
  25. Sowmya Vajjala and Detmar Meurers. 2014. Readability assessment for text simplification: From analysing documents to identifying sentential simplifications. International Journal of Applied Linguistics, 165(2):194–222. find
  26. Evan James Williams. 1959. Regression analysis, volume 14. find
  27. Wiley New York. Helen Yannakoudakis. 2013. Automated assessment of English-learner writing. Technical Report UCAM-CL-TR-842, University of Cambridge, Computer Laboratory. find
  28. Xiaojin Zhu. 2005. Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2016-text-readability-assessment-second-language/