Elamur Research⌕
☰
home / papers / 2023 / Cross-corpus Readability Compatibility Assessment for English Texts

Cross-corpus Readability Compatibility Assessment for English Texts

Paper · 2023 · Zhenzhen Li, Han Ding and Shaohong Zhang

Summary

Readability research relies on many corpora, each with its own difficulty scale. Groups pick different datasets, and nobody has checked whether their labels agree. The authors propose CRCA (Cross-corpus text Readability Compatibility Assessment) to measure how well one corpus's difficulty system carries over to another.

They use six English corpora: CEFR, CLEC, CLOTH, NES (Newsela), OSP (OneStopEnglish) and RACE. Each text is represented by lexical, syntactic and semantic linguistic features, by GloVe vectors, or by a fusion of both. Classifiers are trained on a source corpus and then label texts from a target corpus. The models are XGBoost, SVM, BiLSTM and Attention-BiLSTM. Agreement between predicted and gold target labels is scored with three measures: RJSD, RRNSS and NDCG.

Three findings stand out:

  • Compatibility between corpora can be measured.
  • OSP behaves clearly unlike the other five corpora.
  • Corpus, feature type and classifier interact, and the three metrics tell a consistent story.

The work gives practical guidance on choosing corpora and is a first step toward cross-corpus transfer learning for readability.

Key points

  • Corpus sizes: NES is largest (33,006 texts), followed by RACE (27,931). CLOTH and CLEC each have under 10,000. CEFR (683) and OSP (567, i.e. 189 per level) are the smallest.
  • Level schemes:
    • NES: 11 grades (2–12)
    • CEFR: 6 levels
    • CLEC: 5 levels
    • OSP: 3 levels
    • CLOTH and RACE: 2 levels each (junior vs senior high)
  • Asymmetric compatibility: a model trained on S is evaluated on T, and the score ω(S, T) generally differs from ω(T, S).
  • Assumptions: difficulty levels are integers with equal spacing, and labels are transferable between corpora through some mapping function.
  • Availability: code is released at github.com/zimzsh/CRCA. Classic formulas (ARI, FKGL, GFI, SMOG, CLI, LIX, RIX) are reported as a reference.

Interesting ideas

  • I1. Measure whether two labelled datasets share a scheme by training on one, predicting the other, and comparing predicted with native labels; treat the score as directional rather than symmetric.
  • I2. Judge agreement with several metrics of different kinds (distribution divergence, rank-based error, ranking gain) and trust the conclusion only when they concur.
  • I3. Run a full grid of feature representations × classifiers × corpus pairs instead of declaring one best model, since the paper finds their effects interact.
  • I4. Treat ordinal labels as equally spaced integers so that scales of different granularity become comparable through a simple distance.
  • I5. Use cheap classic formulas as a sanity baseline confirming that dataset labels actually carry the signal being studied.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Before merging treebanks or applying UD dialect conversion, train the parser on treebank A and score its LAS and label distribution on B in both directions. This would flag treebanks whose annotation conventions clash, the way OSP stood out here.
  • I2 (learn): Silver-data agreement among witness parsers could be checked with several independent agreement measures. Accepting a batch only when they align would make evaluation sturdier than relying on one number.
  • I3 (learn): When comparing the perceptron parser with gparse on different treebanks, report the whole grid. A setup that wins on one data source may lose on another.
  • I4 (answer): School-course tests and fables from different sources could be mapped onto one integer difficulty ladder. Readers could then be evaluated on texts matched to a target level.
  • I5 (tokens): Surface statistics from crate en (sentence length, syllables, long words) could quickly check that incoming corpora are graded as claimed, before heavier analysis runs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Guangzhou UniversityGuangzhou · China4 works · 7 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Zhenzhen Li, Han Ding and Shaohong Zhang. 2023. Cross-corpus Readability Compatibility Assessment for English Texts. arXiv:2306.09704.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.09704 · arXiv 2306.09704

@misc{li2023cross,
  title = {{Cross-corpus Readability Compatibility Assessment for English Texts}},
  author = {Zhenzhen Li, Han Ding and Shaohong Zhang},
  year = {2023},
  doi = {10.48550/arxiv.2306.09704},
  eprint = {2306.09704},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.09704}
}

References

55 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. G. Drubin and D. R. Kellogg, ‘‘English as the universal language of science: opportunities and challenges,’’ Molecular biology of the cell, vol. 23, no. 8, pp. 1399–1399, 2012. ↗
  2. T. Huckin et al., ‘‘Second language reading and vocabulary learning.’’ 1995. find
  3. R. E. O’Connor, K. M. Bell, K. R. Harty, L. K. Larkin, S. M. Sackor, and N. Zigmond, ‘‘Teaching reading to poor readers in the intermediate grades: A comparison of text difficulty.’’ Journal of Educational Psychology, vol. 94, no. 3, p. 474, 2002. ↗
  4. S. Vajjala and D. Meurers, ‘‘Readability assessment for text simplification: From analysing documents to identifying sentential simplifications,’’ ITL-International Journal of Applied Linguistics, vol. 165, no. 2, pp. 194–222, 2014. find
  5. Y. Ma, E. Fosler-Lussier, and R. Lofthus, ‘‘Ranking-based readability assessment for early primary children’s literature,’’ in Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2012, pp. 548–552. find
  6. J. Lee and S. Vajjala, ‘‘A neural pairwise ranking model for readability assessment,’’ arXiv preprint arXiv:2203.07450, 2022. ↗
  7. M. Xia, E. Kochmar, and T. Briscoe, ‘‘Text readability assessment for second language learners,’’ arXiv preprint arXiv:1906.07580, 2019. ↗
  8. I. M. Azpiazu and M. S. Pera, ‘‘Multiattentive recurrent neural network architecture for multilingual readability assessment,’’ Transactions of the Association for Computational Linguistics, vol. 7, pp. 421–436, 2019. ↗
  9. I. Madrazo Azpiazu and M. S. Pera, ‘‘Is cross-lingual readability assessment possible?’’ Journal of the Association for Information Science and Technology, vol. 71, no. 6, pp. 644–656, 2020. ↗
  10. S. Vajjala, ‘‘Trends, limitations and open challenges in automatic readability assessment research,’’ arXiv preprint arXiv:2105.00973, 2021. ↗
  11. T. Oakland and H. B. Lane, ‘‘Language, reading, and readability formulas: Implications for developing and adapting tests,’’ International Journal of Testing, vol. 4, no. 3, pp. 239–252, 2004. ↗
  12. H. Mohammadi and S. H. Khasteh, ‘‘Text as environment: A deep reinforcement learning text readability assessment model,’’ arXiv preprint arXiv:1912.05957, 2019. ↗
  13. C. Meng, M. Chen, J. Mao, and J. Neville, ‘‘Readnet: A hierarchical transformer framework for web article readability analysis,’’ in Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part I 42. Springer, 2020, pp. 33–49. find
  14. M. Cha, Y. Gwon, and H. Kung, ‘‘Language modeling by clustering with word embeddings for text readability assessment,’’ in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 2017, pp. 2003–2006. find
  15. J. Y. Kim, K. Collins-Thompson, P. N. Bennett, and S. T. Dumais, ‘‘Characterizing web content, user interests, and search behavior by reading level and topic,’’ in Proceedings of the fifth ACM international conference on Web search and data mining, 2012, pp. 213–222. find
  16. T. Deutsch, M. Jasbi, and S. Shieber, ‘‘Linguistic features for readability assessment,’’ arXiv preprint arXiv:2006.00377, 2020. ↗
  17. B. W. Lee, Y. S. Jang, and J. H.-J. Lee, ‘‘Pushing on text readability assessment: A transformer meets handcrafted linguistic features,’’ arXiv preprint arXiv:2109.12258, 2021. ↗
  18. T. François and C. Fairon, ‘‘An “ai readability” formula for french as a foreign language,’’ in Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, 2012, pp. 466–477.
  19. Z. Weiss, X. Chen, and D. Meurers, ‘‘Using broad linguistic complexity modeling for cross-lingual readability assessment,’’ in Proceedings of the 10th Workshop on NLP for Computer Assisted Language Learning, 2021, pp. 38–54. find
  20. R. Senter and E. A. Smith, ‘‘Automated readability index,’’ Cincinnati Univ OH, Tech. Rep., 1967. find
  21. J. P. Kincaid, R. P. Fishburne Jr, R. L. Rogers, and B. S. Chissom, ‘‘Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel,’’ Naval Technical Training Command Millington TN Research Branch, Tech. Rep., 1975. ↗
  22. R. Gunning et al., ‘‘Technique of clear writing,’’ 1952. find
  23. G. H. Mc Laughlin, ‘‘Smog grading-a new readability formula,’’ Journal of reading, vol. 12, no. 8, pp. 639–646, 1969. find
  24. M. Coleman and T. L. Liau, ‘‘A computer readability formula designed for machine scoring.’’ Journal of Applied Psychology, vol. 60, no. 2, p. 283, 1975. ↗
  25. C.-H. Björnsson, ‘‘Readability of newspapers in 11 languages,’’ Reading Research Quarterly, pp. 480–497, 1983. ↗
  26. J. Anderson, ‘‘Lix and rix: Variations on a little-known readability index,’’ Journal of Reading, vol. 26, no. 6, pp. 490–496, 1983. find
  27. C. of Europe. Council for Cultural Co-operation. Education Committee. Modern Languages Division, Common European framework of reference for languages: Learning, teaching, assessment. Cambridge University Press, 2001. find
  28. G. A. Khushik and A. Huhta, ‘‘Syntactic complexity in finnish-background efl learners’ writing at cefr levels a1–b2,’’ European Journal of Applied Linguistics, vol. 10, no. 1, pp. 142–184, 2022. find
  29. L.-f. Huang, Y.-l. Lin, and T. Gráf, ‘‘Development of the use of discourse markers across different fluency levels of cefr: A learner corpus analysis,’’ Pragmatics, vol. 33, no. 1, pp. 49–77, 2023. ↗
  30. M. Negishi, T. Takada, and Y. Tono, ‘‘A progress report on the development of the cefr-j,’’ in Exploring language frameworks: Proceedings of the ALTE Kraków Conference, 2013, pp. 135–163. find
  31. H. Yang, ‘‘Corpus-based analysis of chinese learner english,’’ Foreign Languages in China, no. 6, pp. 11–13, 2005. find
  32. J. Chen, ‘‘A corpus-based analysis of although errors in chinese efl learners’ written output,’’ Studies in English Language Teaching, vol. 5, no. 3, p. 429, 2017. find
  33. H. He, ‘‘A computer-aided analysis on word form errors in college english writing—a corpus-based study,’’ International Journal of Emerging Technologies in Learning (Online), vol. 11, no. 3, p. 4, 2016. ↗
  34. Q. WANG, ‘‘A corpus-based contrastive study on semantic prosody of english near synonyms: A case study of motive and motivationa corpus-based contrastive study on semantic prosody of english near synonyms: A case study of motive and motivation,’’ Journal of Arts and Humanities, vol. 8, no. 1, pp. 01–15, 2019. ↗
  35. Q. Xie, G. Lai, Z. Dai, and E. Hovy, ‘‘Large-scale cloze test dataset created by teachers,’’ arXiv preprint arXiv:1711.03225, 2017. ↗
  36. S. Liu, X. Zhang, S. Zhang, H. Wang, and W. Zhang, ‘‘Neural machine reading comprehension: Methods and trends,’’ Applied Sciences, vol. 9, no. 18, p. 3698, 2019. ↗
  37. Y. Xu, D. Wang, M. Yu, D. Ritchie, B. Yao, T. Wu, Z. Zhang, T. J.-J. Li, N. Bradford, B. Sun et al., ‘‘Fantastic questions and where to find them: Fairytaleqa–an authentic dataset for narrative comprehension,’’ arXiv preprint arXiv:2203.13947, 2022. ↗
  38. S. Lv, Y. Wang, D. Guo, D. Tang, N. Duan, F. Zhu, M. Gong, L. Shou, R. Ma, D. Jiang et al., ‘‘Pre-training text representations as meta learning,’’ arXiv preprint arXiv:2004.05568, 2020. ↗
  39. W. Xu, C. Callison-Burch, and C. Napoles, ‘‘Problems in current text simplification research: New data can help,’’ Transactions of the Association for Computational Linguistics, vol. 3, pp. 283–297, 2015. ↗
  40. M. Martinc, S. Pollak, and M. Robnik-Šikonja, ‘‘Supervised and unsupervised neural approaches to text readability,’’ Computational Linguistics, vol. 47, no. 1, pp. 141–179, 2021. ↗
  41. S. Vajjala and I. Lučić, ‘‘Onestopenglish corpus: A new corpus for automatic readability assessment and text simplification,’’ in Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications, 2018, pp. 297–304.
  42. J. M. Imperial, ‘‘Bert embeddings for automatic readability assessment,’’ arXiv preprint arXiv:2106.07935, 2021. ↗
  43. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, ‘‘Race: Large-scale reading comprehension dataset from examinations,’’ arXiv preprint arXiv:1704.04683, 2017. ↗
  44. G. Chen, J. Yang, C. Hauff, and G.-J. Houben, ‘‘Learningq: a large-scale dataset for educational question generation,’’ in Proceedings of the International AAAI Conference on Web and Social Media, vol. 12, no. 1, 2018. find
  45. C. Si, Z. Yang, Y. Cui, W. Ma, T. Liu, and S. Wang, ‘‘Benchmarking robustness of machine reading comprehension models,’’ arXiv preprint arXiv:2004.14004, 2020. ↗
  46. J. Malmaud, R. Levy, and Y. Berzak, ‘‘Bridging information-seeking human gaze and machine reading comprehension,’’ arXiv preprint arXiv:2009.14780, 2020. ↗
  47. C. Cortes and V. Vapnik, ‘‘Support-vector networks,’’ Machine learning, vol. 20, pp. 273–297, 1995. ↗
  48. S. Dumais, J. Platt, D. Heckerman, and M. Sahami, ‘‘Inductive learning algorithms and representations for text categorization,’’ in Proceedings of the seventh international conference on Information and knowledge management, 1998, pp. 148–155. find
  49. T. Joachims, ‘‘Text categorization with support vector machines: Learning with many relevant features,’’ in Machine Learning: ECML-98: 10th European Conference on Machine Learning Chemnitz, Germany, April 21–23, 1998 Proceedings. Springer, 2005, pp. 137–142.
  50. S. S. Stevens, ‘‘On the theory of scales of measurement,’’ Science, vol. 103, no. 2684, pp. 677–680, 1946. ↗
  51. J. Lin, ‘‘Divergence measures based on the shannon entropy,’’ IEEE Transactions on Information theory, vol. 37, no. 1, pp. 145–151, 1991. ↗
  52. T. Chai and R. R. Draxler, ‘‘Root mean square error (rmse) or mean absolute error (mae)?–arguments against avoiding rmse in the literature,’’ Geoscientific model development, vol. 7, no. 3, pp. 1247–1250, 2014. ↗
  53. R. Busa-Fekete, G. Szarvas, T. Elteto, and B. Kégl, ‘‘An apple-to-apple comparison of learning-to-rank algorithms in terms of normalized discounted cumulative gain,’’ in ECAI 2012-20th European Conference on Artificial Intelligence: Preference Learning: Problems and Applications in AI Workshop, vol. 242. Ios Press, 2012. find
  54. K. Järvelin, S. L. Price, L. M. Delcambre, and M. L. Nielsen, ‘‘Discounted cumulated gain based evaluation of multiple-query ir sessions,’’ in Advances in Information Retrieval: 30th European Conference on IR Research, ECIR 2008, Glasgow, UK, March 30-April 3, 2008. Proceedings 30. Springer, 2008, pp. 4–15. find
  55. K. Pearson, ‘‘Vii. note on regression and inheritance in the case of two parents,’’ proceedings of the royal society of London, vol. 58, no. 347-352, pp. 240–242, 1895. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2023-cross-corpus-readability-compatibility-assessment/