Elamur Research⌕
☰
home / papers / 2021 / To What Extent Can English-as-a-Second Language Learners Read Economic News Texts?

To What Extent Can English-as-a-Second Language Learners Read Economic News Texts?

Paper · 2021 · Yo Ehara

Summary

Economic decisions often depend on reading English-language news quickly, yet non-native readers may face a language barrier. This challenges the efficient market hypothesis, which assumes no information gaps. The paper asks how much economic news ESL learners can actually read. It builds two readability assessors and validates them on OneStopEnglish, a corpus whose Guardian articles were rewritten by teachers at elementary, intermediate and advanced levels:

  • spvBERT: a supervised BERT classifier fine-tuned on that corpus.
  • Vocabulary-based scorer: an unsupervised model. It uses item response theory, fitted on learners' vocabulary-test answers, to estimate how likely an average learner is to know each word. Corpus frequencies extend these estimates to words the test does not cover.

Both assessors correlate strongly with the gold levels, and they agree closely with each other. Applied to Reuters economic news from 2007, they show that intermediate learners can handle the vast majority of articles. A small share remains advanced. A handful of domain-specific words can sharply lower a given learner's chance of understanding a text.

Key points

  • Data split: OneStopEnglish has 567 texts, 189 per level, divided into 339 for training, 114 for validation and 114 for testing. Unsupervised methods were scored on the test portion only.

  • spvBERT setup: bert-large-cased-whole-word-masking, fine-tuned with Adam for 10 epochs at a learning rate of 1e-5.

  • Vocabulary-based model:

    • It fits a model of the form sigmoid(learner ability − word difficulty) using Ehara (2018) data: 100 learners answering the 100-item Vocabulary Size Test.
    • Word difficulty is a weighted sum of log frequencies from BNC and COCA, fitted with logistic regression.
    • A text's score is the negative log of the probability that the average learner knows every word in it.

  • Test-set correlations:

    MethodSpearmanPearson
    spvBERT0.8660.864
    Vocabulary-based0.7300.715
    Dale-Chall (best classic formula)0.4950.506
    TCN RSRS-simple (prior best, Martinc et al. 2021)not reported0.615

  • Reuters results: The news set comes from Ding et al. and holds 109,110 articles from 2006–2007, roughly 1,000 per month in 2007.

    • Every month of 2007 was about 94–98% intermediate, about 2–5% advanced, and almost never elementary.
    • The conclusion reports 2.4% of articles as unreadable for intermediate learners.
    • One case study used a learner scoring 75/100 on the vocabulary test, estimated to know about 15,000 words. In one article, unfamiliar words such as "annuity" and "veritable" cut this learner's probability of knowing at least 95% of the words from 0.62 to 0.38.

Interesting ideas

  • I1. An unsupervised, interpretable assessor and a supervised black-box model reach similar rankings, so each one validates the other without extra labels.
  • I2. Item response theory puts reader ability and item difficulty on one scale, and frequency features extend the difficulty estimates to items that were never tested.
  • I3. Readability is framed as the probability that a reader knows at least 95% of a text's words, not as an average score, which ties the number to a known comprehension threshold.
  • I4. The model can name the few rare words responsible for a large drop in comprehension probability, which makes the score explainable and actionable.
  • I5. The evaluation corpus holds the same content rewritten at several levels, so labels cannot be predicted from topic alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Pair each neural component with a transparent rule-based counterpart, and treat agreement between them as a silver-data witness rather than trusting either alone.
  • I2 (tokens): Crate en could attach a calibrated difficulty value to each lemma, extrapolated from corpus counts, so later stages can tell which words a given reader probably lacks.
  • I3 (answer): School-course readers could estimate whether a student can handle a passage before answering questions on it, and flag which words block comprehension.
  • I4 (back): Generation and the text → graph → text compressor could target only those blocking words for paraphrase, simplifying text for a chosen reader level without rewriting everything.
  • I5 (learn): Build evaluation sets of parallel rewrites, such as the same fable at different complexity, as a negative control showing a component reacts to structure rather than subject matter.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tokyo Gakugei UniversityTokyo · Japan2 works · 3 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Yo Ehara. 2021. To What Extent Can English-as-a-Second Language Learners Read Economic News Texts?. Proceedings of the Third Workshop on Economics and Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.econlp-1.9

@inproceedings{ehara2021what,
  title = {{To What Extent Can English-as-a-Second Language Learners Read Economic News Texts?}},
  author = {Yo Ehara},
  booktitle = {Proceedings of the Third Workshop on Economics and Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.econlp-1.9},
  url = {https://doi.org/10.18653/v1/2021.econlp-1.9}
}

References

40 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Anderson. 1983. Lix and rix: Variations on a little-known readability index. Journal of Reading, 26(6):490–496. find
  2. Frank B. Baker. 2004. Item Response Theory : Parameter Estimation Techniques, Second Edition. CRC Press. find
  3. David Beglar and Paul Nation. 2007. A vocabulary size test. The Language Teacher, 31(7):9–13. find
  4. Lisa Beinborn, Torsten Zesch, and Iryna Gurevych. 2016. Predicting the Spelling Difficulty of Words for Language Learners. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 73–83, San Diego, CA. Association for Computational Linguistics. find
  5. C. H. Björnsson. 1968. Läsbarhet, Stockholm. find
  6. BNC Consortium. 2007. The british national corpus, version 3 (bnc xml edition). Distributed by Bodleian Libraries, University of Oxford, on behalf of the BNC Consortium http://www.natcorp.ox.ac.uk/. ↗
  7. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283. ↗
  8. Edgar Dale and Jeanne S Chall. 1948. A formula for predicting readability: Instructions. Educational research bulletin, pages 37–54. find
  9. Mark Davies. 2008. The corpus of contemporary american english (coca). Available online at https://www.english-corpora.org/coca/. ↗
  10. The journal version of (Ehara et al., 2012) is (Ehara et al., 2018). find
  11. Yo Ehara, Nobuyuki Shimizu, Takashi Ninomiya, and Hiroshi Nakagawa. 2013. Personalized Reading Support for Second-language Web Documents. ACM Trans. Intell. Syst. Technol., 4(2):31:1–31:19. ↗
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of NAACL, pages 4171–4186, Minneapolis, Minnesota.
  13. Rudolf Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32(3):221–233. ↗
  14. Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2014. Using structured events to predict stock price movement: An empirical investigation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1415–1425, Doha, Qatar. Association for Computational Linguistics. find
  15. Robert Gunning. 1952. The Technique of Clear Writing. McGraw-Hill. find
  16. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch. ↗
  17. Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2015. Deep learning for event-driven stock prediction. In Twenty-fourth international joint conference on artificial intelligence. find
  18. Yo Ehara. 2018. Building an English Vocabulary Knowledge Dataset of Japanese English-as-a-Second-Language Learners Using Crowdsourcing. In Proc. of LREC. find
  19. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR. ↗
  20. Batia Laufer and Geke C. Ravenhorst-Kalovski. 2010. Lexical Threshold Revisited: Lexical Text Coverage, Learners’ Vocabulary Size and Reading Comprehension. Reading in a Foreign Language, 22(1):15–30. ↗
  21. Yo Ehara. 2019. Uncertainty-Aware Personalized Readability Assessments for Second Language Learners. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pages 1909–1916. find
  22. John Lee and Chak Yan Yeung. 2018. Personalizing Lexical Simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 224–232, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  23. Yo Ehara. 2020a. Interpreting neural CWI classifiers’ weights as vocabulary size. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 171–176, Seattle, WA, USA → Online. Association for Computational Linguistics. find
  24. John Lee and Chak Yan Yeung. 2019. Personalized Substitution Ranking for Lexical Simplification. In Proceedings of the 12th International Conference on Natural Language Generation, pages 258–267, Tokyo, Japan. Association for Computational Linguistics. find
  25. Yo Ehara. 2020b. Neural rasch model: How do word embeddings adjust word difficulty? In Computational Linguistics, pages 88–96, Singapore. Springer Singapore. find
  26. John SY Lee. 2021. An editable learner model for text recommendation for language learning. ReCALL, pages 1–15. ↗
  27. Yo Ehara, Yukino Baba, Masao Utiyama, and Eiichiro Sumita. 2016. Assessing Translation Ability through Vocabulary Ability Assessment. In Proc. of IJCAI. find
  28. Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2021. Supervised and Unsupervised Neural Approaches to Text Readability. Computational Linguistics, 47(1):141–179. ↗
  29. Yo Ehara, Yusuke Miyao, Hidekazu Oiwa, Issei Sato, and Hiroshi Nakagawa. 2014. Formalizing Word Sampling for Vocabulary Prediction as Graph-based Active Learning. In Proc. of EMNLP, pages 1374– 1384. find
  30. G Harry Mc Laughlin. 1969. Smog grading-a new readability formula. Journal of reading, 12(8):639–646. find
  31. Yo Ehara, Issei Sato, Hidekazu Oiwa, and Hiroshi Nakagawa. 2012. Mining Words in the Minds of Second Language Learners: Learner-Specific Word Difficulty. In Proceedings of COLING 2012, pages 799–814, Mumbai, India. find
  32. The COLING 2012 Organizing Committee. find
  33. I. Nation. 2006. How Large a Vocabulary is Needed For Reading and Listening? Canadian Modern Language Review, 63(1):59–82. ↗
  34. Gustavo Paetzold and Lucia Specia. 2016. Collecting and Exploring Everyday Language for Predicting Psycholinguistic Properties of Words. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1669–1679, Osaka, Japan. ↗
  35. The COLING 2016 Organizing Committee. find
  36. Yo Ehara, Issei Sato, Hidekazu Oiwa, and Hiroshi Nakagawa. 2018. Mining Words in the Minds of Second Language Learners for Learner-specific Word Difficulty. Journal of Information Processing, 26:267– 275. ↗
  37. RJ Senter and Edgar A Smith. 1967. Automated readability index. Technical report, CINCINNATI UNIV OH. find
  38. Sowmya Vajjala and Ivana Lučić. 2018. OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304, New Orleans, Louisiana. Association for Computational Linguistics.
  39. Chak Yan Yeung and John Lee. 2018. Personalized Text Retrieval for Learners of Chinese as a Foreign Language. In Proc. of COLING, pages 3448–3455. find
  40. Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo H. Paetzold, Lucia Specia, Sanja Štajner, Anaïs Tack, and Marcos Zampieri. 2018. A Report on the Complex Word Identification Shared Task 2018. arXiv:1804.09132 [cs]. ArXiv: 1804.09132. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ehara-2021-what-extent-can-english-as/