Elamur Research⌕
☰
home / papers / 2018 / Predicting CEFRL levels in learner English on the basis of metrics and full texts

Predicting CEFRL levels in learner English on the basis of metrics and full texts

Paper · 2018 · Taylor Arnold, Nicolas Ballier, Thomas Gaillat et al.

Summary

The authors ask whether the CEFRL proficiency level of a learner's English essay can be estimated automatically, without relying on error spotting. They took 41,626 essays written by French learners from the EFCAMDAT corpus. Each essay was described with lexical diversity, syntactic complexity (L2SCA) and readability measures, computed with Lu's tools and koRpus. These representations were compared with term-frequency and raw word-sequence inputs. Because the level distribution is heavily skewed, they trained a separate binary classifier for each pair of neighbouring levels instead of one multiclass model. They also kept all essays from the same prompt topic on one side of the train/test split, so that models could not succeed by recognising the topic. Gradient boosted trees on the metrics, extended with UPOS, UD relation and Zipf-band counts, gave the best test results at the low end: 0.916 AUC for A1=>A2 and 0.904 for A2=>B1. Distinctions between higher levels stayed weak. The study suggests that compact, interpretable metrics transfer to unseen topics better than lexical models do.

Key points

  • Data: EFCAMDAT texts from French learners. There are 41,626 essays (about 3.3M tokens) from 7,695 learners, covering 128 topic units. Levels range from 17,605 A1 essays down to only 76 at C2.
  • Feature families:
    • lexical diversity indices (TTR variants, MTLD, MATTR, HD-D, Yule's K, etc.);
    • L2SCA syntactic measures (MLS, MLT, DC/C, CN/T, etc.);
    • readability formulae;
    • in the "Metrics+" set, added counts of UPOS tags, UD relations and Zipf-scale word bands, extracted with cleanNLP.
  • Models compared:
    • GBT with max depth 4, learning rate 0.01 and 100 trees;
    • Elastic Net on term frequencies;
    • LSTM on word sequences.
  • Lexical models overfit badly. For example, the LSTM scored 0.985 AUC on train but 0.863 on test for A1=>A2, and close to chance above B1. Metrics+ GBT was the best on test overall, though C1=>C2 reached only 0.558. A cascading multiclass setup reached 70.34% accuracy.
  • Feature importance was dominated by word tokens, word types and W. Clause-level ratios such as DC.C and CN.T appeared mainly for higher-level pairs.

Interesting ideas

  • I1. Grouping the train/test split by writing prompt keeps topic words from acting as a stand-in for the label, and the train-versus-test AUC gap exposes the leak.
  • I2. An ordinal, skewed label set can be handled by one binary model per pair of adjacent levels, which concentrates each model on subtle neighbouring differences.
  • I3. Simple histograms of UPOS tags and UD relations per text serve as compact, explainable descriptors of a text's syntactic profile.
  • I4. Learners' variable spelling can inflate token and type counts, so it matters whether a metric runs on raw or tokenized text.
  • I5. The UPOS tagset merges commas and full stops under one tag, which hides information-structure signals; the authors propose re-tagging them separately.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Split silver and gold evaluation data by source text or genre rather than by sentence, so repair rules and parser scores cannot depend on memorised vocabulary.
  • I2 (answer): Graded readers or school-course test difficulty could be estimated pairwise (grade n vs n+1) instead of with a single multiclass judge.
  • I3 (learn): Comparing per-document deprel/UPOS histograms against treebank norms could act as a cheap drift alarm on parser output, catching silver batches whose relation mix looks implausible.
  • I4 (tokens): Normalise or flag misspellings before computing type/token statistics, and record which layer each statistic was computed on.
  • I5 (tree): Keep a fine-grained punctuation subtype alongside the UD punct relation so that apposition and sentence-boundary cues remain available for parsing and later stages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Approches Linguistiques Théoriques, Appliquées et Expérimentales : langues et cultures connectéesParis · France6 works · 260 citations

Also on the byline: Linguistique Ingénierie et Didactique des Langues.

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksDevelopmental and Educational Psychology · 260 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksEvaluation & benchmarks · 685 worksGrammatical error correction & learner language · 80 works

Cite

Cite as Taylor Arnold, Nicolas Ballier, Thomas Gaillat and Paula Lisson. 2018. Predicting CEFRL levels in learner English on the basis of metrics and full texts. arXiv:1806.11099.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1806.11099 · arXiv 1806.11099

@misc{arnold2018predicting,
  title = {{Predicting CEFRL levels in learner English on the basis of metrics and full texts}},
  author = {Taylor Arnold, Nicolas Ballier, Thomas Gaillat and Paula Lisson},
  year = {2018},
  doi = {10.48550/arxiv.1806.11099},
  eprint = {1806.11099},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1806.11099}
}

References

11 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Theodora Alexopoulou, Marije Michel, [Lu10] Akira Murakami, and Detmar Meurers. Task effects on linguistic complexity and accuracy: A large-scale learner corpus analysis employing natural language processing techniques. Language Learning, [Lu12] 67(S1):180–208, 2017. ↗
  2. Nicolas Ballier and Thomas Gaillat. Classifying French learners of English with written-based lexical and complexity met- [Lu14] rics. In JEP-TALN-RECITAL 2016, volume 9 of ELTAL, pages 1–14, Paris, France, 2016. find
  3. Xiaobin Chen and Detmar Meurers. Ctap: A web-based tool supporting automatic complexity analysis. In CL4LC, pages [mm17] 113–119, 2016. find
  4. Scott A. Crossley, Tom Salsbury, Danielle S. McNamara, and Scott Jarvis. Predicting lexical proficiency in language learner texts using computational indices. Language Testing, 28(4):561–580, 2011. Miao Chen and Klaus Zechner. Computing and Evaluating Syntactic Complexity Features for Automated Scoring of Spontaneous Non-native Speech. Association for Computational Linguistics, 2011. Brendan Flanagan and Sachio Hirokawa. The Relationship of English Foreign Language Learner Proficiency and an Entropy Based Measure. Information Engineering Express, 1(3):29–38, 2015. Jeroen Geertzen, Theodora Alexopoulou, and Anna Korhonen. Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat). 2013. Derrick Higgins, Xiaoming Xi, Klaus Zechner, and David Williamson. A Three-stage Approach to the Automated Scoring of Spontaneous Spoken Responses. Comput. Speech Lang., 25(2):282–306, 2011. Claudia Leacock and Martin Chodorow. C-rater: Automated Scoring of Short-Answer Questions. Computers and the Humanities, 37(4):389–405, 2003. Xiaofei Lu. Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4):474–496, 2010. Xiaofei Lu. The Relationship of Lexical Richness to the Quality of ESL Learners’ Oral Narratives. The Modern Language Journal, 96(2):190–208, 2012. Xiaofei Lu. Computational Methods for Corpus Annotation and Analysis. Springer, Dordrecht, 2014. T. Mitchell. Towards robust computerised marking of free-text responses. 2002. m.eik michalke. koRpus: An R Package for Text Analysis, 2017. (Version 0.10-2). find
  5. Ellis B. Page. The Use of the Computer in Analyzing Student Essays. International Review of Education, 14(2):210–225, 1968. ↗
  6. Fiona J Tweedie and R Harald Baayen. How variable may a constant be? measures of lexical richness in perspective. Computers and the Humanities, 32(5):323–352, 1998. ↗
  7. Y. Tono. Automatic Extraction of L2 Criterial Lexicogrammatical Features across Pseudo-longitudinal Learner Corpora. Eurosla. 2013. find
  8. Sowmya Vajjala. Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education, 28(1):79–105, Mar 2018. ↗
  9. Sowmya Vajjala and Detmar Meurers. On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, 2012.
  10. Su-Youn Yoon and Suma Bhat. Assessment of ESL Learners’ Syntactic Competence Based on Similarity Measures. EMNLP-CoNLL ’12. Association for Computational Linguistics, 2012. find
  11. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. A New Dataset and Method for Automatically Grading ESOL Texts. In ACL Proceedings: Human Language Technologies - Volume 1, 2011. This figure "Boosted-tree.png" is available in "png" format from: http://arxiv.org/ps/1806.11099v1 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arnold-2018-predicting-cefrl-levels-learner-english/