Elamur Research⌕
☰
home / papers / 2020 / Un prototype en ligne pour la prédiction du niveau de compétence en anglais des productions écrites (A prototype for web-based prediction of English proficiency levels in writings)

Un prototype en ligne pour la prédiction du niveau de compétence en anglais des productions écrites (A prototype for web-based prediction of English proficiency levels in writings)

Paper · 2020 · Thomas Gaillat, Nicolas Ballier, Annanda Sousa et al.

Summary

This demonstration paper presents a proof-of-concept tool that automatically places English learners' essays on CEFR proficiency levels. The authors trained a supervised classifier on EFCAMDAT, a corpus of over 40,000 level-annotated learner texts. Each text is described by 769 features from a chain of existing analysers covering syntactic, lexical, discourse and psycholinguistic complexity, plus readability and spelling. The features describe what the learner actually produces and do not rely on annotated errors. A novel feature family, "microsystems", records how often competing linguistic forms are used relative to one another, independent of the rest of the text. Among the models tried, logistic regression performed best, reaching 82% accuracy. The system has two parts: a public web interface that returns a predicted CEFR class with its probability, and a command-line batch pipeline that builds metric datasets. The work shows how criterial features from second-language research can be turned into a usable assessment tool for teachers.

Key points

  • Training data: EFCAMDAT, more than 40,000 texts and about 3,298,343 tokens, already annotated with proficiency levels (Geertzen et al., 2013).
  • There are 769 features, and the best model is logistic regression, with a reported accuracy of 82%.
  • The feature tools are a modified L2SCA, LCA, TAALES, TAASC, TAACO, Pyenchant 2.0.0 (an Aspell-based spelling check) and Textstat (readability indices such as Dale_Chall, plus n-gram frequencies).
  • The architecture uses interconnected Docker modules written in Python 3.6. Because one tool depends on a proprietary corpus, only a reduced version of the pipeline is publicly released.
  • Planned extensions include:
    • speech input through the SpeechRecognition library;
    • morphological microsystem metrics;
    • visualisations comparing an individual learner with typical learner profiles.

Interesting ideas

  • I1. Measuring "microsystems", i.e. the usage ratios among a small set of competing forms, captures paradigmatic choices that length-based syntactic measures miss, and it can be applied to any closed set of alternatives.
  • I2. Features describe the positive properties a writer shows rather than counting errors, which avoids dependence on costly error annotation.
  • I3. Many heterogeneous off-the-shelf analysers are concatenated into one feature vector and fed to a transparent linear model, so each prediction can be traced back to named metrics.
  • I4. The same metric pipeline serves both an interactive interface (class plus probability) and a batch mode that writes per-text metric datasets for research.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): compute ratios between competing constructions in parsed UD trees, so a skewed distribution in a new parser's output signals a dialect or conversion problem.
  • I2 (answer): grade student answers on school-course tests by which structures and relations they correctly express, giving an explainable profile rather than a list of mistakes.
  • I3 (learn): estimate text difficulty from interpretable tree and lexical metrics, then order self-play or training texts from easy to hard.
  • I4 (learn): add a batch mode that dumps per-text metric tables, so evaluation runs and negative controls can be compared with simple statistics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université Paris CitéParis · France20 works · 1,426 citations🔬Laboratoire d'Informatique de Paris-NordVilletaneuse · France10 works · 510 citations🎓Université Sorbonne Paris NordVilletaneuse · France9 works · 468 citations🔬Approches Linguistiques Théoriques, Appliquées et Expérimentales : langues et cultures connectéesParis · France10 works · 260 citations🔬Linguistique Ingénierie et Didactique des LanguesRennes · France5 works · 14 citations🎓Université Rennes 2Rennes · France4 works · 9 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Thomas Gaillat, Nicolas Ballier, Annanda Sousa, Manon Bouyé, Andrew Simpkin, Bernardo Stearns and Manel Zarrouk. 2020. Un prototype en ligne pour la prédiction du niveau de compétence en anglais des productions écrites (A prototype for web-based prediction of English proficiency levels in writings). Actes de la 6e conférence conjointe Journées d'Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Volume 4 : Démonstrations et résumés d'articles internationaux.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gaillat2020un,
  title = {{Un prototype en ligne pour la prédiction du niveau de compétence en anglais des productions écrites (A prototype for web-based prediction of English proficiency levels in writings)}},
  author = {Thomas Gaillat, Nicolas Ballier, Annanda Sousa, Manon Bouyé, Andrew Simpkin, Bernardo Stearns and Manel Zarrouk},
  booktitle = {Actes de la 6e conférence conjointe Journées d'Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Volume 4 : Démonstrations et résumés d'articles internationaux},
  year = {2020},
  url = {https://aclanthology.org/2020.jeptalnrecital-demos.8/}
}

References

14 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B ALLIER N., G AILLAT T., S IMPKIN A., S TEARNS B., B OUYÉ M. & Z ARROUK M. (2019). A supervised learning model for the automatic assessment of language levels based on learner errors. In European Conference on Technology Enhanced Learning, p. 308–320 : Springer. find
  2. B ANSAL S. (2018). Textstat Python package. shivam5992/textstat. Accessible sur https://github.com/ ↗
  3. B REZINA V. & P ALLOTTI G. (2019). Morphological complexity in written l2 texts. Second language research, 35(1), 99–119. find
  4. D ECHERT , H ANS -W ILHELM (1983). How a story is done in a second language. In C. F ÆRCH & G. K ASPER , Éds., Strategies in interlanguage communication, p. 175–195. London ; New York : Longman. OCLC : 644977107. find
  5. E UROPEAN C OUNCIL (2001). Common European Framework of Reference for Languages : Learning, teaching, assessment. Cambridge : Cambridge University Press. G AILLAT T., S IMPKIN A., B ALLIER N., S TEARNS B., S OUSA A., B OUYÉ M. & Z ARROUK M. (submitted). Predicting CEFR levels in learners of English : the use of microsystem criterial features in a machine learning approach. Journal With Anonymous Submission. find
  6. G EERTZEN J., A LEXOPOULOU T. & K ORHONEN A. (2013). Automatic Linguistic Annotation of Large Scale L2 Databases : The EF-Cambridge Open Language Database (EFCamDat). In R. T. M ILLER , K. I. M ARTIN , C. M. E DDINGTON , A. H ENERY , N. M IGUEL , A. T SENG , A. T UNINETTI & D. W ALTER , Éds., Proceeedings of the 31st Second Language Research Forum, Carnegie Mellon : Cascadilla Press. find
  7. H AWKINS J. A. & F ILIPOVI Ć L. (2012). Criterial Features in L2 English : Specifying the Reference Levels of the Common European Framework. United Kingdom : Cambridge University Press. find
  8. H OUSEN A., K UIKEN F. & V EDDER I., Éds. (2012). Dimensions of L2 performance and proficiency : complexity, accuracy and fluency in SLA, volume 32 de Language Learning & Language Teaching (LL&LT). Amsterdam, Pays-Bas, Etats-Unis d’Amérique : John Benjamins Publishing Company. find
  9. K ELLY R. (2016). PyEnchant a spellchecking library for Python. Accessible sur https://pythonhosted.org/pyenchant. ↗
  10. K YLE K., C ROSSLEY S. & B ERGER C. (2018). The tool for the automatic analysis of lexical sophistication (TAALES) : version 2.0. Behavior Research Methods, 50(3), 1030–1046. find
  11. L U X. (2014). Computational Methods for Corpus Annotation and Analysis. Dordrecht : Springer. find
  12. M ERKEL D. (2014). Docker : lightweight linux containers for consistent development and deployment. Linux journal, 2014(239), 2. find
  13. S OUSA A., B ALLIER N., G AILLAT T., S TEARNS B., Z ARROUK M., S IMPKIN A. & B OUYÉ M. (2020). From Linguistic Research Projects to Language Technology Platforms : A Case Study in learner data. In Proceedings of the 1st International Workshop on Language Technology Platforms IWLTP 2020, Marseille : LREC / ACL. find
  14. Z HANG A. (2017). Speech recognition (version 3.8) python library. Accessible sur https://pypi.org/project/SpeechRecognition/. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gaillat-2020-un-prototype-en-ligne-pour/