Elamur Research⌕
☰
home / papers / 2023 / A new learner language data set for the study of English for Specific Purposes at university

A new learner language data set for the study of English for Specific Purposes at university

Paper · 2023 · Cyriel Mallart, Nicolas Ballier, Jen-Yu Li et al.

Summary

CELVA.Sp is a new, freely available collection of university learner English built for research on English for Specific Purposes (ESP). The authors argue that existing learner corpora such as ICLE, EFCAMDAT, CLC/FCE, PELIC and NUCLE have several gaps. Their raw files can be hard to obtain, their proficiency labels are often unverified, they carry little information about learning behaviour, and they mostly use general-English prompts. This makes comparisons across academic disciplines difficult.

To address this, mostly French undergraduates in ten fields all answered one prompt. They described an invention, experiment or technique from their own discipline and then gave their opinion on its impact. Their proficiency was placed on the CEFR scale with a DIALANG test. Texts were gathered in class through a custom MOODLE database, then cleaned and pseudonymized to meet GDPR requirements. Finally, they were annotated automatically in the UD framework with UDPipe.

The release contains 671 texts with behavioural metadata, distributed as CoNLL-U embedded in CSV. It supports work on proficiency prediction, error modelling and comparing domain-specific learner profiles.

Key points

  • The corpus has 671 texts by French-L1 students in ten domains, including medicine, pharmacy, computer science, sport and mathematics. Texts average 215 words (SD 116.35) and were written in 45 minutes during 2018–2020 at two universities in the same city.
  • CEFR levels (six in total) come only from the 30 cloze items of DIALANG's writing module. The authors plan expert rating by six certified assessors, with an inter-rater agreement check.
  • Annotation was produced by UDPipe with the GUM-trained English model (english-gum-ud-2.5-191206). It covers UPOS, XPOS, lemmas, FEATS, HEAD and DEPREL. Its accuracy on this data was not evaluated.
  • Each learner is identified by a 64-character HMAC-SHA256 pseudonym of their email, generated with a secret key held by the curators. Personal names inside the texts are replaced with "Alex Dupont".
  • The metadata covers years of study, reading frequency, media exposure, trips abroad, past advanced-English classes and other L2s. The data is hosted on Huma-Num Nakala (DOI 10.34847/nkl.41d57kb0), and the MOODLE collection module is released for reuse.

Interesting ideas

  • I1. Giving every writer one identical prompt that can be adapted to their own field keeps genre and discourse type constant, so domain becomes the only major variable between subsets. This is a clean way to build comparable slices of data.
  • I2. A keyed hash of a stable identifier gives pseudonyms that are unique and repeatable. Only the key holder can find a person's records, which allows tracking someone over time and also deleting their data on request.
  • I3. Personal names are replaced with a natural-looking full name instead of initials or symbols, which keeps the text linguistically realistic for downstream processing.
  • I4. The authors state openly that the UD layer is automatic and unevaluated, and they note that the GUM model is regarded as dependable on both L1 and L2 text. This makes the corpus a natural test case for parser robustness on learner language.
  • I5. Collection fields were restricted to fixed values to prevent inconsistent category names, which reduces cleaning work later.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): build evaluation sets where a single factor changes (domain, register, writer skill) and the task stays the same, so drops in accuracy can be traced to that factor.
  • I2 (learn): tag every training or silver-data source with keyed hashes, so specific documents can later be traced or removed without storing raw identities.
  • I3 (tokens): when anonymizing test texts, use realistic placeholder names so the tagger and NER-like steps see normal proper nouns instead of strange tokens that distort the analysis.
  • I4 (tree): run gparse and the perceptron parser on CELVA.Sp and compare them with the bundled UDPipe-GUM trees, treating disagreements, especially at low CEFR levels, as pointers to learner-English constructions that need repair rules.
  • I5 (world): apply the same controlled-vocabulary discipline when compiling values, measures and entity types into the knowledge layer, so that equivalent states do not end up under several names.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Cyriel Mallart, Nicolas Ballier, Jen-Yu Li, Andrew Simpkin, Bernardo Stearns, Rémi Venant and Thomas Gaillat. 2023. A new learner language data set for the study of English for Specific Purposes at university. Proceedings of the 4th Conference on Language, Data and Knowledge.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mallart2023new,
  title = {{A new learner language data set for the study of English for Specific Purposes at university}},
  author = {Cyriel Mallart, Nicolas Ballier, Jen-Yu Li, Andrew Simpkin, Bernardo Stearns, Rémi Venant and Thomas Gaillat},
  booktitle = {Proceedings of the 4th Conference on Language, Data and Knowledge},
  year = {2023},
  url = {https://aclanthology.org/2023.ldk-1.27/}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Charles Alderson and Ari Huhta. 2005. The development of a suite of computer-based diagnostic tests based on the Common European Framework. Language Testing, 22(3):301–320. ↗
  2. Jeroen Geertzen, Theodora Alexopoulou, and Anna Korhonen. 2013. Automatic Linguistic Annotation of Large Scale L2 Databases: The EF-Cambridge Open Language Database (EFCamDat). In Selected Proceedings of the 2012 Second Language Research Forum: Building Bridges between Disciplines. Cascadilla Press. find
  3. Maxime Amblard, Bruno Guillaume, Siyana Pavlova, and Guy Perrier. 2022. Graph querying for semantic annotations. In Proceedings of the 18th Joint ACL - ISO Workshop on Interoperable Semantic Annotation within LREC2022, pages 95–101. European Language Resources Association. ↗
  4. Mihir Bellare, Ran Canetti, and Hugo Krawczyk. 1996. Keying Hash Functions for Message Authentication. In Advances in Cryptology, pages 1–15. Springer. find
  5. Gaëtanelle Gilquin. 2015. From design to collection of learner corpora. In The Cambridge Handbook of Learner Corpus Research, Cambridge Handbooks in Language and Linguistics, pages 9–34. Cambridge University Press. ↗
  6. Marcus Callies. 2015. Learner corpus methodology. In The Cambridge Handbook of Learner Corpus Research, Cambridge Handbooks in Language and Linguistics, pages 35–56. Cambridge University Press. ↗
  7. Sylviane Granger, Maı̈té Dupont, Fanny Meunier, Hubert Naets, and Magali Paquot. 2020. The International Corpus of Learner English. Version 3. Presses universitaires de Louvain. find
  8. Zellig S. Harris. 1954. Distributional structure. WORD, 10(2-3):146–162. find
  9. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. volume 3944, pages 177– 190, Berlin, Heidelberg. Springer Berlin Heidelberg. Book Title: Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment Series Title: Lecture Notes in Computer Science. find
  10. Alan Juffs, Na-Rae Han, and Ben Naismith. 2020. The University of Pittsburgh English Language Institute Corpus (PELIC). find
  11. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751. Association for Computational Linguistics.
  12. Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a Large Annotated Corpus of Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. 2021. Revisiting Few-sample BERT Fine-tuning.
  13. Kristopher Kyle, Masaki Eguchi, Aaron Miller, and Theodore Sither. 2022. A Dependency Treebank of Spoken Second Language English. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 39–45, Seattle, Washington. Association for Computational Linguistics.
  14. Yann LeCun, Y. Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, 521:436–44. find
  15. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, (2):255–308.
  16. Magali Paquot, Tove Larsson, Hilde Hasselgård, Signe O. Ebeling, Damien De Meyere, Larry Valentin, Natalia J. Laso, Isabel Verdaguer, and Sanne van Vuuren. 2022. The Varieties of English for Specific Purposes dAtabase (VESPA): Towards a multi-L1 and multi-register learner corpus of disciplinary writing. Research in Corpus Linguistics, 10(2):1–15. ↗
  17. Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of the Tenth International Conference on Language Resources and Evaluation, pages 4290–4297. European Language Resources Association. find
  18. Anaı̈s Tack, Thomas François, Sophie Roekhaut, and Cédrick Fairon. 2017. Human and Automated CEFR-based Grading of Short Answers. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 169–179. Association for Computational Linguistics. find
  19. Joel Tetreault, Jill Burstein, Ekaterina Kochmar, Claudia Leacock, and Helen Yannakoudakis, editors. 2018. Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, New Orleans, Louisiana. find
  20. H. Yannakoudakis, Øe Andersen, A. Geranpayeh, T. Briscoe, and D. Nicholls. 2018. Developing an automated writing placement system for ESL learners. Accepted: 2019-02-16T00:31:04Z Publisher: Informa UK Limited. ↗
  21. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, pages 180–189. Association for Computational Linguistics.
  22. Amir Zeldes. 2017. The gum corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mallart-2023-new-learner-language-data-set/