Elamur Research⌕
☰
home / papers / 2016 / From distributions to labels: A lexical proficiency analysis using learner corpora

From distributions to labels: A lexical proficiency analysis using learner corpora

Summary

The authors address the following problem: how to derive, from word frequency lists collected from a corpus of essays by learners of Swedish as a second language, a lexicon in which each word is assigned a single CEFR level. Such lists show how a word's usage is distributed across levels A1–C1, so a rule for choosing one level is needed. The authors propose taking the first level at which a word's usage increases noticeably compared to the previous one. Before this, frequencies are adjusted for "diversity", i.e., the number of different authors who used the word. This removes a bias that would otherwise assign basic verbs such as heta or göra an inflated level of B1. Since no gold-standard annotation exists, the result was compared against an independent model. In a Word2Vec space where the essay level is included as an additional context, words of the same level mostly end up close to each other. On five classes, the best model agrees with the frequency-based method in 51–67% of cases, and disagreements mostly concern adjacent levels. This approach yields lexical resources built on the language of the learners themselves rather than of native speakers.

Key points

  • Data: the SweLL corpus — 339 essays at levels A1–C1, about 144K tokens. Each essay was rated by at least two teachers, with inter-rater agreement (Krippendorff's alpha) of 0.80. Lemmatization and morphological annotation were done with the Sparv pipeline.
  • Method: a word's level is determined by the difference in relative frequencies between adjacent levels. The threshold of 0.4 was chosen together with teachers. The diversity measure is the number of occurrences of a word at a level divided by the number of different learners who used it. It is averaged with the normalized frequency.
  • Evaluation: 100 random words were taken, 20 per level, and two tests were run in gensim Word2Vec: word–label and nearest neighbours. The "indexed" model with a window of 60 yielded 51 matches in the first test and 67 for the first nearest neighbour. The non-indexed model with a window of 10 yielded only 18 and 38.
  • Graded picture: the semantic space shows how strongly a word gravitates toward adjacent levels. For example, resa (A2) is closer to B1, while fredag (A2) is closer to A1.
  • Applications: a Språkbanken web tool (larkalabb/texteval) highlights the words of a text by CEFR level. In addition, the list serves as a feature in automated essay scoring systems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Education

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alfter2016distributions,
  title = {{From distributions to labels: A lexical proficiency analysis using learner corpora}},
  author = {Alfter, David and Bizzoni, Yuri and Agebjörn, Anders and Volodina, Elena and Pilán, Ildikó},
  booktitle = {Proceedings of the joint workshop on NLP for Computer Assisted Language Learning and NLP for Language Acquisition},
  year = {2016},
  url = {https://aclanthology.org/W16-6501/}
}

References

9 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Artstein and M. Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  2. Lars Borin, Markus Forsberg, and Johan Roxendal. 2012. Korp - the corpus infrastructure of Språkbanken. In LREC, pages 474–478. Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Press Syndicate of the University of Cambridge. find
  3. Eva Forsbom. 2006. A swedish base vocabulary pool. In Swedish Language Technology conference, Gothenburg. Thomas François, Elena Volodina, Ildikó Pilán, and Anaı̈s Tack. 2016. SVALex: a CEFR-graded Lexical Resource for Swedish Foreign and Second Language Learners. In LREC 2016. Håkan Jansson, So… find
  4. Adam Kilgarriff, Frieda Charalabopoulou, Maria Gavrilidou, Janne Bondi Johannessen, Saussan Khalil, Sofie Johansson Kokkinakis, Robert Lew, Serge Sharoff, Ravikiran Vadlapudi, and Elena Volodina. 2014. Corpus-based vocabulary lists for language learners for nine languages. Language resources and eva… find
  5. K. Krippendorff. 1980. Content Analysis: An Introduction to Its Methodology. Chapter 12. find
  6. Sage, Beverly Hills, CA. Lorena Llozhi. 2016. SweLL list. A list of productive vocabulary generated from second language learners’ essays. Master’s Thesis. University of Gothenburg. find
  7. Tomas Mikolov and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems. Katarina Heimann Mühlenbock and Sofie Johansson Kokkinakis. 2012. SweVoc-a Swedish vocabulary resource for CALL. In Proceedings of the S… find
  8. COLING 2016. Osaka, Japan. find
  9. Elena Volodina, Ildikó Pilán, Stian Rødven Eide, and Hannes Heidarsson. 2014. You get what you annotate: a pedagogically annotated corpus of coursebooks for Swedish as a Second Language. In Proceedings of the third workshop on NLP for computer-assisted language learning at SLTC 2014, Uppsala Unive… find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alfter-2016-distributions-labels-lexical-proficiency-analysis/