Elamur Research⌕
☰
home / papers / 2022 / MWE for Essay Scoring English as a Foreign Language

MWE for Essay Scoring English as a Foreign Language

Paper · 2022 · Rodrigo Wilkens, Daiane Seibert, Xiaoou Wang et al.

Summary

Automated scoring of learner English usually builds its descriptors from single tokens or n-grams. Multiword expressions (MWEs) such as idioms, phrasal verbs, compounds and lexical bundles are rarely used, even though they are tied to proficiency. The authors ask whether MWE-based descriptors help predict a learner's CEFR level. They look at two properties: how often MWEs occur and how concrete or abstract they are.

The data come from EFCAMDAT essays. The sample is limited to the ten most frequent learner nationalities, C1 and C2 are merged because both are sparse, and the essays are downsampled so each nationality contributes equally per level. Next to the MWE features, the authors compute a large set of classic descriptors: word frequency, orthographic neighbours, lexical norms, lexical sophistication, and UD-based POS, morphology and dependency counts from Stanza. They also add verb tense, language development and coherence. Each descriptor is tested on its own for how well it separates levels, and then compared with the others.

Classic features predict level better than MWE features. Still, MWE concreteness shows an encouraging trend across levels. The study is useful because it brings an MWE concreteness resource into interpretable, feature-based proficiency assessment.

Key points

  • The corpus is EFCAMDAT, with over 1M essays from 198 nationalities. It is heavily skewed: about 40% of texts are by Brazilians, 53.04% are at A1 and only 0.16% at C2.
  • The sample keeps the top 10 nationalities and merges C1 with C2. Essays per level per nationality are capped, ranging from 2469 (Brazil, Germany) down to 347 (Taiwan).
  • Linguistic annotation comes from the Stanza parser, giving 17 Universal POS tags, 56 morphological features and 37 UD dependency relations. These are combined into 19 verb-tense features "as they are commonly taught".
  • The lexical feature groups are:
    • 18 frequency features, from EFLLex and SUBTLEX contextual diversity;
    • 8 orthographic/phonological neighbour features;
    • MRC lexical norms (age of acquisition, concreteness, familiarity, imageability) plus out-of-vocabulary rates;
    • 6 sophistication features based on Dale and Chall.
  • MWE concreteness relies on the Muraki et al. (2022) ratings for about 62 thousand English MWEs. The paper's contributions are a concreteness profile per CEFR level, a test of each score's individual power, and a comparison with classic features.

Interesting ideas

  • I1. Rating concreteness at the level of a whole multiword unit, not word by word, captures meaning that crosses word boundaries (e.g. "break the ice" vs "break a bone").
  • I2. Balancing the corpus by capping each nationality at the smallest per-level count removes L1 and level skew as a confound before features are compared.
  • I3. Verb tenses are derived from combinations of UPOS and morphological features, matching the categories teachers use, rather than from raw tags.
  • I4. Each feature family is judged first on its own discriminative power and only then against classic baselines. This shows whether a new signal adds anything beyond strong, cheap predictors such as length.
  • I5. Transparency grading of verb+object combinations (transparent, semi-transparent, non-transparent) is a useful axis for deciding how compositionally to interpret a phrase.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): The compiled lexicon could store concreteness or abstractness for MWE entries, so the world builder can tell a literal physical event from an idiomatic state.
  • I2 (learn): When building silver evaluation sets from skewed sources, capping per stratum keeps metric gains from simply reflecting the majority genre or source.
  • I3 (tokens): A small rule layer could map UD FEATS plus auxiliaries to pedagogical tense labels (present perfect, past continuous), which gives explainable tense output for event ordering.
  • I4 (learn): New features or repair rules could be tested alone and against a trivial baseline, as a negative-control-style check before they are adopted.
  • I5 (frames): Frame building could keep transparent verb-object pairs compositional, while non-transparent ones are mapped to a single lexicalised frame before VerbNet role assignment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksIdioms & multiword expressions · 217 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Rodrigo Wilkens, Daiane Seibert, Xiaoou Wang and Thomas François. 2022. MWE for Essay Scoring English as a Foreign Language. Proceedings of the 2nd Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI) within the 13th Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wilkens2022mwe,
  title = {{MWE for Essay Scoring English as a Foreign Language}},
  author = {Rodrigo Wilkens, Daiane Seibert, Xiaoou Wang and Thomas François},
  booktitle = {Proceedings of the 2nd Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI) within the 13th Language Resources and Evaluation Conference},
  year = {2022},
  url = {https://aclanthology.org/2022.readi-1.9/}
}

References

62 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adamchik, V. A., Hyclak, T. J., Sedlak, P., and Taylor, L. W. (2019). Wage returns to english proficiency in poland. Journal of Labor Research, 40(3):276–295. ↗
  2. Alexopoulou, T., Michel, M., Murakami, A., and Meurers, D. (2017). Task effects on linguistic complexity and accuracy: A large-scale learner corpus analysis employing natural language processing techniques. Language Learning, 67(S1):180–208. ↗
  3. Arnaud, P. J. and Béjoint, H. (1992). Vocabulary and applied linguistics. Springer. ↗
  4. Arnold, T., Ballier, N., Gaillat, T., and Lissón, P. (2018). Predicting cefrl levels in learner english on the basis of metrics and full texts. arXiv preprint arXiv:1806.11099. ↗
  5. Arnon, I. and Snider, N. (2010). More than words: Frequency effects for multi-word phrases. Journal of memory and language, 62(1):67–82. ↗
  6. Balikas, G. (2018). Lexical bias in essay level prediction. arXiv preprint arXiv:1809.08935. ↗
  7. Ballier, N., Gaillat, T., Simpkin, A., Stearns, B., Bouyé, M., and Zarrouk, M. (2019). A supervised learning model for the automatic assessment of language levels based on learner errors. In European Conference on Technology Enhanced Learning, pages 308–320. Springer. find
  8. Ballier, N., Canu, S., Petitjean, C., Gasso, G., Balhana, C., Alexopoulou, T., and Gaillat, T. (2020). Machine learning for learner english: A plea for creating learner data challenges. International Journal of Learner Corpus Research, 6(1):72–103. find
  9. Berzak, Y., Kenney, J., Spadine, C., Wang, J. X., Lam, L., Mori, K. S., Garza, S., and Katz, B. (2016). Universal dependencies for learner english. arXiv preprint arXiv:1605.04278. ↗
  10. Boyd, M. and Cao, X. (2009). Immigrant language proficiency, earnings, and language policies. Canadian Studies in Population [ARCHIVES], 36(1-2):63–86. ↗
  11. Bullinaria, J. A. and Levy, J. P. (2007). Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior research methods, 39(3):510–526. ↗
  12. Calzolari, N., Fillmore, C. J., Grishman, R., Ide, N., Lenci, A., MacLeod, C., and Zampolli, A. (2002). Towards best practice for multiword expressions in computational lexicons. In LREC, volume 2, pages 1934–1940. find
  13. Carroll, J. B. (1964). Language and thought. Reading Improvement, 2(1):80. find
  14. Chaudron, C. and Parker, K. (1990). Discourse markedness and structural markedness: The acquisition of English noun phrases. Studies in second language acquisition, 12(1):43–64. ↗
  15. Chen, X. and Meurers, D. (2016). Ctap: A web-based tool supporting automatic complexity analysis. In Proceedings of the workshop on computational linguistics for linguistic complexity (CL4LC), pages 113–119. find
  16. Chen, X., Alexopoulou, T., and Tsimpli, I. (2021). Automatic extraction of subordinate clauses and its application in second language acquisition research. Behavior Research Methods, 53(2):803–817. ↗
  17. Covington, M. A. and McFall, J. D. (2010). Cutting the gordian knot: The moving-average type– token ratio (mattr). Journal of quantitative linguistics, 17(2):94–100. ↗
  18. Dahlmann, I. and Adolphs, S. (2007). Pauses as an indicator of psycholinguistically valid multi-word expressions (mwes)? In Proceedings of the workshop on a broader perspective on multiword expressions, pages 49–56. find
  19. Dale, E. and Chall, J. S. (1948). A formula for predicting readability: Instructions. Educational research bulletin, pages 37–54. find
  20. Du, X., Afzaal, M., and Al Fadda, H. (2022). Collocation use in efl learners’ writing across multiple language proficiencies: A corpus-driven study. Frontiers in Psychology, 13:752134–752134. ↗
  21. François, T. and Watrin, P. (2011). On the contribution of mwe-based features to a readability formula for french as a foreign language. In Proceedings of the International Conference Recent Advances in Natural Language Processing 2011, pages 441–447. find
  22. Gaillat, T., Simpkin, A., Ballier, N., Stearns, B., Sousa, A., Bouyé, M., and Zarrouk, M. (2021). Predicting cefr levels in learners of english: the use of microsystem criterial features in a machine learning approach. ReCALL, pages 1–17. ↗
  23. Garner, J. R. (2016). A phrase-frame approach to investigating phraseology in learner writing across proficiency levels. International Journal of Learner Corpus Research, 2(1):31–67. ↗
  24. Glucksberg, S. (1989). Metaphors in conversation: How are they understood? why are they used? Metaphor and Symbol, 4(3):125–143. ↗
  25. Graesser, A. C., McNamara, D. S., Louwerse, M. M., and Cai, Z. (2004). Coh-metrix: Analysis of text on cohesion and language. Behavior research methods, instruments, & computers, 36(2):193–202. ↗
  26. Guiraud, P. (1959). Problèmes et méthodes de la statistique linguistique, volume 2. D. Reidel. find
  27. Heafield, K., Pouzyrevsky, I., Clark, J. H., and Koehn, P. (2013). Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 690–696.
  28. Herdan, G. (1960). Type-token mathematics, volume 4. Mouton. find
  29. Herdan, G. (1966). The advanced theory of language as choice and chance. Springer Berlin. ↗
  30. Jackendoff, R. (1997). Twistin’the night away. Language, pages 534–559. find
  31. Jung, Y., Crossley, S., and McNamara, D. (2019). Predicting second language writing proficiency in learner texts using computational tools. The Journal of Asia TEFL, 16(1):37–52. ↗
  32. Kitaev, N., Cao, S., and Klein, D. (2019). Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy, July. Association for Computational Linguistics.
  33. Kneser, R. and Ney, H. (1995). Improved backing-off for m-gram language modeling. In 1995 international conference on acoustics, speech, and signal processing, volume 1, pages 181–184. IEEE. find
  34. Kyle, K. (2016). Measuring syntactic development in L2 writing: Fine grained indices of syntactic complexity and usage-based indices of syntactic sophistication. Ph.D. thesis, Georgia State University, Atlanta, Georgia. find
  35. Lan, G., Zhang, Q., Lucas, K., Sun, Y., and Gao, J. (2022). A corpus-based investigation on noun phrase complexity in l1 and l2 english writing. English for Specific Purposes, 67:4–17. ↗
  36. Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International journal of corpus linguistics, 15(4):474–496. ↗
  37. Michel, M., Murakami, A., Alexopoulou, T., and Meurers, D. (2019). Effects of task type on morphosyntactic complexity across proficiency: Evidence from a large learner corpus of a1 to c2 writings. Instructed Second Language Acquisition, 3(2):124–152. find
  38. Pendakur, K. and Pendakur, R. (2007). Colour my world: Have earnings gaps for canadianborn ethnic minorities changed over time? find
  39. Römer, U. and Berger, C. M. (2019). Observing the emergence of constructional knowledge: Verb patterns in german and spanish learners of english at different proficiency levels. Studies in Second Language Acquisition, 41(5):1089–1110. find
  40. Römer, U. (2019). A corpus perspective on the development of verb constructions in second language learners. International Journal of Corpus Linguistics, 24(3):268–290. ↗
  41. Sag, I. A., Baldwin, T., Bond, F., Copestake, A., and Flickinger, D. (2002). Multiword expressions: A pain in the neck for nlp. In International conference on intelligent text processing and computational linguistics, pages 1–15. Springer.
  42. Srichanyachon, N. (2012). Teacher written feedback for l2 learners’ writing development. Humanities, Arts and Social Sciences Studies (Former Name Silpakorn University Journal of Social Sciences, Humanities, and Arts), pages 7–17. find
  43. Stolcke, A. (2002). Srilm-an extensible language modeling toolkit. In Seventh international conference on spoken language processing. find
  44. Stubbs, M. (2007). An example of frequent english phraseology: distributions, structures and functions. In Corpus linguistics 25 years on, pages 87–105. Brill. ↗
  45. Tack, A., François, T., Roekhaut, S., and Fairon, C. (2017). Human and automated cefr-based grading of short answers. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 169–179. find
  46. Tono, Y. (2013). Automatic extraction of l2 criterial lexico-grammatical features across pseudo-longitudinal learner corpora: using edit distance and variability-based neighbour clustering. L2 vocabulary acquisition, knowledge and use, pages 149–176. find
  47. Vajjala, S. (2018). Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education, 28(1):79–105. ↗
  48. Weigle, S. C. (2013). English language learners and automated scoring of essays: Critical considerations. Assessing Writing, 18(1):85–99. ↗
  49. Wilkens, R., Alfter, D., Wang, X., Pintard, A., Tack, A., Yancey, K., and François, T. (2022). Fabra: French aggregator-based readability assessment toolkit. In Language Resources and Evaluation Conference (LREC). find
  50. Wray, A. (1999). Formulaic language in learners and native speakers. Language teaching, 32(4):213–231. ↗
  51. Wray, A. (2002). Formulaic language and the lexicon. ERIC. ↗
  52. Yannakoudakis, H., Briscoe, T., and Medlock, B. (2011). A new dataset and method for automatically grading esol texts. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 180–189.
  53. Yngve, V. H. (1960). A model and an hypothesis for language structure. Proceedings of the American philosophical society, 104(5):444–466. find
  54. Zilio, L., Wilkens, R., and Fairon, C. (2018). Investigating productive and receptive knowledge: A profile for second language learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3467–3478. 8. find
  55. Language Resource References Baroni, M., Bernardini, S., Ferraresi, A., and Zanchetta, E. (2009). The WaCky wide web: A collection of very large linguistically processed web-crawled corpora. Language resources and evaluation, 43(3):209–226. ↗
  56. Bartlett, S., Kondrak, G., and Cherry, C. (2009). On the syllabification of phonemes. In Proceedings of human language technologies: The 2009 annual conference of the north american chapter of the association for computational linguistics, pages 308– 316. find
  57. Brysbaert, M. and New, B. (2009). Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english. Behavior research methods, 41(4):977– 990. ↗
  58. Coltheart, M. (1981). The mrc psycholinguistic database. The Quarterly Journal of Experimental Psychology Section A, 33(4):497–505. ↗
  59. Dürlich, L. and François, T. (2018). Efllex: A graded lexical resource for learners of english as a foreign language. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  60. Geertzen, J., Alexopoulou, T., and Korhonen, A. (2013). Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat). Selected Proceedings of the 31st Second Language Research Forum (SLRF), Cascadilla Press, MA. find
  61. Muraki, E., Abdalla, S., Brysbaert, M., and Pexman, P. (2022). Concreteness ratings for 62 thousand english multiword expressions. 03. ↗
  62. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., and Manning, C. D. (2020). Stanza: A python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07082. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wilkens-2022-mwe-essay-scoring-english-as/