Elamur Research⌕
☰
home / papers / 2021 / katildakat at SemEval-2021 Task 1: Lexical Complexity Prediction of Single Words and Multi-Word Expressions in English

katildakat at SemEval-2021 Task 1: Lexical Complexity Prediction of Single Words and Multi-Word Expressions in English

Paper · 2021 · Katja Voskoboinik

Summary

This paper describes a submission to SemEval-2021 Task 1, Lexical Complexity Prediction (LCP). The task asks for a score from 0 to 1 for how hard a target is for native speakers, given its context. Targets are either single words or two-word MWEs (adjective–noun or noun–noun). The data is the CompLex dataset, which covers Bible, Europarl and biomedical text. The author does not build separate systems for the two tracks. Both target types are treated as one kind of lexical unit and described with the same features: averaged BERT target and context embeddings plus seven hand-crafted features. A linear regression and a small MLP are trained on the pooled data. Joint training helped, most clearly for MWEs. Classic word frequency remained the strongest single predictor even next to contextual embeddings. Feature rankings also differed between single words and MWEs. The work suggests that MWE complexity can be learned well by borrowing data from single words, and that interpretable features stay relevant.

Key points

  • Representation:
    • BERT base (cased) embeddings. The target vector (subword-averaged where needed, always averaged for MWEs) is averaged with the [CLS] context vector, giving 768 dimensions.
    • Seven hand-crafted features are appended, for 775 dimensions in total.
    • The features are WordPiece token count, masked-target BERT log-probability, Morfessor morph count, morph frequency, WordNet synset count, character length, and wordfreq Zipf frequency.
  • Models and tools:
    • scikit-learn linear regression and an MLP with one ReLU layer of 20 units and alpha 0.9.
    • An optional reduction to the 400 best dimensions, chosen by F-score for the linear model and by mutual information for the MLP.
    • Morfessor 2.0 was trained on COCA samples, excluding the Academic texts.
  • Joint-training results (Pearson, single words / MWEs):
    • LR, all features: 0.666 / 0.783
    • NN, all features: 0.712 / 0.785
    • LR, 400 dims: 0.688 / 0.796
    • NN, 400 dims: 0.707 / 0.774
  • Separately trained systems scored LR 0.669 / 0.678 and NN 0.706 / 0.752. MWEs therefore gained clearly from pooling.
  • Feature importance:
    • Word frequency ranked first everywhere, with BERT token count second.
    • Morph frequency was weak, ranking only 110th by mutual information for MWEs.
    • Character length mattered for MWEs but not for single words.

Interesting ideas

  • I1. Treating MWEs and single words as the same kind of unit and training on both lets the small MWE set benefit from the much larger single-word set.
  • I2. The number of WordPiece pieces a word splits into works as a free, implicit rarity signal, because the tokenizer merges only frequent strings.
  • I3. A masked language model's probability for the target in its slot measures how predictable the word is in its context, separately from its general frequency.
  • I4. Rank features separately for each subset (single words, MWEs, joint) and with two criteria (F-score for linear relations, mutual information for nonlinear ones) to see which signals matter where.
  • I5. Interpretable features like frequency and WordNet polysemy stay competitive alongside contextual embeddings, so a transparent model need not give up much accuracy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Multiword lexicon entries in the knowledge layer could get properties such as difficulty or familiarity estimated from a model trained jointly with single lemmas, instead of from sparse MWE-only data.
  • I2 (tokens): The en tagger and lemmatizer could use subword fragmentation as a cheap flag for rare or unknown words, routing them to morphological fallback rules with an explainable reason.
  • I3 (back): When generating text from the meaning graph, candidate wordings could be scored by in-context predictability so the generator prefers simpler, more expected words for easy-reading output.
  • I4 (learn): Evaluation reports could break down which explainable features drive decisions per construction type, exposing cases where a rule fits words but fails on multiword units.
  • I5 (answer): School-course readers could estimate how hard a question's vocabulary is from frequency and synset counts, giving a traceable difficulty signal without a black-box model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Aalto UniversityEspoo · Finland13 works · 648 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksIdioms & multiword expressions · 217 works

Cite

Cite as Katja Voskoboinik. 2021. katildakat at SemEval-2021 Task 1: Lexical Complexity Prediction of Single Words and Multi-Word Expressions in English. Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.semeval-1.91

@inproceedings{voskoboinik2021katildakat,
  title = {{katildakat at SemEval-2021 Task 1: Lexical Complexity Prediction of Single Words and Multi-Word Expressions in English}},
  author = {Katja Voskoboinik},
  booktitle = {Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)},
  year = {2021},
  doi = {10.18653/v1/2021.semeval-1.91},
  url = {https://doi.org/10.18653/v1/2021.semeval-1.91}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Alfter and Ildikó Pilán. 2018. SB@GU at the complex word identification 2018 shared task. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 315–321, New Orleans, Louisiana. Association for Computational Linguistics. find
  2. David Alfter and Elena Volodina. 2018. Towards single word lexical complexity prediction. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 79–88, New Orleans, Louisiana. Association for Computational Linguistics. find
  3. Sandra Aluisio, Lucia Specia, Caroline Gasperin, and Carolina Scarton. 2010. Readability assessment for text simplification. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–9, Los Angeles, California. Association for Computational Linguistics.
  4. Hélène Deacon, Michael Kieffer, and Annie Laroche. 2014. The relation between morphological awareness and reading comprehension: Evidence from mediation and longitudinal models. Scientific Studies of Reading, 18. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  6. Christiane Fellbaum, editor. 1998. WordNet: an electronic lexical database. MIT Press. find
  7. Lilli Kimppa, Yury Shtyrov, Suzanne Hut, Laura Hedlund, Miika Leminen, and Alina Leminen. 2019. Acquisition of l2 morphology by adult language learners. Cortex, 116. ↗
  8. Gustavo Paetzold and Lucia Specia. 2016. SemEval 2016 task 11: Complex word identification. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 560– 569, San Diego, California. Association for Computational Linguistics.
  9. Sarah E. Petersen and Mari Ostendorf. 2007. Text simplification for language learners: A corpus analysis. In In Proceedings of Workshop on Speech and Language Technology for Education. find
  10. Iria del Rı́o. 2019. Linguistic features and proficiency classification in L2 Spanish and L2Portuguese. In Proceedings of the 8th Workshop on NLP for Computer Assisted Language Learning, pages 31–40, Turku, Finland. LiU Electronic Press. find
  11. Matthew Shardlow. 2013. A comparison of techniques to automatically identify complex words. In 51st Annual Meeting of the Association for Computational Linguistics Proceedings of the Student Research Workshop, pages 103–109, Sofia, Bulgaria. Association for Computational Linguistics.
  12. Matthew Shardlow, Michael Cooper, and Marcos Zampieri. 2020. Complex: A new corpus for lexical complexity predicition from likert scale data. In Proceedings of the 1st Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI). find
  13. Matthew Shardlow, Richard Evans, Gustavo Paetzold, and Marcos Zampieri. 2021. Semeval-2021 task 1: Lexical complexity prediction. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2021). ↗
  14. Robyn Speer, Joshua Chin, Andrew Lin, Sara Jewett, and Lance Nathan. 2018. Luminosoin-sight/wordfreq: v2.2. find
  15. Peter; Grönroos Stig-Arne; Kurimo Mikko Virpioja, Sami; Smit. 2013. Morfessor 2.0: Python implementation and extensions for morfessor baseline. D4 julkaistu kehittämis- tai tutkimusraportti tai -selvitys. find
  16. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  17. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. ↗
  18. Victoria Yaneva, Irina Temnikova, and Ruslan Mitkov. 2016. Evaluating the readability of text simplification output for readers with cognitive disabilities. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 293–299, Portorož, Slovenia. European Language Resources Association (ELRA). find
  19. Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo Paetzold, Lucia Specia, Sanja Štajner, Anaı̈s Tack, and Marcos Zampieri. 2018. A report on the complex word identification shared task 2018. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 66–78, New Orleans, Louisiana. Association for Computational Linguistics. find
  20. Seid Muhie Yimam, Sanja Štajner, Martin Riedl, and Chris Biemann. 2017. CWIG3G2 - complex word identification task across three text genres and two user groups. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 401–407, Taipei, Taiwan. Asian Federation of Natural Language Processing. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/voskoboinik-2021-katildakat-semeval-2021-task-1/