Elamur Research⌕
☰
home / papers / 2022 / Comparing Native and Learner Englishes Using a Large Pre-trained Language Model

Comparing Native and Learner Englishes Using a Large Pre-trained Language Model

Summary

The paper looks for an automatic way to assess the depth of lexical knowledge among learners of English as an L2. Depth is understood here as how close learners' use of a word is to native speakers' use. Existing automatic metrics mostly capture only vocabulary size. Depth tests, such as tests on collocations or synonyms, are costly and scale poorly. The author proposes comparing contextualized word embeddings (CWE) from BERT. For the 1000 most frequent words, he computes vector centroids separately in the native-speaker corpus LOCNESS and in essays by Japanese learners from EFCAMDAT, divided into 16 proficiency levels. He then divides the distance between the centroids of the two corpora by the within-corpus distance. As the level rises, word use generally approaches the native-speaker norm. At the highest levels the distance increases slightly. Words with similar use often reveal their discourse function. The largest discrepancies, by contrast, are mostly explained by the different topics of the corpora. The work shows that CWE can be used for large-scale comparison of language communities, but the topical domain must be carefully controlled.

Key points

  • Data. From EFCAMDAT, only texts by Japanese learners were taken: 3,441 authors, 21,374 essays, ≈1.6 million words, levels 1–16. From LOCNESS, only argumentative essays by US students were taken: 176 texts, 149,574 words. Average essay length differs greatly: ≈75 words in EFCAMDAT versus ≈850 in LOCNESS.
  • Model and metric. BERT is used via the flair library: a whole sentence is given as input, and vectors are taken from the last layer. The score is the ratio of the between-corpus distance between centroids to the within-corpus distance. The comparison is made on 100 words that are in the top 1000 in all 17 corpora and subcorpora.
  • Main result. The average distance to native-speaker use decreases as the level rises. The exception is levels 12–16, where it increases slightly. The author attributes this either to unequal samples and essay topics or to a genuine U-shaped curve, i.e., the phenomenon of backsliding.
  • Word examples. For basketball, the distance between corpora is almost five times the within-corpus distance. Native speakers write this word in the context of college sports, learners in the context of hobbies and daily routine. The word actually proved sensitive to domain, whereas even did not.
  • Next steps. The author proposes analyzing other BERT layers, aligning the corpora by domain, and using multilingual BERT to test whether learners' vectors shift toward the equivalent of the word in their native language (L1 influence).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Education

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{aoyama2022comparing,
  title = {{Comparing Native and Learner Englishes Using a Large Pre-trained Language Model}},
  author = {Aoyama, Tatsuya},
  booktitle = {Proceedings of the 11th Workshop on NLP for Computer Assisted Language Learning},
  year = {2022},
  doi = {10.3384/ecp190001},
  url = {https://doi.org/10.3384/ecp190001}
}

References

32 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54–59. find our card
  2. Marco Del Tredici and Raquel Fernández. 2017. Semantic variation in online communities of practice. In IWCS 2017 - 12th International Conference on Computational Semantics - Long papers. find
  3. Marco Del Tredici, Raquel Fernández, and Gemma Boleda. 2019. Short-term meaning shift: A distributional exploration. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2069–2075, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073–1094, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  6. Katrin Erk. 2012. Vector space models of word meaning and phrase meaning: A survey. Language and Linguistics Compass, 6(10):635–653. ↗
  7. Jeroen Geertzen, Theodora Alexopoulou, Anna Korhonen, et al. 2013. Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat). In Proceedings of the 31st Second Language Research Forum. Somerville, MA: Cascadilla Proceedings Project, pages 240–254. Citeseer. find
  8. Xiaofei Lu. 2012. The relationship of lexical richness to the quality of esl learners’ oral narratives. The Modern Language Journal, 96(2):190–208. ↗
  9. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. ↗
  10. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia. Association for Computational Linguistics. find our card
  11. Jennifer M Goldschneider and Robert M DeKeyser. 2001. Explaining the “natural order of l2 morpheme acquisition” in english: A meta-analysis of multiple determinants. Language learning, 51(1):1–50. find
  12. Sylviane Granger. 2014. The computer learner corpus: a versatile new source of data for sla research. In Learner English on computer, pages 3–18. Routledge. ↗
  13. James Milton. 2009. Measuring second language vocabulary acquisition. In Measuring Second Language Vocabulary Acquisition. Multilingual Matters. ↗
  14. William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1501, Berlin, Germany. Association for Computational Linguistics. find our card
  15. Ian SP Nation. 2001. Learning vocabulary in another language. Cambridge university press. ↗
  16. John M Norris and Lourdes Ortega. 2009. Towards an organic approach to investigating caf in instructed sla: The case of complexity. Applied linguistics, 30(4):555–578. ↗
  17. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. find
  18. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. find
  19. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  20. Manfred Pienemann. 1998. Language processing and second language development: Processability theory, volume 15. John Benjamins Publishing. find
  21. Yan Huang, Jeroen Geertzen, Rachel Baker, Anna Korhonen, Theodora Alexopoulou, and EF Education First. 2017. The ef cambridge open language database (efcamdat): Information for users. find
  22. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. ↗ our card
  23. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics. find our card
  24. G Salton. 1971. The smart system. Retrieval Results and Future Plans, 260. find
  25. Larry Selinker. 1972. Interlanguage. International Review of Applied Linguistics in Language Teaching, 10(1-4):209–231. find
  26. James P Lantolf and Ali Aljaafreh. 1995. Second language learning in the zone of proximal development: A revolutionary experience. International Journal of Educational Research, 23(7):619–632. ↗
  27. Larry Selinker and John T Lamendella. 1981. Updating the interlanguage hypothesis. Studies in Second Language Acquisition, 3(2):201–220. ↗
  28. Jennifer Sikos and Sebastian Padó. 2018. Using embeddings to compare FrameNet frames across languages. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pages 91–101, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  29. Peter Skehan et al. 1998. A cognitive approach to language learning. Oxford University Press. find
  30. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. ↗ our card
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. find
  32. Marjorie Wesche and T Sima Paribakht. 1996. Assessing second language vocabulary knowledge: Depth versus breadth. Canadian Modern Language Review, 53(1):13–40. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aoyama-2022-comparing-native-learner-englishes-large/