Elamur Research⌕
☰
home / papers / 2023 / Morphosyntactic probing of multilingual BERT models

Morphosyntactic probing of multilingual BERT models

Paper · 2023 · Judit Acs, Endre Hamerlik, Roy Schwartz et al.

Summary

The paper asks how much morphological knowledge multilingual masked language models hold and where in a sentence the cues for a word's morphological tag are found. The authors built a probing benchmark from the Universal Dependencies treebanks. It has 247 tasks covering 42 languages from 10 families. In each instance, a target word in its sentence must be labelled with a morphological feature value, and ambiguous forms are included. They trained lightweight probes on mBERT and XLM-RoBERTa representations and compared them with a character-level LSTM baseline (chLSTM) that has no pre-training. To locate the useful information, they used two methods. The first is a new perturbation scheme that masks the target, the left or right context, or shuffles word order. The second is Shapley values computed per context position. Both pre-trained models clearly beat the baseline. The target word carries most of the signal. Preceding words matter more than following ones, even though Transformers have no built-in directional bias. This points to a property of language rather than of the models.

Key points

  • The dataset has 247 probing tasks across 42 languages and 10 families, derived from UD treebanks. Data, code and results are available at github.com/juditacs/morphology-probes.
  • mBERT and XLM-RoBERTa make about a third fewer errors than task-specific baselines with a comparable number of trained parameters. They stay strong even when parts of the input are masked.
  • The left context outweighs the right in 172 of 247 tasks (binomial sign test p ≈ 6.26·10⁻⁵). Exceptions, notably in the Indic family, have linguistic explanations, and context masking barely affects Uralic tasks.
  • XLM-RoBERTa's vocabulary is roughly twice the size of mBERT's, and its embedding layer appears better suited to multilingual morphology. It achieves higher accuracy at the embedding layer and gains little from the first Transformer layer.
  • Morphological tags and POS categories generalize better than individual languages, as measured by the variance of Shapley values. Patterns of perturbation effects partly recover the standard typological grouping of languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations🎓Hebrew University of JerusalemJerusalem · Israel65 works · 9,387 citations🎓Budapest University of Technology and EconomicsBudapest · Hungary4 works · 688 citations🎓Comenius University BratislavaBratislava · Slovakia2 works · 26 citations🔬HUN-REN Institute for Computer Science and ControlBudapest · Hungary2 works · 26 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksMorphology · 405 worksInterpretability · 130 works

Cite

Cite as Judit Acs, Endre Hamerlik, Roy Schwartz, Noah A. Smith and Andras Kornai. 2023. Morphosyntactic probing of multilingual BERT models. arXiv:2306.06205 · Nat. Lang. Eng. 30 (2024) 753-792.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1017/s1351324923000190 · arXiv 2306.06205

@misc{acs2023morphosyntactic,
  title = {{Morphosyntactic probing of multilingual BERT models}},
  author = {Judit Acs, Endre Hamerlik, Roy Schwartz, Noah A. Smith and Andras Kornai},
  year = {2023},
  doi = {10.1017/s1351324923000190},
  eprint = {2306.06205},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1017/s1351324923000190}
}

References

94 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ács, J. 2019. Exploring BERT’s vocabulary. http://juditacs.github.io/2019/02/19/ bert-tokenization-stats.html. Accessed: 2023-03-09. ↗
  2. Ács, J., Kádár, Á., and Kornai, A. 2021. Subword pooling makes a difference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2284–2295, Online. Association for Computational Linguistics. find
  3. Adelaar, A. 2005. The Austronesian languages of Asia and Madagascar: A historical perspective. The Austronesian languages of Asia and Madagascar, 1:1–42. find
  4. Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
  5. OpenReview.net. Anderson, S. 2005. Aspects of the Theory of Clitics Oxford. find
  6. Oxford University Press. Anderson, S. R., Brown, L., Gaby, A., and Lecarme, J. 2006. Life on the edge: There’s morphology there after all! Lingue e linguaggio, 5(1):33–48. find
  7. Arnardóttir, Þ., Hafsteinsson, H., Sigurðsson, E. F., Bjarnadóttir, K., Ingason, A. K., Jónsdóttir, H., and Steingrímsson, S. 2020. A Universal Dependencies conversion pipeline for a Penn-format constituency treebank. In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pp. 16…
  8. Arps, D., Samih, Y., Kallmeyer, L., and Sajjad, H. 2022. Probing for constituency structure in neural language models. arXiv:2204.06201 [cs]. arXiv: 2204.06201. find
  9. Belinkov, Y. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219.
  10. Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., and Glass, J. 2017a. What do Neural Machine Translation Models Learn about Morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 861–872, Vancouver, Canada. Association for…
  11. Belinkov, Y. and Glass, J. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72.
  12. Belinkov, Y., Màrquez, L., Sajjad, H., Durrani, N., Dalvi, F., and Glass, J. 2017b. Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Lo…
  13. Ben Zaken, E., Goldberg, Y., and Ravfogel, S. 2022. BitFit: Simple parameter-efficient fine-tuning for Transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1–9, Dublin, Ireland. Association…
  14. Bengio, Y., Ducharme, R., and Vincent, P. 2000. A neural probabilistic language model. Advances in Neural Information Processing Systems, 13. find
  15. Bisazza, A. and Tump, C. 2018. The Lazy Encoder: A Fine-Grained Analysis of the Role of Morphology in Neural Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2871– 2876, Brussels, Belgium. Association for Computational Linguistics. find
  16. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146.
  17. Brill, E. 1993. Automatic grammar induction and parsing free text: A transformation-based approach. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993. find
  18. Brown, L. 2001. A grammar of Nias Selatan. Can, B., Aleçakır, H., Manandhar, S., and Bozşahin, C. 2022. Joint learning of morphology and syntax with cross-level contextual information flow. Natural Language Engineering, pp. 1–33. find
  19. Chi, E. A., Hewitt, J., and Manning, C. D. 2020. Finding Universal Grammatical Relations in Multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5564–5577, Online. Association for Computational Linguistics. find
  20. Choueka, Y. and Lusignan, S. 1985. Disambiguation by short contexts. Computers and the Humanities, 19(3):147–157. find
  21. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440… find
  22. Conneau, A. and Kiela, D. 2018. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  23. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364.
  24. Conneau, A., Kruszewski, G., Lample, G., Barrault, L., and Baroni, M. 2018a. What you can cram into a single \\$&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2…
  25. Conneau, A. and Lample, G. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 7057–7067. find
  26. Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., and Stoyanov, V. 2018b. XNLI: Evaluating Cross-lingual Sentence Representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475–2485, Brussels, Belgium. Association for Co… find
  27. Corbett, G. G. 1998. Morphology and agreement. In The Handbook of Morphology, pp. 191–205. find
  28. Blackwell. Cotterell, R., Kirov, C., Sylak-Glassman, J., Walther, G., Vylomova, E., Xia, P., Faruqui, M., Kübler, S., Yarowsky, D., Eisner, J., and Hulden, M. 2017. find
  29. The CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, Vancouver, Canada. Association for Computational Linguistics.
  30. Deal, A. R. 2015. Interaction and satisfaction in ϕ-agreement. In Proceedings of the 45th New England Linguistic Society Annual Meetin, pp. 179–192. find
  31. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 4171–4186, Minneapolis, Minnesota. As…
  32. Edmiston, D. 2020. A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages. arXiv:2004.03032 [cs]. arXiv: 2004.03032. — [arXiv:2004.03032](https://arxiv.org/abs/2004.03032) find
  33. Ethayarajh, K. and Jurafsky, D. 2021. Attention flows are Shapley value explanations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 49–54, Online. Ass… find
  34. Ettinger, A. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48.
  35. Ettinger, A., Elgohary, A., and Resnik, P. 2016. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 134–139, Berlin, Germany. Association for Computational Linguistics.
  36. Gage, P. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23–38. find
  37. Goldberg, Y. 2019. Assessing BERT’s Syntactic Abilities. arXiv:1901.05287 [cs].
  38. Gupta, A., Boleda, G., Baroni, M., and Padó, S. 2015. Distributional vectors encode referential attributes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 12–21. find
  39. Harris, F. J. 1978. On the use of windows for harmonic analysis with the DFT. Proc. IEEE, 66(1):51–83. find
  40. Hart, B. and Risley, T. R. 1995. Meaningful Differences in the Everyday Experience of Young American Children. find
  41. Brookes Publishing Company, Inc. Hewitt, J., Ethayarajh, K., Liang, P., and Manning, C. 2021. Conditional probing: measuring usable information beyond a baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1626–1639, Online and Punta Cana, Dominica… find
  42. Hewitt, J. and Liang, P. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733–2743, Hong Kong, China. As… find
  43. Hewitt, J. and Manning, C. D. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4129–4138, Minne…
  44. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research,… find
  45. PMLR. Householder, F. 1981. The syntax of Apollonius Dyscolus. find
  46. John Benjamins. Htut, P. M., Phang, J., Bordia, S., and Bowman, S. R. 2019. Do attention heads in BERT track syntactic dependencies? In NY Academy of Sciences NLP, Dialog, and Speech Workshop. find
  47. Jawaid, B. and Zeman, D. 2011. Word-order issues in English-to-Urdu statistical machine translation. The Prague Bulletin of Mathematical Linguistics, 95(95):87–106. find
  48. Kádár, Á., Chrupala, G., and Alishahi, A. 2017. Representation of Linguistic Form and Function in Recurrent Neural Networks. Computational Linguistics, 43(4):761–780.
  49. Kann, K. and Schütze, H. 2016. MED: The LMU system for the SIGMORPHON 2016 shared task on morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pp. 62–70, Berlin, Germany. Association for Computational Linguistic… find
  50. Karpathy, A. and Fei-Fei, L. 2017. Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Pattern Anal. Mach. Intell., 39(4):664–676. find
  51. Kingma, D. P. and Ba, J. 2015. Adam: A method for stochastic optimization. In 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, conference track proceedings.
  52. Kiparsky, P. 2009. On the architecture of Panini’s grammar. In Sanskrit Computational Linguistics, pp. 33–94. find
  53. Kitaev, N., Cao, S., and Klein, D. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics, pp. 3499–3505, florence, italy. Association for Computational Linguistics.
  54. Kolda, T. G. and Bader, B. W. 2009. Tensor decompositions and applications. SIAM review, 51(3):455–500. find
  55. Kondratyuk, D. and Straka, M. 2019. 75 languages, 1 model: parsing Universal Dependencies universally. In Proceedings of the 2019 EMNLP-IJCNLP conference, pp. 2779–2795, hong kong, china. Association for Computational Linguistics.
  56. Kote, N., Biba, M., Kanerva, K., Rönnqvist, S., and Ginter, F. 2019. Morphological tagging and lemmatization of Albanian: a manually annotated corpus and neural models. find
  57. Kudo, T. and Richardson, J. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66–71, Brussels, Belgium. Association…
  58. Kurimo, M., Virpioja, S., Turunen, V., and Lagus, K. 2010. Morpho challenge competition 2005–2010: evaluations and results. In Proceedings of the 11th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, pp. 87–95. Association for Computational Linguistics. find
  59. Köhn, A. 2015. What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2067–2073, Lisbon, Portugal. Association for Computational Linguistics. find
  60. Lapointe, S. 1992. Life on the edge: argument in favor of an autolexical account of edge inflections. In Proceedings of the Chicago Linguistic Society, volume 28, pp. 318–332. find
  61. Lapointe, S. G. 1990. Edge features in GPSG. In Papers from the 26th Regional Meeting of the Chicago Linguistic Society, pp. 221–235. find
  62. Chicago Linguistic Society. Li, P.-H., Fu, T.-J., and Ma, W.-Y. 2020. Why attention? Analyze BiLSTM deficiency and its remedies in the case of NER. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8236–8244. find
  63. Li, X. L. and Liang, P. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582–4597, Onl… find
  64. Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., and Smith, N. A. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 1073–1094, Minneapolis, Minn…
  65. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. 2019b. RoBERTa: A robustly optimized BERT pretraining approach.
  66. Loshchilov, I. and Hutter, F. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  67. Luhn, H. P. 1959. Keyword-in-Context Index for Technical Literature (KWIC Index). American Documentation, 11(4):288– 295. find
  68. Lundberg, S. M. and Lee, S. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4765–4774.
  69. Mikhailov, V., Serikov, O., and Artemova, E. 2021. Morph call: Probing morphosyntactic content of multilingual Transformers. In Proceedings of the Third Workshop on Computational Typology and Multilingual NLP, pp. 97–121, Online. Association for Computational Linguistics. find
  70. Mikolov, T., Chen, K., Corrado, G., and Dean, J. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  71. Nemeskey, D. M. 2020. Natural Language Processing Methods for Language Modeling. PhD thesis, Eötvös Loránd University. find
  72. Nichols, J. 1986. Head-marking and dependent-marking grammar. Language, 62:56–119. Nivre, J., de Marneffe, M.-C., Ginter, F., Hajič, J., Manning, C. D., Pyysalo, S., Schuster, S., Tyers, F., and Zeman, D. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedin…
  73. European Language Resources Association. Pennington, J., Socher, R., and Manning, C. D. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543.
  74. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 2227–2237, New Orleans, Louisiana. Ass…
  75. Qian, P., Qiu, X., and Huang, X. 2016. Investigating Language Universal and Specific Properties in Word Embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1478–1488, Berlin, Germany. Association for Computational Lingui… find
  76. Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., and Huang, X. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872–1897.
  77. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. 2018. Improving language understanding by generative pre-training. Preprint. Work in progress. find
  78. Ravichander, A., Belinkov, Y., and Hovy, E. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3363–3377, Online. Association for Computation…
  79. Ravishankar, V., Gökırmak, M., Øvrelid, L., and Velldal, E. 2019. Multilingual probing of deep pre-trained contextual encoders. In Proceedings of the First NLPL Workshop on Deep Learning for Natural Language Processing, pp. 37–47, Turku, Finland. find
  80. Linköping University Electronic Press. Reif, E., Yuan, A., Wattenberg, M., Viegas, F. B., Coenen, A., Pearce, A., and Kim, B. 2019. Visualizing and measuring the geometry of BERT. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. Şahin, G. G., Vania, C., Kuzne…
  81. Shapiro, N. T., Paullada, A., and Steinert-Threlkeld, S. 2021. A multilabel approach to morphosyntactic probing. arXiv preprint arXiv:2104.08464. find
  82. Shapley, L. S. 1951. Notes on the n-person game – ii: The value of an n-person game. Technical report, RAND Corporation. find
  83. Sharan, V., Khakade, S., Liang, P., and Valiant, G. 2018. Prediction with a short memory. In Proc. find
  84. Shi, X., Padhi, I., and Knight, K. 2016. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1526–1534, Austin, Texas. Association for Computational Linguistics.
  85. Sinha, K., Parthasarathi, P., Pineau, J., and Williams, A. 2021. UnNatural Language Inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7329–7346,…
  86. Srivastava, R. K., Greff, K., and Schmidhuber, J. 2015. Highway networks. arXiv preprint arXiv:1505.00387.
  87. Tenney, I., Das, D., and Pavlick, E. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019) 4593-4601.
  88. Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R. T., Kim, N., Durme, B. V., Bowman, S. R., Das, D., and Pavlick, E. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations,…
  89. OpenReview.net. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. 2017. attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
  90. Veldhoen, S., Hupkes, D., and Zuidema, W. H. 2016. Diagnostic classifiers revealing how neural networks process hierarchical structure. In CoCo@ NIPS. find
  91. Voita, E. and Titov, I. 2020. Information-theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 183–196, Online. Association for Computational Linguistics.
  92. Warstadt, A., Cao, Y., Grosu, I., Peng, W., Blix, H., Nie, Y., Alsop, A., Bordia, S., Liu, H., Parrish, A., Wang, S.-F., Phang, J., Mohananey, A., Htut, P. M., Jeretic, P., and Bowman, S. R. 2019. Investigating BERT’s knowledge of language: Five analysis methods with NPIs. In Proceedings of the 2019…
  93. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnic… find
  94. Zhang, K. W. and Bowman, S. R. 2018. Language modeling teaches you more syntax than translation does: Lessons learned through auxiliary task analysis. arXiv preprint arXiv:1809.10040. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/acs-2023-morphosyntactic-probing-multilingual-bert-models/