Elamur Research⌕
☰
home / papers / 2019 / Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation

Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation

Summary

Word Sense Disambiguation (WSD) has been hard to improve. Annotated data such as SemCor covers only a small part of WordNet, so earlier systems that used contextual embeddings had to fall back on the most frequent sense (MFS) for many words. Loureiro and Jorge build sense-level embeddings from BERT that cover every sense in WordNet 3.0. They start from contextual embeddings of SemCor annotations. They then fill the gaps by spreading these vectors through WordNet's structure (synsets, hypernyms, lexnames) and by adding information from glosses. The resulting resource, LMMS, needs no task-specific model. A plain 1-NN match between a token's contextual embedding and the sense vectors outperforms earlier neural sequence models on the Raganato et al. (2017a) evaluation framework, with no MFS fallback. The authors also test a harder setting that ignores lemma and part-of-speech, and they use the sense vectors to probe what language models know. This matters because it gives a simple, full-coverage way to connect contextual representations to WordNet concepts.

Key points

  • Even a basic k-NN with MFS fallback over BERT embeddings beats all prior systems. Full WordNet coverage adds a further 1.9% F1 and removes the need for the fallback.
  • Coverage spans all of WordNet 3.0: 117,659 synsets, 206,949 senses, 147,306 lemmas and 45 lexnames. Released variants include LMMS 1024 and LMMS 2048.
  • In Uninformed Sense Matching, each token is compared against all 200K+ senses instead of at most 59 candidates. This setting is much harder, but it shows that entity knowledge picked up in pretraining helps disambiguate verbs such as "played".
  • The sense vectors support concept-level analysis of NLMs, such as gender-bias scores that separate senses sharing a lemma (e.g., doctor as PhD vs. medic).
  • Code and data: github.com/danlou/lmms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

TopicsWord sense disambiguationLanguage models & pretraining

Institutions

🎓Universidade do PortoPorto · Portugal9 works · 1,657 citations🔬INESC TECPorto · Portugal7 works · 952 citations

Cite

Cite as Daniel Loureiro and Alípio Jorge. 2019. Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1569 · arXiv 1906.10007

@inproceedings{loureiro2019language,
  title = {{Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation}},
  author = {Daniel Loureiro and Alípio Jorge},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1569},
  eprint = {1906.10007},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1569}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations (ICLR). find
  2. Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro. 2014. An enhanced Lesk word sense disambiguation algorithm through a distributional semantic model. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1591–1600, Dublin, Irel… our card
  3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. our card
  4. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pages 4356–4364, USA. Curran… our card
  5. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  6. Jose Camacho-Collados and Mohammad Taher Pilehvar. 2018. From word to sense embeddings: A survey on vector representations of meaning. J. Artif. Int. Res., 63(1):743–788. find
  7. Jose Camacho-Collados, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Nasari: Integrating explicit knowledge and corpus statistics for a multilingual representation of concepts and entities. Artificial Intelligence, 240:36 – 64. find
  8. Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1025–1035, Doha, Qatar. Association for Computational Linguistics. our card
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805v1. our card
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  11. Christiane Fellbaum. 1998. In WordNet : an electronic lexical database. MIT Press. find
  12. Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Embeddings for word sense disambiguation: An evaluation study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 897–907, Berlin, Germany. Association for Co… find
  13. Minh Le, Marten Postma, Jacopo Urbani, and Piek Vossen. 2018. A deep dive into word sense disambiguation with LSTM. In Proceedings of the 27th International Conference on Computational Linguistics, pages 354–365, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  14. Doug Lenat, Mayank Prakash, and Mary Shepherd. 1986. Cyc: Using common sense knowledge to overcome brittleness and knowledge acquistion bottlenecks. AI Mag., 6(4):65–85. find
  15. Michael Lesk. 1986. Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings of the 5th Annual International Conference on Systems Documentation, SIGDOC ’86, pages 24–26, New York, NY, USA. ACM. find
  16. Daniel Loureiro and Alı́pio Mário Jorge. 2019. Liaad at semdeep-5 challenge: Word-in-context (wic). In SemDeep-5@IJCAI 2019, page forthcoming. find
  17. Fuli Luo, Tianyu Liu, Zexue He, Qiaolin Xia, Zhifang Sui, and Baobao Chang. 2018a. Leveraging gloss knowledge in neural word sense disambiguation by hierarchical co-attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1402–1411, Brussels, Belgiu… find
  18. Fuli Luo, Tianyu Liu, Qiaolin Xia, Baobao Chang, and Zhifang Sui. 2018b. Incorporating glosses into neural word sense disambiguation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2473–2482, Melbourne, Australia. Association… find
  19. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Germany. Association for Computational Linguistics. our card
  20. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems -Volume 2, NIPS’13, pages 3111–3119, USA. Curran As… our card
  21. George A. Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G. Thomas. 1994. Using a semantic concordance for sense identification. In HUMAN LANGUAGE TECHNOLOGY: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994. our card
  22. Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Computing Surveys, 41(2):10:1– 10:69. find
  23. Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017b. Neural sequence learning models for word sense disambiguation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1156–1167, Copenhagen, Denmark. Association for Computational Linguisti… find
  24. Philip Resnik. 1997. Selectional preference and sense disambiguation. In Tagging Text with Lexical Semantics: Why, What, and How? find
  25. Sascha Rothe and Hinrich Schütze. 2015. AutoExtend: Extending word embeddings to embeddings for synsets and lexemes. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long P… find
  26. Loı̈c Vial, Benjamin Lecouteux, and Didier Schwab. 2018. Improving the coverage and the generalization ability of neural word sense disambiguation through hypernymy and hyponymy relationships. CoRR, abs/1811.00960. — [arXiv:1811.00960](https://arxiv.org/abs/1811.00960) find
  27. Dayu Yuan, Julian Richardson, Ryan Doherty, Colin Evans, and Eric Altendorf. 2016. Semi-supervised word sense disambiguation with neural models. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1374–1385, Osaka, Japan. find
  28. The COLING 2016 Organizing Committee. find
  29. Zhi Zhong and Hwee Tou Ng. 2010. It makes sense: A wide-coverage word sense disambiguation system for free text. In Proceedings of the ACL 2010 System Demonstrations, pages 78–83, Uppsala, Sweden. Association for Computational Linguistics. find
  30. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… our card
  31. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  32. Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017a. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99–… our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/loureiro-2019-language-modelling-makes-sense-propagating/