Elamur Research⌕
☰
home / papers / 2020 / Breaking Through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information

Breaking Through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information

Summary

Neural supervised systems lead Word Sense Disambiguation (WSD), but they make little use of the relations (hypernymy, similarity, derivation, etc.) stored in lexical knowledge bases such as WordNet. They also struggle with senses that are rare or absent in the training data. The authors propose EWISER, which extends EWISE. A frozen BERT Large (cased) encoder supplies contextual vectors, built by summing its top four layers, to a small two-layer feedforward classifier with swish activation. Two kinds of prior knowledge are added. First, the output matrix starts from off-the-shelf pretrained synset embeddings and is then trained further. Second, a "structured logits" step spreads scores along WordNet edges by multiplying the logits with a sparse, normalized adjacency matrix before the softmax. The two additions are complementary and let the model pick synsets never seen in training. EWISER sets a new state of the art on nearly every English all-words setting and is the first system to exceed 80 F1 on the combined benchmark. Trained only on English data, it also leads multilingual WSD. This shows graph knowledge can be cheaply integrated into neural models.

Key points

  • Structured logits are computed as Q = ZAᵀ + Z. In the matrix A, each incoming WordNet edge gets weight 1/N, where N is the target synset's number of incoming links. This prevents bias toward highly connected synsets.
  • The base graph uses similarity, verb group and derivationally related edges. Variants add hypernymy, hyponymy, both, or their transitive closures. The best overall model is EWISER hyper, trained on SemCor plus tagged WordNet glosses and examples.
  • EWISER is the first system to pass 80 F1 on the concatenation of the standard English all-words benchmarks. The authors describe this level as matching the estimated inter-annotator upper bound.
  • In cross-lingual evaluation on datasets updated for BabelNet 4.0.1, it beats SensEmBERT on all four SemEval-2013 datasets, by nearly 8 points on Italian. On SemEval-2015 it beats SyntagRank by 6.1 points on Spanish and 2.8 points on Italian.
  • The model can still miss senses that never appear in SemCor, such as the dental sense of "root". Code and pretrained models are available at github.com/SapienzaNLP/ewiser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics Word sense disambiguation Knowledge graphs & ontologies

 Institutions

🎓Sapienza University of RomeRome · Italy29 works · 6,807 citations

Cite

Cite as Michele Bevilacqua and Roberto Navigli. 2020. Breaking Through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.255

@inproceedings{bevilacqua2020breaking,
  title = {{Breaking Through the 80\% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information}},
  author = {Michele Bevilacqua and Roberto Navigli},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.255},
  url = {https://doi.org/10.18653/v1/2020.acl-main.255}
}

References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre and Aitor Soroa. 2009. Personalizing PageRank for word sense disambiguation. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 33–41, Athens, Greece. Association for Computational Linguistics. find
  2. Michele Bevilacqua and Roberto Navigli. 2019. Quasi Bidirectional Encoder Representations from Transformers for word sense disambiguation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 122– 131, Varna, Bulgaria. INCOMA Ltd. find
  3. Kong, China. Association for Computational Linguistics. find
  4. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015. find
  5. Sawan Kumar, Sharmistha Jat, Karan Saxena, and Partha Talukdar. 2019. Zero-shot word sense disambiguation using sense definition embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5670–5681, Florence, Italy. Association for Computational Lin… find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  7. Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack’s wife Hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5962–5971, Florence, Italy. Associat… find
  8. Philip Edmonds and Scott Cotton. 2001. SENSEVAL-2: Overview. In Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 1–5, Toulouse, France. Association for Computational Linguistics. find
  9. Daniel Loureiro and Alı́pio Jorge. 2019. Language modelling makes sense: Propagating representations through WordNet for full-coverage word sense disambiguation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5682–5691, Florence, Italy. Association… our card
  10. Steffen Eger, Paul Youssef, and Iryna Gurevych. 2018. Is it time to swish? Comparing deep learning activation functions across NLP tasks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4415–4424, Brussels, Belgium. Association for Computational Lingu… find
  11. Fuli Luo, Tianyu Liu, Qiaolin Xia, Baobao Chang, and Zhifang Sui. 2018. Incorporating glosses into neural word sense disambiguation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2473–2482, Melbourne, Australia. Association… find
  12. Christian Hadiwinoto, Hwee Tou Ng, and Wee Chung Gan. 2019. Improved word sense disambiguation using pre-trained contextualized word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua… find
  13. Marco Maru, Federico Scozzafava, Federico Martelli, and Roberto Navigli. 2019. SyntagNet: Challenging supervised word sense disambiguation with lexical-semantic combinations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Con… find
  14. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  15. George A. Miller. 1995. WordNet: A lexical database for english. Commun. ACM, 38(11):39–41. our card
  16. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I… our card
  17. Andrea Moro and Roberto Navigli. 2015. SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 288–297, Denver, Colorado. Association for Computational Linguistics. find
  18. Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics, 2:231– 244. our card
  19. Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Comput. Surv., 41(2):10:1–10:69. find
  20. Roberto Navigli. 2018. Natural Language Understanding: Instructions for (Present and Future) Use. In Proc. of the 27th International Joint Conference on Artificial Intelligence (IJCAI-18), pages 5697–5702, Stockholm, Sweden. find
  21. Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 task 12: Multilingual word sense disambiguation. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages… find
  22. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. find
  23. Tommaso Pasini. 2020. The knowledge acquisition bottleneck problem in multilingual word sense disambiguation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2020. ijcai.org. find
  24. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… our card
  25. Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… find
  26. Mohammad Taher Pilehvar and Nigel Collier. 2016. De-conflated semantic representations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1680–1690, Austin, Texas. Association for Computational Linguistics. find
  27. Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. SemEval-2007 task-17: English lexical sample, SRL and all words. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 87–92, Prague, Czech Republic. Association for Computational Ling… our card
  28. Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017. Neural sequence learning models for word sense disambiguation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1156–1167, Copenhagen, Denmark. Association for Computational Linguistic… find
  29. Prajit Ramachandran, Barret Zoph, and Quoc V. Le. 2018. Searching for activation functions. In Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track. find
  30. Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020. SensEmBERT: Context-enhanced sense embeddings for multilingual word sense disambiguation. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020. find
  31. Federico Scozzafava, Marco Maru, Fabrizio Brignone, Giovanni Torrisi, and Roberto Navigli. 2020. Personalized PageRank with syntagmatic information for multilingual word sense disambiguation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for… find
  32. Benjamin Snyder and Martha Palmer. 2004. The English all-words task. In Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 41–43, Barcelona, Spain. Association for Computational Linguistics. find
  33. Tolga Uslu, Alexander Mehler, Daniel Baumartz, and Wahed Hemati. 2018. FastSense: An efficient word sense disambiguation classifier. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELR… find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998–600… our card
  35. Loı̈c Vial, Benjamin Lecouteux, and Didier Schwab. 2019. Sense vocabulary compression through the semantic knowledge of WordNet for neural word sense disambiguation. In Proceedings of the Global Word-Net Conference, pages 108–117. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bevilacqua-2020-breaking-through-80-glass-ceiling/