Elamur Research
☰
home / papers / 2019 / GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge

GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1355.pdf

Summary

Word Sense Disambiguation (WSD) means choosing the correct WordNet sense of an ambiguous word in context. Earlier supervised neural systems used sense definitions (glosses) only weakly and barely beat per-lemma "word expert" classifiers. The authors recast WSD as sentence-pair classification. For each target word, they pair the context sentence with the gloss of every candidate WordNet 3.0 sense. A fine-tuned BERT BASE model then labels each pair yes or no, and the sense with the highest yes probability wins. They test three variants:

  • classifying from the target token's representation;
  • classifying from the [CLS] token;
  • [CLS] plus "weak supervision", where quotation marks highlight the target and its lemma is prepended to the gloss.

Training uses SemCor3.0, and evaluation uses the Raganato et al. (2017b) all-words benchmark. GlossBERT(Sent-CLS-WS) reaches 77.0 F1 on the concatenated test sets, a new state of the art. This shows that one shared gloss-aware model can outperform both feature-engineered word experts and earlier gloss-based neural networks.

Key points

  • Data: SemCor3.0 for training (226,036 annotations); test sets are SE2, SE3, SE13 and SE15, with SE07 (455 instances) as the dev set; glosses come from WordNet 3.0.
  • Setup: uncased BERT BASE (110M parameters), 4 epochs, learning rate 2e-5, batch size 64; BERT LARGE did slightly worse.
  • Results for GlossBERT(Sent-CLS-WS): 77.0 F1 overall vs. 71.1 for HCAN; 72.5 on SE07; 80.4 on SE15; 67.1 on verbs vs. 58.2 for HCAN.
  • A plain BERT(Token-CLS) baseline without glosses scored only 68.6, so the gains come from the gloss pairing rather than from BERT alone.
  • Highlighting the target matters: Token-CLS (76.3) beats Sent-CLS (75.8), and the weakly supervised variant performs best in most settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Word sense disambiguation

Cited by

141 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 140 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1355 · arXiv 1908.07245

@inproceedings{huang2019glossbert,
  title = {{GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge}},
  author = {Huang, Luyao and Sun, Chi and Qiu, Xipeng and Huang, Xuanjing},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1355},
  eprint = {1908.07245},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1355}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Oier López de Lacalle, and Aitor Soroa. 2014. Random walks for knowledge-based word sense disambiguation. Computational Linguistics, 40(1):57–84. find our card
  2. Satanjeev Banerjee and Ted Pedersen. 2002. An adapted lesk algorithm for word sense disambiguation using wordnet. In International conference on intelligent text processing and computational linguistics, pages 136–145. Springer. find
  3. Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro. 2014. An enhanced lesk word sense disambiguation algorithm through a distributional semantic model. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1591–1600. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  5. Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Embeddings for word sense disambiguation: An evaluation study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 897–907. find
  6. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  7. Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017a. Neural sequence learning models for word sense disambiguation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1156–1167. find
  8. Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017b. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99–… find our card
  9. Hui Shen, Razvan Bunescu, and Rada Mihalcea. 2013. Coarse to fine grained sense disambiguation in wikipedia. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, volume 1, pages 22–31… find
  10. Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sh… find our card
  11. Zhi Zhong and Hwee Tou Ng. 2010. It makes sense: A wide-coverage word sense disambiguation system for free text. In Proceedings of the ACL 2010 system demonstrations, pages 78–83. find