📄 SenseBERT: Driving Some Sense into BERT

Summary

Self-supervised language models such as BERT learn from word forms, which are only an ambiguous stand-in for meaning. For example, "bass" may denote a fish or an instrument. Levine et al. introduce SenseBERT, which adds a second pre-training objective: besides recovering each masked token, the model must predict its WordNet supersense. No sense-annotated corpus is required. WordNet lists the coarse categories each vocabulary item can take, and when a word has several candidates, a soft-labeling loss rewards probability placed on any of them. A supersense mapping is learned alongside BERT's word embedding matrix and also contributes to the input embeddings.

The authors evaluate on SemEval-SS, a supersense version of the Raganato et al. (2017) WSD suite. SenseBERT clearly outperforms BERT there, with the largest gains when only a linear classifier is trained on frozen embeddings. A single SenseBERT LARGE model also set a new state of the art on WiC.

The work shows that weak supervision from an expert-built lexical resource can make pre-trained representations more meaning-aware, while overall GLUE performance stays roughly unchanged.

Key points

• Supersense inventory: the model uses 45 WordNet supersense categories (26 noun, 15 verb, 3 adjective, 1 adverb). Short words, stop words and sub-word pieces receive no supersense label.
• SemEval-SS setup: models are trained on SemCor (226,036 examples) and tested on SenseEval/SemEval data (7,253 examples).
  • Frozen setting: SenseBERT BASE scores 75.6 vs. 65.1 for BERT BASE, and SenseBERT LARGE scores 79.5 vs. 67.3 for BERT LARGE.
  • Fine-tuned setting: SenseBERT BASE reaches 83.0, above BERT LARGE at 81.1.
• WiC (SuperGLUE): SenseBERT LARGE scores 72.1, about 2.5 points above BERT LARGE (69.6). It also beats RoBERTa (69.9), KnowBERT-W+W (70.9) and the BERT sense-embedding approach (67.7).
• GLUE: the average score is essentially unchanged, at 77.9 for SenseBERT BASE vs. 77.5 for the authors' own BERT BASE.