基于语言模型增强的中文关系抽取方法

Summary

Chinese relation extraction (RE) classifies the semantic relation between two entities in a sentence. The best results come from fine-tuned pre-trained language models, but those models have very large parameter counts and are expensive to store and run. The authors propose a cheaper alternative based on multi-task learning. A lightweight character-level BiLSTM relation classifier also trains a bidirectional language model that shares its embedding and encoder layers. Ordinary language models learn to predict the next character. Here, the language model instead learns to match the masked-character probability distributions produced by BERT-wwm, using KL divergence as the loss. This transfers general linguistic knowledge from BERT-wwm into the small model. The authors frame the effect as a form of automatic data augmentation, since the model is exposed to alternative phrasings that express the same relation. The method was evaluated on SanWen, ACE2004 and ACE2005. It beat lexicon-based approaches and reached about 95% of the pre-trained model's performance with roughly 1% of its parameters. This makes it attractive where compute is limited.

Key points

• Architecture: The base model combines 100-dimensional pre-trained character embeddings with 10-dimensional relative-position embeddings for the head and tail entities. These feed a BiLSTM with hidden size 200 per direction, followed by a softmax classifier. The language-model heads reuse the shared encoder.
• Teacher signal: BERT-wwm (whole-word masking, 12-layer Transformer, pre-trained on Chinese Wikipedia) supplies soft labels for each masked character. These are truncated to the top 100 candidates and renormalized.
• Training objective: The total loss is the RE cross-entropy plus λ times the forward and backward KL terms, with λ = 0.01. Training used Adam with a learning rate of 0.0005 and dropout of 0.5.
• Data:
  • SanWen: literary texts, 10 relation types.
  • ACE2005: news, 18 relation types.
  • ACE2004: news, 19 relation types.
  • The ACE corpora use a 75/25 split, following Li et al.
  • Results are reported as F1 and AUC.
• Baselines and positioning: The method was compared with a CNN model (Zeng et al.), an LSTM model (Zhang et al.), MG-Lattice and Flat-Lattice. It differs from knowledge distillation because it learns task-independent language knowledge from a pre-trained model, rather than from a task-specific RE teacher.