Elamur Research⌕
☰
home / papers / 2017 / Empower Sequence Labeling with Task-Aware Neural Language Model

Empower Sequence Labeling with Task-Aware Neural Language Model

Summary

Neural sequence labelers for POS tagging, chunking and NER have many parameters, but annotated corpora are usually small. Extra supervision from related tasks is often unavailable, especially for low-resource languages or niche domains. Large pre-trained language models can supply character-level knowledge, but they need external corpora and very costly training. The authors propose LM-LSTM-CRF, which trains a character-level language model jointly with the tagger on the same training sentences, so no extra annotation or corpus is required.

A shared forward and backward character LSTM predicts the next word at each word boundary. Separate highway layers then map its output into one space for the language model and another for the tagger. These features are concatenated with fine-tuned pre-trained word embeddings and passed to a word-level bi-LSTM with a CRF on top.

Evaluated on CoNLL03 NER, CoNLL00 chunking and WSJ Penn Treebank POS, the model reports state-of-the-art results. On CoNLL03 it reaches F1 91.71±0.10 after about 6 hours on one GPU. This shows that task-guided co-training can replace expensive pre-training.

Key points

  • Benchmarks: CoNLL 2003 NER, CoNLL 2000 chunking, and the WSJ portion of the Penn Treebank for POS tagging. The authors report state-of-the-art results on all three.
  • On CoNLL03 NER, F1 is 91.71±0.10, with training taking roughly 6 hours on a single GPU. No additional annotations are used.
  • For contrast, one language model in the earlier pre-training approach (Peters et al. 2017) needed 32 GPUs for over half a month.
  • The design uses four highway units (forward/backward × LM/SL) with ReLU, a word-level bi-LSTM, and a CRF over BIOES-style labels.
  • Ablations on CoNLL03:
    • Co-training without highway layers (NH) does worse than using highway layers with no LM objective (NL).
    • As hidden size grows, LM-LSTM-CRF keeps improving. LSTM-CNN-CRF peaks at size 200, and LSTM-CRF drops at that size.

Interesting ideas

  • I1. Self-supervision from the labeled data itself. A language-model objective on the training sentences adds a learning signal without any new data. This is attractive for low-resource languages such as Ukrainian, where both annotations and compute are limited.
  • I2. Task-specific adapters over a shared encoder. When the auxiliary and main tasks pull in different directions, give each its own transformation layer (here, highway units) on top of the shared features. The paper shows that naive sharing can actually hurt, so this mediation step is worth copying in any multi-task setup.
  • I3. Word-level LM objective on a character encoder. Predicting the next word only at word boundaries pushes the character LSTM toward lexical features rather than memorizing spelling. This is a cheap way to get morphology-aware representations.
  • I4. Choose the transfer strategy by parameter budget. The word level holds most of the parameters, so pre-trained embeddings are simply fine-tuned. The character level has few parameters, so it is co-trained instead. This split avoids costly full pre-training.
  • I5. Scaling hidden size as an overfitting probe. Comparing how each model behaves as hidden size grows shows whether an auxiliary objective really regularizes the model. It is a simple diagnostic that can be reused in other experiments.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Computing & GPUs

Topics: Named entity recognition · Treebanks · Part-of-speech tagging

Cited by

86 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 81 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.04109 · arXiv 1709.04109

@misc{liu2017empower,
  title = {{Empower Sequence Labeling with Task-Aware Neural Language Model}},
  author = {Liu, Liyuan and Shang, Jingbo and Xu, Frank F. and Ren, Xiang and Gui, Huan and Peng, Jian and Han, Jiawei},
  year = {2017},
  doi = {10.48550/arxiv.1709.04109},
  eprint = {1709.04109},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.04109}
}

References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chieu, H. L., and Ng, H. T. 2002. Named entity recognition: A maximum entropy approach using global information. In COLING. find
  2. Chiu, J. P. C., and Nichols, E. 2016. Named entity recognition with bidirectional lstm-cnns. find our card
  3. TACL. Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu, K.; and Kuksa, P. P. 2011. Natural language processing (almost) from scratch. JMLR. find our card
  4. Fernandez, J.; Yu, Z.; and Downey, D. 2017. Vecshare: A framework for sharing word representation vectors. find
  5. Florian, R.; Ittycheriah, A.; Jing, H.; and Zhang, T. 2003. Named entity recognition through classifier combination. In CoNLL. find our card
  6. Glorot, X., and Bengio, Y. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. find
  7. Hashimoto, K.; Xiong, C.; Tsuruoka, Y.; and Socher, R. 2016. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv:1611.01587. find our card
  8. Jozefowicz, R.; Vinyals, O.; Schuster, M.; Shazeer, N.; and Wu, Y. 2016. Exploring the limits of language modeling. arXiv:1602.02410. find
  9. Jozefowicz, R.; Zaremba, W.; and Sutskever, I. 2015. An empirical exploration of recurrent network architectures. In ICML. find
  10. Karpathy, A. 2015. The unreasonable effectiveness of recurrent neural networks. http://karpathy.github.io/2015/05/21/rnn-effectiveness/. Accessed: 2017-08-22. ↗
  11. Lafferty, J. D.; McCallum, A.; and Pereira, F. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML. find
  12. Lample, G.; Ballesteros, M.; Kawakami, K.; Subramanian, S.; and Dyer, C. 2016. Neural architectures for named entity recognition. In NAACL-HLT. find
  13. Liu, L.; Ren, X.; Zhu, Q.; Zhi, S.; Gui, H.; Ji, H.; and Han, J. 2017. Heterogeneous Supervision for Relation Extraction: A Representation Learning Approach. Proc. EMNLP. find
  14. Luo, G.; Huang, X.; Lin, C.-Y.; and Nie, Z. 2015. Joint named entity recognition and disambiguation. In EMNLP. find
  15. Ma, X., and Hovy, E. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In ACL. find our card
  16. Manning, C. D. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In International Conference on Intelligent Text Processing and Computational Linguistics. Springer. find our card
  17. Marcus, M. P.; Marcinkiewicz, M. A.; and Santorini, B. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics. find our card
  18. McCallum, A., and Li, W. 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In CoNLL. find our card
  19. Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. find
  20. Peng, N., and Dredze, M. 2016. Improving named entity recognition for chinese social media with word segmentation representation learning. In ACL. find
  21. Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In EMNLP. find
  22. Peters, M. E.; Ammar, W.; Bhagavatula, C.; and Power, R. 2017. Semi-supervised sequence tagging with bidirectional language models. arXiv:1705.00108. find our card
  23. Ratinov, L., and Roth, D. 2009. Design challenges and misconceptions in named entity recognition. In CoNLL. find our card
  24. Rei, M. 2017. Semi-supervised multitask learning for sequence labeling. In ACL. find
  25. Reimers, N., and Gurevych, I. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. arXiv preprint arXiv:1707.09861. find our card
  26. Sha, F., and Pereira, F. 2003. Shallow parsing with conditional random fields. In NAACL-HLT. find
  27. Shang, J.; Liu, J.; Jiang, M.; Ren, X.; Voss, C. R.; and Han, J. 2017. Automated phrase mining from massive text corpora. arXiv:1702.04457. — [arXiv:1702.04457](https://arxiv.org/abs/1702.04457) find
  28. Søgaard, A., and Goldberg, Y. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In ACL. find our card
  29. Søgaard, A. 2011. Semisupervised condensed nearest neighbor for part-of-speech tagging. In NAACL-HLT. find
  30. Srivastava, R. K.; Greff, K.; and Schmidhuber, J. 2015. find
  31. Sun, X. 2014. Structure regularization for structured prediction. In NIPS. find
  32. Tjong Kim Sang, E. F., and Buchholz, S. 2000. Introduction to the conll-2000 shared task: Chunking. In Learning language in logic and CoNLL. find our card
  33. Tjong Kim Sang, E. F., and De Meulder, F. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Natural language learning at NAACL-HLT. find
  34. Yang, Z.; Salakhutdinov, R.; and Cohen, W. W. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. arXiv:1703.06345. — [arXiv:1703.06345](https://arxiv.org/abs/1703.06345) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2017-empower-sequence-labeling-task-aware/