Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1604.05529

Summary

The paper examines how well bi-LSTM part-of-speech taggers work outside English. It tests many languages, several input granularities, small training sets and noisy labels. The authors use a hierarchical bi-LSTM. A lower sequence bi-LSTM builds a word vector from character or unicode byte embeddings. That vector is joined with a word embedding and passed to a context bi-LSTM that assigns tags. They also propose a multi-task variant, FREQBIN, which predicts each token's tag and a binned log frequency of the token in the training data. The extra objective pushes the network to represent rare and common words differently. The models are tested on 22 UD v1.2 treebanks and the WSJ and compared with TnT (an HMM tagger) and a CRF tagger. Combining word and character inputs gives the best representation, and FREQBIN reaches state-of-the-art accuracy. It helps most with out-of-vocabulary words in morphologically rich languages. The work matters because neural taggers turn out to need less training data than commonly assumed, though they degrade more under heavy label noise.

Key points

Data: UD v1.2 with 17 universal POS tags, using every language with at least 60k tokens (22 languages), plus WSJ with 45 tags. Baselines are TnT and a crfsuite-based CRF. Code is released as bplank/bilstm-aux (built on CNN/pycnn).
Macro-average accuracy: TnT 94.61, CRF 94.27, word-only bi-LSTM 92.37, w + ~c with POLYGLOT 96.50 and FREQBIN 96.52. FREQBIN is best on 12 of 22 languages.
On OOV tokens, FREQBIN raises average accuracy from 83.48 to 87.98. Gains are strongest for Arabic, Farsi, Hebrew and Finnish. Most of the benefit from sub-token input falls on rare words in Slavic and non-Indo-European languages.
The words-plus-characters model beats TnT on every language except Indonesian, with gains above 2 points for Hebrew and Slovene. On WSJ it scores 97.22, comparable to earlier neural taggers.
TnT wins with very little data. The bi-LSTM becomes competitive at about 500 training sentences, or about 100 for non-Indo-European languages. When more than 30% of labels are corrupted, the bi-LSTM loses more accuracy than TnT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Part-of-speech tagging

Cited by

168 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 157 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1604.05529 · arXiv 1604.05529

@misc{plank2016multilingual,
  title = {{Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss}},
  author = {Plank, Barbara and Søgaard, Anders and Goldberg, Yoav},
  year = {2016},
  doi = {10.48550/arxiv.1604.05529},
  eprint = {1604.05529},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1604.05529}
}

References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed Word Representations for Multilingual NLP. In CoNLL. find our card
Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs. In EMNLP. find
Thorsten Brants. 2000. Tnt: a statistical part-of-speech tagger. In Proceedings of the sixth conference on Applied natural language processing. find our card
Hao Cheng, Hao Fang, and Mari Ostendorf. 2015. Open-Domain Name Error Detection using a Multi-Task RNN. In EMNLP. find
Kyunghyun Cho. 2015. Natural language understanding with distributed representation. ArXiv, abs/1511.07916. — [arXiv:1511.07916](https://arxiv.org/abs/1511.07916) find
Grzegorz Chrupała. 2013. Text segmentation with character-level text embeddings. In Workshop on Deep Learning for Audio, Speech and Language Processing, ICML. find
Michael Collins. 2002. Discriminative training methods for Hidden Markov Models. In EMNLP. find
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2016. Multilingual language processing from bytes. In NAACL. find
Yoav Goldberg. 2015. A primer on neural network models for natural language processing. ArXiv, abs/1510.00726. find
Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610. find
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
Tobias Horsmann, Nicolai Erbs, and Torsten Zesch. 2015. Fast or accurate?–a comparative evaluation of pos tagging models. In Proceedings of the International Conference of the German Society for Computational Linguistics and Language Technology. find
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991. find our card
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2015. Character-aware neural language models. arXiv preprint arXiv:1508.06615. find our card
E. Kiperwasser and Y. Goldberg. 2016. Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations. ArXiv e-prints. find our card
Matthieu Labeau, Kevin Löser, and Alexandre Allauzen. 2015. Non-lexical neural architecture for fine-grained pos tagging. In EMNLP. find
Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In EMNLP. find our card
Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In EMNLP. find
Christopher D Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In Computational Linguistics and Intelligent Text Processing. Springer. find our card
Joakim Nivre, Željko Agić, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Riyaz Ahmad Bhat, Cristina Bosco, Sam Bowman, Giuseppe G. A. Celano, Miriam Connor, Marie-Catherine de Marneffe, Arantza Diaz de Ilarraza, Kaja Dobrovoljc, Timothy D… find
Barbara Plank, Dirk Hovy, Ryan McDonald, and Anders Søgaard. 2014. Adapting taggers to twitter using not-so-distant supervision. In COLING. find
Cicero D Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In ICML. find
Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recurrent neural networks. In ICML. find
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
Peilu Wang, Yao Qian, Frank K. Soong, Lei He, and Hai Zhao. 2015. Part-of-speech tagging with bidirectional long short-term memory recurrent neural network. pre-print, abs/1510.06168. — [arXiv:1510.06168](https://arxiv.org/abs/1510.06168) find
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649–657. find our card
Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In ACL. find