Elamur Research
☰
home / papers / 2016 / Deep multi-task learning with low level tasks supervised at lower layers

Deep multi-task learning with low level tasks supervised at lower layers

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-2038.pdf

Summary

Earlier deep multi-task learning (MTL) systems in NLP, such as Collobert et al. (2011), trained every task from the same top layer of the network. Søgaard and Goldberg ask whether a linguistic hierarchy, in which low-level analyses inform higher-level ones, should instead shape where each task is supervised. They use a greedy sequence tagger built on a 3-layer bi-LSTM, implemented with the CNN library and initialized with pre-trained embeddings. Each task gets its own softmax classifier attached to a chosen layer. POS tagging serves as the auxiliary task for syntactic chunking and CCG supertagging, with data from the English Penn Treebank. Training POS at the innermost layer and the main task at the outermost layer beats both a single-task model and conventional top-layer MTL. The same placement also helps domain adaptation on OntoNotes 4.0, where only POS labels exist in the target domains. Together, these results offer a simple way to merge cascaded pipelines with shared-representation learning, and they show that layer placement is a meaningful design choice.

Key points

  • Main results. Each figure lists, in order: single-task, both tasks on layer 3, and POS on layer 1 with the main task on layer 3.
    • Chunking F1 was 95.28, 95.30 and 95.56. The best score exceeds Suzuki and Isozaki (2008) at 95.15 and Collobert et al. (2011) MTL at 94.10.
    • CCG supertagging accuracy was 91.04, 92.94 and 93.26. The best score edges past Xu et al. (2015) at 93.00.
  • Data splits. All data come from the Penn Treebank.
    • POS and CCG train on sections 0–18; chunking trains on sections 15–18.
    • Section 19 is used for development.
    • Section 20 is the chunking test set; section 23 is the CCG test set.
    • CCG is scored only on the 425 most frequent supertags.
  • Domain adaptation. WSJ newswire is the source domain, and four OntoNotes 4.0 genres are the targets: broadcast, broadcast news, magazines and weblogs.
    • Lower-layer POS supervision gave the highest chunking F1 in every target genre, with gains of about 0.2–1.4 points over the baseline.
    • Supervising POS at the top layer gave little or no benefit.
  • Rademacher complexity. This measures how well a model fits randomly relabeled chunking data, reported relative to chance. Values were 1.298 (single-task), 1.034 (top-layer MTL) and 0.990 (lower-layer POS), which suggests the auxiliary task acts as a regularizer.
  • Tasks that did not benefit. Adding NER (CoNLL 2003), SemCor supersenses, or the Streusle corpus (MWE and supersense annotation) brought no gains. The authors take this as evidence that MTL helps mainly when the tasks are closely related, such as syntactic ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

139 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 126 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-2038

@inproceedings{sogaard2016deep,
  title = {{Deep multi-task learning with low level tasks supervised at lower layers}},
  author = {Søgaard, Anders and Goldberg, Yoav},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-2038},
  url = {https://doi.org/10.18653/v1/p16-2038}
}

References

13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Junho Yim, Heechul Jung, ByungIn Yoo amd Changkyu Choi, Dusik Park, and Junmo Kim. 2015. Rotating Your Face Using Multi-task Deep Neural Network. In CVPR. find
  2. Rich Caruana. 1993. Multitask learning: a knowledge-based source of inductive bias. In ICML. find
  3. Kyunghyun Cho. 2015. Natural language understanding with distributed representation. CoRR, abs/1511.07916. — [arXiv:1511.07916](https://arxiv.org/abs/1511.07916) find
  4. Cha Zhang and Zhengyou Zhang. 2014. Improving Multiview Face Detection with Multi-Task Deep Convolutional Neural Networks. In WACV. find
  5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
  6. Jerry Zhu, Timothy Rogers, and Bryan Gibson. 2009. Human Rademacher complexity. In NIPS. find
  7. Jeffrey L. Elman. 1990. Finding Structure in Time. Cognitive Science, 14(2):179–211, March. find
  8. Yoav Goldberg. 2015. A primer on neural network models for natural language processing. CoRR, abs/1510.00726. find
  9. Sepp Hochreiter and Juergen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735–1780. find
  10. Ozan Irsoy and Claire Cardie. 2014. Opinion Mining with Deep Recurrent Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 720–728, Doha, Qatar, October. Association for Computational Linguistics. find
  11. M.-T. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser. 2015. Multi-task Sequence to Sequence Learning. ArXiv e-prints, November. find our card
  12. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan Mc-Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. find
  13. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic, June. Association for Computational Linguistics. find