Elamur Research⌕
☰
home / papers / 2018 / What can we learn from Semantic Tagging?

What can we learn from Semantic Tagging?

Summary

The authors test whether semantic tagging helps other NLP tasks when used as an auxiliary task in multi-task learning (MTL). Semantic tagging is the annotation of words with language-neutral semantic categories from the Parallel Meaning Bank project. There are three main tasks: UPOS tagging, UD parsing and NLI. For each, a single-task model is compared with three ways of sharing parameters. The first is a fully shared network (FSN). The second is a partially shared network (PSN). The third is the Learning What to Share (LWS) scheme: each task has its own and a shared subspace, and a learned sigmoid gate determines how much information flows between them in each direction. The base models are a bi-LSTM tagger, a biaffine parser based on Dozat and Manning, and ESIM. For parsing and NLI, semantic tags were assigned by an automatic tagger trained on sem-PMB. LWS performs best on all four datasets. Full parameter sharing helps the syntactic tasks but hurts NLI. Thus, word-level semantic annotation is useful even for sentence-level tasks, provided the model itself chooses what to take from the auxiliary task. The authors also propose a general framework for assessing the usefulness of any auxiliary task.

Key points

  • The semantic tag set contains 80 tags in 13 coarse classes. It distinguishes negation, modality, quantifier types, named entity types, and verb tense and aspect.
  • Data: UD 2.0 English for UPOS and parsing, sem-PMB 0.1.0 (silver for training, gold for testing), SNLI and SICK-E for NLI. For UD, SNLI and SICK-E, semantic tags were assigned automatically by the Stanford POS Tagger trained on sem-PMB.
  • Results, single-task model → LWS: UPOS 92.12 → 95.54, UD DEP (LAS/UAS) 80.24/84.87 → 81.39/86.00, SNLI 87.01 → 87.51, SICK-E 81.30 → 84.57.
  • On NLI, the FSN scheme performs worse than the single-task model. On the small SICK-E, accuracy drops to 56.69. For NLI, only LWS yields an improvement, and it is largest on the smaller SICK-E dataset.
  • Control experiment on SNLI: the architecture was kept unchanged, but the auxiliary task was given zero weight. PSN and LWS then fall to the level of the single-task model. Hence the gain comes from the semantic tag signal itself, not from the more complex architecture.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1526 · arXiv 1808.09716

@inproceedings{abdou2018what,
  title = {{What can we learn from Semantic Tagging?}},
  author = {Abdou, Mostafa and Kulmizev, Artur and Ravishankar, Vinit and Abzianidze, Lasha and Bos, Johan},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1526},
  eprint = {1808.09716},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1526}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Ludmann, Duc-Duy Nguyen, and Johan Bos. 2017. The parallel meaning bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the 15th Conference of… find our card
  2. Lasha Abzianidze and Johan Bos. 2017. Towards universal semantic tagging. In Proceedings of the 12th International Conference on Computational Semantics (IWCS 2017) – Short Papers, pages 1–6, Montpellier, France. find
  3. Jonathan Baxter et al. 2000. A model of inductive bias learning. J. Artif. Intell. Res.(JAIR), 12(149-198):3. find
  4. Yonatan Belinkov, Lluı́s Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. arXiv preprint arXiv:1801.07772. find our card
  5. Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic tagging with deep residual networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3531–3541. find
  6. The COLING 2016 Organizing Committee. find
  7. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computational Linguistics. find
  8. Rich Caruana. 1998. Multitask learning. In Learning to learn, pages 95–133. Springer. find
  9. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1657–1668. find
  10. Yoeng-Jin Chu. 1965. On the shortest arborescence of a directed graph. Science Sinica, 14:1396–1400. find
  11. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  12. Timothy Dozat and Christopher D Manning. 2016. Deep biaffine attention for neural dependency parsing. arXiv preprint arXiv:1611.01734. find our card
  13. Timothy Dozat, Peng Qi, and Christopher D Manning. 2017. Stanford’s graph-based neural dependency parser at the conll 2017 shared task. Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 20–30. find our card
  14. Jack Edmonds. 1967. Optimum branchings. Journal of Research of the national Bureau of Standards B, 71(4):233–240. find
  15. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2016. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. find our card
  16. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  17. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  18. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. — [arXiv:1605.05101](https://arxiv.org/abs/1605.05101) find
  19. Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. 2017. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5334–5343. find
  20. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330. find our card
  21. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of compositional distributional semantic models. In LREC, pages 216–223. find
  22. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, et al. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computat… find
  23. Elliot Meyerson and Risto Miikkulainen. 2017. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. arXiv preprint arXiv:1711.00108. — [arXiv:1711.00108](https://arxiv.org/abs/1711.00108) find
  24. Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3994–4003. find
  25. Joakim Nivre, Lars Ahrenberg Zeljko Agic, et al. 2017. Universal dependencies 2.0. lindat/clarin digital library at the institute of formal and applied linguistics, charles university, prague. find
  26. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  27. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. arXiv preprint arXiv:1604.05529. find our card
  28. Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2017. Sluice networks: Learning what to share between loosely related tasks. arXiv preprint arXiv:1705.08142. — [arXiv:1705.08142](https://arxiv.org/abs/1705.08142) find
  29. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 231–235. find our card
  30. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume… find

↑ back to top

This page in Markdown →