Elamur Research⌕
☰
home / papers / 2017 / Deep Semantic Role Labeling with Self-Attention

Deep Semantic Role Labeling with Self-Attention

Summary

The paper addresses semantic role labeling (SRL): for each predicate in a sentence, finding its arguments and assigning them roles such as ARG0, ARG1 or AM-TMP. Earlier end-to-end systems were built on deep LSTMs. The authors argue that recurrent models struggle with long-distance relations and sentence structure, because they squeeze the whole history into one fixed-size state and process tokens strictly in order. They propose DeepAtt, a stack of identical blocks. Each block pairs a nonlinear sub-layer (BiLSTM, gated convolution, or feed-forward) with a multi-head self-attention sub-layer, so any two tokens are linked directly. The model sees only the words plus a binary mask marking the predicate. It treats SRL as BIO tagging and decodes with a plain softmax and argmax. On the CoNLL-2005 and CoNLL-2012 shared task datasets, a single model reaches new state-of-the-art F1 scores. The feed-forward variant is also very fast. The work matters because it shows that attention-based encoders can replace recurrent ones for structured sequence labeling without any syntactic input.

Key points

  • Single-model results are F1 = 83.4 on CoNLL-2005 and F1 = 82.7 on CoNLL-2012, which beat the previous best systems by 1.8 and 1.0 points. On out-of-domain data, the gain over He et al. (2017) is 2.0 F1.
  • The final networks are 10 layers deep and use residual connections with layer normalization. Attention follows the multi-head scaled dot-product formulation of Vaswani et al. (2017).
  • The three nonlinear sub-layer options are:
    • a BiLSTM whose two directions are summed rather than concatenated;
    • a GLU convolution with filter width 3;
    • a two-layer ReLU feed-forward network with 800 hidden units.
  • The feed-forward variant processes about 50K tokens per second on one Titan X GPU. Source code is released at github.com/XMUNLP/Tagger.
  • In the error analysis, span identification barely changes compared with He et al. (2017) (91.87 vs 91.92). Role classification on gold spans improves much more (87.10 vs 88.88), so finding constituent boundaries remains the weak spot. Confusions between ARG2 and AM-DIR, AM-LOC and AM-MNR are reduced but still present.

Interesting ideas

  • I1. Minimal input representation. The model uses only word embeddings plus a 0/1 predicate-mask embedding, with no syntax, context windows or hand-made features. The same trick suits any task conditioned on one target token, such as relation or event-argument tagging in languages without good parsers.
  • I2. Alternating attention with a nonlinear block. Attention outputs are only weighted averages, so the authors pair each attention step with a nonlinear transform to add expressive power. This sub-layer is interchangeable (RNN/CNN/FFN), which makes it easy to trade accuracy against speed without redesigning the model.
  • I3. Dropping structured decoding. Instead of a CRF or constrained decoding, they use independent per-token argmax. BIO violations are repaired by a simple rule: the label of the B tag decides the label of the whole span. This keeps inference fast and simple, and could be worth trying first on other tagging tasks before adding heavier decoders.
  • I4. Splitting the error source. The evaluation separates span-finding accuracy from role-classification accuracy given gold spans, then adds a confusion matrix over frequent labels. This shows exactly which stage limits performance, a diagnostic that transfers well to NER, chunking or any span-labeling evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Computing & GPUs

Topics: Semantic role labeling

Cited by

78 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 71 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1712.01586 · arXiv 1712.01586

@misc{tan2017deep,
  title = {{Deep Semantic Role Labeling with Self-Attention}},
  author = {Tan, Zhixing and Wang, Mingxuan and Xie, Jun and Chen, Yidong and Shi, Xiaodong},
  year = {2017},
  doi = {10.48550/arxiv.1712.01586},
  eprint = {1712.01586},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1712.01586}
}

References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. find our card
  2. Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  3. Bastianelli, E.; Castellucci, G.; Croce, D.; and Basili, R. 2013. Textual inference and meaning representation in human robot interaction. In Proceedings of the Joint Symposium on Semantic Processing. Textual Inference and Structures in Corpora, 65–69. find
  4. Bengio, Y.; Ducharme, R.; Vincent, P.; and Janvin, C. 2003. A neural probabilistic language model. Journal of Machine Learning Research 3:1137–1155. Carreras, X., and Màrquez, L. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Comp… find
  5. Cheng, J.; Dong, L.; and Lapata, M. 2016. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733. find our card
  6. Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu, K.; and Kuksa, P. 2011. Natural language processing (almost) from scratch. Journal of Marchine Learning Research 2493–2537. find our card
  7. Dan, S., and Lapata, M. 2007. Using semantic roles to improve question answering. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. find our card
  8. Dauphin, Y. N.; Fan, A.; Auli, M.; and Grangier, D. 2016. Language modeling with gated convolutional networks. arXiv preprint arXiv:1612.08083. FitzGerald, N.; Täckström, O.; Ganchev, K.; and Das, D. 2015. Semantic role labeling with neural network factors. In Proceedings of the Conference on Empi… find our card
  9. Gehring, J.; Auli, M.; Grangier, D.; Yarats, D.; and Dauphin, Y. N. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122. find our card
  10. Genest, P.-E., and Lapalme, G. 2011. Framework for abstractive summarization using text-to-text generation. In Proceedings of the Workshop on Monolingual Text-To-Text Generation, 64–73. Association for Computational Linguistics. find
  11. Gildea, D., and Jurafsky, D. 2002. Automatic labeling of semantic roles. Computational linguistics 28(3):245–288. find our card
  12. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778. find
  13. He, L.; Lee, K.; Lewis, M.; and Zettlemoyer, L. 2017. Deep semantic role labeling: What works and whats next. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find our card
  14. Kingsbury, P.; Palmer, M.; and Marcus, M. 2002. Adding semantic annotation to the penn treebank. In Proceedins of the human language technology conference. find
  15. Knight, K., and Luk, S. K. 1994. Building a large-scale knowledge base for machine translation. In AAAI, volume 94, 773–778. find
  16. Koomen, P.; Punyakanok, V.; Roth, D.; and Yih, W.-t. 2005. Generalized inference with multiple semantic role labeling systems. In Proceedings of the 9th Conference on Computational Natural Language Learning, 181–184. Lin, Z.; Feng, M.; Santos, C. N. d.; Yu, M.; Xiang, B.; Zhou, B.; and Bengio, Y. 20… find our card
  17. Luong, M.-T.; Pham, H.; and Manning, C. D. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. find
  18. Marcheggiani, D.; Frolov, A.; and Titov, I. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. arXiv preprint arXiv:1701.02593. find our card
  19. Moschitti, A.; Morarescu, P.; and Harabagiu, S. M. 2003. Open domain information extraction via automatic semantic labeling. In FLAIRS Conference’03, 397–401. find
  20. Nair, V., and Hinton, G. E. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), 807–814. find
  21. Palmer, M.; Gildea, D.; and Xue, N. 2010. Semantic Role Labeling. Synthesis Lectures on Human Language Technology Series. Morgan and Claypool. Parikh, A. P.; Täckström, O.; Das, D.; and Uszkoreit, J. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.019… find our card
  22. Pascanu, R.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2013. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026. find
  23. Paulus, R.; Xiong, C.; and Socher, R. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304. find our card
  24. Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods on Natural Language Processing, 1532–1543. find
  25. Pradhan, S.; Hacioglu, K.; Ward, W.; Martin, J. H.; and Jurafsky, D. 2005. Semantic role chunking combining complementary syntactic views. In Proceedings of the 9th Conference on Computational Natural Language Learning, 217– 220. Pradhan, S.; Moschitti, A.; Xue, N.; Ng, H. T.; Björkelund, A.; Uryup… find
  26. Shen, T.; Zhou, T.; Long, G.; Jiang, J.; Pan, S.; and Zhang, C. 2017. Disan: Directional self-attention network for rnn/cnn-free language understanding. arXiv preprint arXiv:1709.04696. find
  27. Srivastava, N.; Hinton, G. E.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research 15(1):1929–1958. find
  28. Surdeanu, M.; Harabagiu, S.; Williams, J.; and Aarseth, P. 2003. Using predicate-argument structures for information extraction. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, 8–15. find our card
  29. Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2818–2826. Täckström, O.; Ganchev, K.; and Das, D. 2015. Efficient inference and struc… find
  30. Toutanova, K.; Haghighi, A.; and Manning, C. D. 2008. A global joint model for semantic role labeling. Computational Linguistics 34:161–191. find
  31. Ueffing, N.; Haffari, G.; and Sarkar, A. 2007. Transductive learning for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 25–32. find
  32. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. find
  33. Wu, D., and Fung, P. 2009. Semantic roles for smt: a hybrid two-pass model. In Proceedings of Human Language Technologies, 13–16. Association for Computational Linguistics. find
  34. Zeiler, M. D. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card
  35. Zhou, J., and Xu, W. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tan-2017-deep-semantic-role-labeling-self/