Elamur Research⌕
☰
home / papers / 2017 / A Recurrent Neural Model with Attention for the Recognition of Chinese Implicit Discourse Relations

A Recurrent Neural Model with Attention for the Recognition of Chinese Implicit Discourse Relations

Summary

The paper addresses sense classification of implicit discourse relations in Chinese, where no connective signals the relation. Implicit relations are far more frequent in Chinese than in English, yet Chinese discourse parsing has received little research attention. Earlier neural systems from the CoNLL 2016 shared task relied on feedforward networks that discard word order. The authors instead propose a bidirectional LSTM with an attention layer. It reads Arg1 and Arg2 as one concatenated sequence, with special boundary markers, and uses only word embeddings as input, initialized from 300-dimensional Chinese Gigaword vectors. They also introduce partial argument sampling, a data augmentation method that adds single-argument training instances alongside the full pairs. On the CDTB sense-only track, the model achieves state-of-the-art test accuracy, beating word order-agnostic baselines. Attention visualizations reveal which tokens drive each decision. The work shows that recurrent, sequence-aware modeling can be competitive for this task, while remaining simple, interpretable and largely language-independent.

Key points

  • The model labels 257 of 352 implicit relations correctly on the CoNLL 2016 CDTB test set, an accuracy of 73.01%. The previous best system, the feedforward model of Wang and Lan (2016), reached 72.42%.
  • In the ablation study, removing attention lowers test accuracy by 2.70 points, and removing partial argument sampling lowers it by 5.74 points (both p < .001).
  • Each training pair is expanded into duplicated full pairs plus separate Arg1-only and Arg2-only samples. EntRel relations count as implicit, and AltLex relations are excluded.
  • Training uses fixed 256-token sequences, 300-dimensional LSTMs, a dropout rate of 0.5, Adam with a batch size of 80, and is implemented in Keras.
  • The model was applied to the English PDTB with the Chinese hyperparameters left untuned. It scored 27.09% accuracy, above the 22.01% majority baseline. The code is publicly released by the ACoLi lab at Goethe University Frankfurt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2040

@inproceedings{ronnqvist2017recurrent,
  title = {{A Recurrent Neural Model with Attention for the Recognition of Chinese Implicit Discourse Relations}},
  author = {Rönnqvist, Samuel and Schenk, Niko and Chiarcos, Christian},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2040},
  url = {https://doi.org/10.18653/v1/p17-2040}
}

References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tsutomu Hirao, Yasuhisa Yoshida, Masaaki Nishino, Norihito Yasuda, and Masaaki Nagata. 2013. Single-Document Summarization as a Tree Knapsack Problem. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seatt… find
  2. Fatemeh Torabi Asr and Vera Demberg. 2015. Uniform Information Density at the Level of Discourse Relations: Negation Markers and Discourse Connective Omission. In 11th International Conference on Computational Semantics (IWCS). page 118. http://www.coli.uni-saarland.de/ fatemeh/iwcs2015.pdf. ↗
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR abs/1409.0473. http://arxiv.org/abs/1409.0473. ↗ our card
  4. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9(8):1735– 1780. https://doi.org/10.1162/neco.1997.9.8.1735. ↗
  5. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit Discourse Relation Detection via a Deep Architecture with Gated Relevance Network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, German… find our card
  6. Hen-Hsen Huang and Hsin-Hsi Chen. 2011. Chinese Discourse Relation Recognition. In Proceedings of 5th International Joint Conference on Natural Language Processing. Asian Federation of Natural Language Processing, Chiang Mai, Thailand, pages 1442–1446. http://www.aclweb.org/anthology/I11-1170. ↗ our card
  7. Minwei Feng, Bing Xiang, Michael R. Glass, Lidan Wang, and Bowen Zhou. 2015. Applying deep learning to answer selection: A study and an open task. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2015, Scottsdale, AZ, USA, December 13-17, 2015. pages 813–820. https://doi… — [doi:10.1109/asru.2015.7404872](https://doi.org/10.1109/asru.2015.7404872) find
  8. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association… find
  9. Vanessa Wei Feng and Graeme Hirst. 2012. Text-level Discourse Parsing with Rich Linguistic Features. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1. Association for Computational Linguistics, Stroudsburg, PA, USA, ACL ’12, pages 60–… find our card
  10. Ping Jian, Xiaohan She, Chenwei Zhang, Pengcheng Zhang, and Jian Feng. 2016. Discourse Relation Sense Classification Systems for CoNLL-2016 Shared Task. In Proceedings of the CoNLL-16 shared task. Association for Computational Linguistics, pages 158–163. https://doi.org/10.18653/v1/K16-2022. ↗
  11. David Graff and Ke Chen. 2005. Chinese Gigaword. LDC Catalog No.: LDC2003T09, ISBN, 1:58563– 58230. find
  12. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR abs/1412.6980. http://arxiv.org/abs/1412.6980. ↗
  13. Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks 18(5-6):602–610. https://doi.org/10.1016/j.neunet.2005.06.042. — [doi:10.1016/j.neunet.2005.06.042](https://doi.org/10.1016/j.neunet.2005.06.042) ↗
  14. Alex Lascarides and Nicholas Asher. 1993. Temporal Interpretation, Discourse Relations and Commonsense entailment. Linguistics and Philosophy 16(5):437–493. find
  15. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521(7553):436–444. find
  16. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. pages 1693– 1701. find our card
  17. Andrew Hickl. 2008. Using Discourse Commitments to Recognize Textual Entailment. In Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1. Association for Computational Linguistics, Stroudsburg, PA, USA, COLING ’08, pages 337–344. http://dl.acm.org/citation.cfm?id=… ↗
  18. Yang Liu and Sujian Li. 2016. Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. pages 1224–1233. htt… find our card
  19. William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text 8(3):243–281. find our card
  20. Mingyu Sun and Joyce Y Chai. 2007. Discourse processing for context question answering based on linguistic knowledge. Knowledge-Based Systems 20(6):511–526. find
  21. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… find
  22. Rakshit S. Trivedi and Jacob Eisenstein. 2013. Discourse connectors for latent subjectivity in sentiment analysis. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9-14, 2013, Westin Peachtree Plaza Hotel, Atl… find
  23. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic Sense Prediction for Implicit Discourse Relations in Text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 -Volume… find our card
  24. Jianxiang Wang and Man Lan. 2016. Two End-to-end Shallow Discourse Parsers for English and Chinese in CoNLL-2016 Shared Task. In Proceedings of the CoNLL-16 shared task. Association for Computational Linguistics, pages 33–40. https://doi.org/10.18653/v1/K16-2004. ↗ our card
  25. Bonnie L. Webber. 2004. D-LTAG: extending lexicalized TAG to discourse. Cognitive Science 28(5):751–779. http://dblp.uni-trier.de/db/journals/cogsci/cogsci28.html. ↗
  26. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings, 6th International Conference on Language Resources and Evaluation. Marrakech, Morocco, pages 2961–2968. find our card
  27. Gregor Weiss and Marko Bajec. 2016. Discourse Sense Classification from Scratch using Focused RNNs. In Proceedings of the CoNLL-16 shared task. Association for Computational Linguistics, pages 50–54. https://doi.org/10.18653/v1/K16-2006. ↗
  28. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016. Shallow Discourse Parsing Using Convolutional Neural Network. In Proceedings of the CoNLL-16 shared task. Association for Computational Linguistics, pages 70–77. https://doi.org/10.18653/v1/K16-2010. ↗ our card
  29. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Bonnie Webber, Attapol Rutherford, Chuan Wang, and Hongmin Wang. 2016. The CoNLL-2016 Shared Task on Shallow Discourse Parsing. In Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task. Association for Computational… find
  30. Hannah Rohde and William Horton. 2010. Why or what next? Eye movements reveal expectations about discourse direction. Talk at the 23rd Annual CUNY Conference on Human Sentence Processing. New York, NY. find
  31. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow Convolutional Neural Network for Implicit Discourse Relation Recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21,… find our card
  32. Attapol Rutherford and Nianwen Xue. 2016. Robust Non-Explicit Neural Discourse Parser in English and Chinese. In Proceedings of the CoNLL-16 shared task. Association for Computational Linguistics, pages 55–59. https://doi.org/10.18653/v1/K16-2007. ↗ our card
  33. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berl… find our card
  34. Attapol T. Rutherford, Vera Demberg, and Nianwen Xue. 2016. Neural Network Models for Implicit Discourse Relation Classification in English and Chinese without Surface Features. CoRR abs/1606.01990. http://arxiv.org/abs/1606.01990. — [arXiv:1606.01990](https://arxiv.org/abs/1606.01990) ↗
  35. Yuping Zhou and Nianwen Xue. 2012. PDTB-style Discourse Annotation of Chinese Text. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Jeju Island, Korea, pages 69–77. http://www.aclweb.org/an… ↗
  36. Niko Schenk, Christian Chiarcos, Kathrin Donandt, Samuel Rönnqvist, Evgeny Stepanov, and Giuseppe Riccardi. 2016. Do We Really Need All Those Rich Linguistic Features? A Neural Network-Based Approach to Implicit Sense Labeling. In Proceedings of the CoNLL-16 shared task. Association for Computation… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ronnqvist-2017-recurrent-neural-model-attention-recognition/