Elamur Research⌕
☰
home / papers / 2016 / Implicit Discourse Relation Recognition with Context-aware Character-enhanced Embeddings

Implicit Discourse Relation Recognition with Context-aware Character-enhanced Embeddings

Summary

Implicit discourse relations are links between two text spans (Arg1, Arg2) that lack an overt connective such as "so" or "but". They are the hardest part of discourse parsing. Annotated data is scarce, so models built on word-level features or embeddings struggle, particularly with rare words. Qin, Zhang and Zhao propose a hybrid neural classifier that adds character-derived vectors to ordinary word embeddings. In the character module, a CNN with max-pooling reads each word's characters. The resulting word sequence then goes through bidirectional LSTMs, so every vector reflects its sentence context. These vectors are joined with standard word embeddings to form "enhanced embeddings". A second CNN turns them into argument representations, followed by hidden layers and a softmax classifier. Experiments on the Penn Discourse Treebank (PDTB) cover multi-class and binary sense classification, with ablations comparing word-only, character-only and combined variants. Fusing the two sources at the embedding level outperformed every variant and matched or exceeded prior state of the art. The study shows that context-sensitive subword information benefits a semantic task where data sparsity is a central obstacle.

Key points

  • Data and evaluation: Experiments use the PDTB. The binary setup trains on Sections 2–20, develops on 0–1 and tests on 21–22. Training negatives are randomly sampled to balance classes, and scores are reported as Macro-F1.
  • Binary F1 of Char+Word-Enhanced:
    • Comparison: 38.67
    • Contingency: 54.91
    • Expansion+EntRel: 80.66
    • Temporal: 32.76
    • Average: 51.75, slightly above Chen et al. (2016) at 51.72 and Rutherford & Xue (2014) at 50.81. Only on Comparison does it trail Chen et al. (40.17).
  • Ablation findings:
    • Character-only embeddings did worse than simply summing word vectors.
    • Stacking BiLSTMs on the character CNN improved results.
    • Word-level BiLSTMs using only the last state lagged behind CNN.
    • Concatenating at the sentence level was weaker than merging at the embedding level.
  • Character module details: The alphabet covers upper- and lowercase letters, digits and punctuation. The module applies several filter widths with tanh activation, followed by one-max pooling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{qin2016implicit,
  title = {{Implicit Discourse Relation Recognition with Context-aware Character-enhanced Embeddings}},
  author = {Qin, Lianhui and Zhang, Zhisong and Zhao, Hai},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1180/}
}

References

43 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from RST discourse parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2212–2218, Lisbon, Portugal, November. find our card
  2. Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 69–73, Sofia, Bulgaria, August. find our card
  3. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2201–2211, Lisbon, Portugal, September. find our card
  4. Deng Cai and Hai Zhao. 2016. Neural Word Segmentation Learning for Chinese. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 409–420, Berlin, Germany, August. find
  5. Joyce Y Chai and Rong Jin. 2004. Discourse structure for context question answering. In Proceedings of the Workshop on Pragmatics of Question Answering at North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL), volume 2004, pages 23–30, San D… find
  6. Changge Chen, Peilu Wang, and Hai Zhao. 2015. Shallow discourse parsing using constituent parsing tree. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task (CONLL), pages 37–41, Beijing, China, July. find our card
  7. Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54rd Annual Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany, August. find our card
  8. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  9. Robert Fisher and Reid Simmons. 2015. Spectral semi-supervised discourse relation classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 89–93, Beijing, C… find our card
  10. Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. 2005. Bidirectional lstm networks for improved phoneme classification and recognition. In International Conference on Artificial Neural Networks (ICANN), pages 799–804, Warsaw, Poland. find
  11. Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013. Hybrid speech recognition with deep bidirectional lstm. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 273–278, Olomouc, Czech Republic. find
  12. Francisco Guzmán, Shafiq Joty, Lluı́s Màrquez, and Preslav Nakov. 2014. Using discourse structure improves machine translation evaluation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 687–698, Baltimore, Maryland, June. find
  13. Christian Hardmeier. 2012. Discourse in statistical machine translation: A survey and a case study. Discours-Revue de linguistique, psycholinguistique et informatique, 11. find
  14. Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric P Xing. 2016a. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2410–2420, Berlin, Germany, August. find
  15. Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric P Xing. 2016b. Deep neural networks with massive learned knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), Austin, USA, November. find
  16. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics (TACL), 3:329–344. find our card
  17. Yangfeng Ji, Gongbo Zhang, and Jacob Eisenstein. 2015. Closing the gap: Domain adaptation from explicit to implicit discourse relations. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2219–2224, Lisbon, Portugal, September. find our card
  18. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse-driven language models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pag… find
  19. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence, pages 2741–2749, Phoenix, USA. find our card
  20. Junyi Jessy Li and Ani Nenkova. 2014. Reducing sparsity improves the recognition of implicit discourse relations. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 199–207, Philadelphia, USA. find our card
  21. Zhongyi Li, Hai Zhao, Chenxi Pang, Lili Wang, and Huan Wang. 2016. A constituent syntactic parse tree based discourse parser. In Proceedings of the Twentieth Conference on Computational Natural Language Learning -Shared Task (CONLL), pages 60–64, Berlin, Germany, August. find our card
  22. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 343–351, Suntec,Singapore. find our card
  23. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 20(02):151–184. find
  24. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proc… find our card
  25. Sameer Maskey and Julia Hirschberg. 2005. Comparing lexical, acoustic/prosodic, structural and discourse features for speech summarization. In the 9th biennial conference of the International Speech Communication Association (ISCA) and the 6th in the annual series of INTERSPEECH events (INTERSPEECH… find
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (3), pages 3111–3119, South Lake Tahoe, Nevada, USA, December. find
  27. Gabriel Murray, Steve Renals, Jean Carletta, and Johanna Moore. 2006. Incorporating speaker and discourse features into speech summarization. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (N… find
  28. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 108–112, Seoul, South Korea, July. find our card
  29. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of he Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Proc… find our card
  30. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The Penn Discourse TreeBank 2.0. In The 6th edition of the Language Resources and Evaluation Conference (LREC), pages 2961–2968, Marrakech, Morocco. find our card
  31. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016a. Shallow discourse parsing using convolutional neural network. In Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task (CONLL), pages 70–77, Berlin, Germany, August. find our card
  32. Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016b. A stacking gated neural architecture for implicit discourse relation classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), Austin, USA, November. find our card
  33. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 645–654, Gothenburg, Swe… find our card
  34. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies,… find our card
  35. Suzan Verberne, Lou Boves, Nelleke Oostdijk, and Peter-Arno Coppen. 2007. Evaluating discourse-based answer extraction for why-question answering. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 735–736, Amsterdam, Holl… find
  36. Peilu Wang, Yao Qian, Frank K Soong, Lei He, and Hai Zhao. 2015. Word embedding for recurrent neural network based TTS synthesis. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4879–4883, Brisbane, Australia. find
  37. Peilu Wang, Yao Qian, Frank K. Soong, Lei He, and Hai Zhao. 2016. Learning distributed word representations for bidirectional LSTM recurrent neural network. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… find
  38. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The CoNLL-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task (CONLL), pages 1–16, Beijing, China… find our card
  39. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Bonnie Webber, Attapol Rutherford, Chuan Wang, and Hongmin Wang. 2016. The CoNLL-2016 shared task on shallow discourse parsing. In Proceedings of the Twentieth Conference on Computational Natural Language Learning - Shared Task (CONLL), pages 1–19, Berlin, G… find
  40. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015a. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2230–2235, Lisbon, Portugal, Se… find our card
  41. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems (NIPS), pages 649–657, Montral, Quebec, Canada. find our card
  42. Zhisong Zhang, Hai Zhao, and Lianhui Qin. 2016. Probabilistic graph-based dependency parsing with convolutional neural network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1382–1392, Berlin, Germany, August. find
  43. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING), pages 1507–1514, Beijing, China, August. find our card

↑ back to top

This page in Markdown →