Elamur Research⌕
☰
home / papers / 2020 / Weakly-Supervised Modeling of Contextualized Event Embedding for Discourse Relations

Weakly-Supervised Modeling of Contextualized Event Embedding for Discourse Relations

Summary

The paper addresses how to represent events in long narratives so that models can reason about the different ways events relate, not just whether they co-occur. Prior script-learning embeddings mostly measure event similarity, which cannot tell apart plausible continuations that require knowing causes or contrasts. The authors convert each document into a Narrative Graph. Nodes are predicate-centric events. Edges encode sequential order (Next), shared coreferent entities (CNext), and six PDTB-derived discourse relations, which a simple rule-based annotator detects through explicit connectives. Each node's text is first encoded locally with BERT. A Relational Graph Convolution Network (R-GCN) then contextualizes the node over the graph. Training is weakly supervised on the NYT section of English Gigaword: the model learns to recover deleted edges and to reject wrong node–edge pairings. On intrinsic tasks and implicit discourse sense classification, the model beats strong baselines. This shows that graph-level event context adds information that word-level contextual embeddings miss.

Key points

  • Relation set. The graph uses eight relation types: Next, CNext, Before, After, Sync., Contrast, Reason and Result. They were extracted automatically from about 1.42M Gigaword NYT documents, yielding roughly 274M Next edges and 66M CNext edges.
  • Discourse sense prediction. On predicting the sense between event pairs, the R-GCN layer adds about 21 points of absolute accuracy over the Event-BERT-ft baseline.
  • Triplet classification. The macro-averaged minority-class F1 is 67.19. CNext is the hardest relation (F1 43.36), which the authors attribute to coreference noise and long distances between linked events.
  • CoNLL 2016 implicit discourse sense classification. NG embeddings concatenated with ELMo reach 42.84 F1 on the test set and 43.91 F1 on the blind set. This beats the shared-task leaders and EventTransE (39.05 test / 38.35 blind).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.446

@inproceedings{lee2020weakly,
  title = {{Weakly-Supervised Modeling of Contextualized Event Embedding for Discourse Relations}},
  author = {Lee, I-Ta and Pacheco, Maria Leonor and Goldwasser, Dan},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.446},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.446}
}

References

46 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Or Biran and Kathleen McKeown. 2013. Aggregated word pair features for implicit discourse relation disambiguation. find our card
  2. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787–2795. find
  3. Nathanael Chambers and Daniel Jurafsky. 2008. Unsupervised learning of narrative event chains. In ACL, volume 94305, pages 789–797. find
  4. Kai-Wei Chang, Wen-tau Yih, Bishan Yang, and Christopher Meek. 2014. Typed tensor decomposition of knowledge bases for relation extraction. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1568–1579. find
  5. Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 933–941. JMLR. org. find our card
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  7. Dan Goldwasser and Xiao Zhang. 2016. Understanding satirical articles using common-sense. Transactions of the Association for Computational Linguistics, 4:537–549. find
  8. Mark Granroth-Wilding and Stephen Clark. 2016. What happens next? event prediction using a compositional neural network model. In AAAI, pages 2727–2733. find
  9. Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855–864. find
  10. Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in neural information processing systems, pages 1024–1034. find our card
  11. Rujun Han, Qiang Ning, and Nanyun Peng. 2019. Joint event and temporal relation extraction with shared representations and structured prediction. arXiv preprint arXiv:1909.05360. find
  12. Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907. find our card
  13. I-Ta Lee and Dan Goldwasser. 2018. Feel: Featured event embedding learning. AAAI, pages 4840–4847. find
  14. I-Ta Lee and Dan Goldwasser. 2019. Multi-relational script learning for discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4214–4226. find our card
  15. Chang Li and Dan Goldwasser. 2019. Encoding social information with graph convolutional networks forpolitical perspective detection in news media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2594– 2604. find
  16. Zhongyang Li, Xiao Ding, and Ting Liu. 2018. Constructing narrative event evolutionary graph for script event prediction. arXiv preprint arXiv:1805.05081. find
  17. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  18. Eric Malmi, Daniele Pighin, Sebastian Krause, and Mikhail Kozhevnikov. 2017. Automatic prediction of discourse connectives. CoRR, abs/1702.00992. find our card
  19. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In ACL (System Demonstrations), pages 55–60. find our card
  20. Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. arXiv preprint arXiv:1703.04826. find our card
  21. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  22. Allen Nie, Erin Bennett, and Noah Goodman. 2019. Dissent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4497–4510. find our card
  23. Maria Leonor Pacheco, I-Ta Lee, Xiao Zhang, Abdullah Khan Zehady, Pranjal Daga, Di Jin, Ayush Paro-lia, and Dan Goldwasser. 2016. Adapting event embedding for implicit discourse relation recognition. Proceedings of the CoNLL-16 shared task, pages 136–142. find our card
  24. Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108–112. Association for Computational Linguistics. find our card
  25. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English gigaword fifth edition ldc2011t07. dvd. Philadelphia: Linguistic Data Consortium. find
  26. Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701–710. find
  27. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  28. Karl Pichotta and Raymond Mooney. 2014. Statistical script learning with multi-argument events. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 220–229. find
  29. Karl Pichotta and Raymond J Mooney. 2016a. Learning statistical scripts with lstm recurrent neural networks. In AAAI. find
  30. Karl Pichotta and Raymond J Mooney. 2016b. Using sentence-level lstm language models for script inference. arXiv preprint arXiv:1604.02993. find
  31. Rashmi Prasad, Eleni Miltsakaki, Nikhil Dinesh, Alan Lee, Aravind Joshi, Livio Robaldo, and Bonnie L Webber. 2007. The penn discourse treebank 2.0 annotation manual. find
  32. Attapol Rutherford and Nianwen Xue. 2016. Robust non-explicit neural discourse parser in english and chinese. Proceedings of the CoNLL-16 shared task, pages 55–59. find our card
  33. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. find
  34. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57:45. find
  35. Jianxiang Wang and Man Lan. 2016. Two end-to-end shallow discourse parsers for english and chinese in conll-2016 shared task. Proceedings of the CoNLL-16 shared task, pages 33–40. find our card
  36. Zhongqing Wang, Yue Zhang, and Ching-Yun Chang. 2017. Integrating order information and event relation for script event prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 57–67. find
  37. Noah Weber, Niranjan Balasubramanian, and Nathanael Chambers. 2017. Event representations with tensor-based compositions. arXiv preprint arXiv:1711.07611. — [arXiv:1711.07611](https://arxiv.org/abs/1711.07611) find
  38. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. find
  39. Conll 2016 shared task on multilingual shallow discourse parsing. CoNLL-16 shared task. find our card
  40. Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 1507–1514. Association for Computational Linguis… find our card
  41. Roger C Schank and Robert P Abelson. 1977. Scripts, plans, goals and understanding: An inquiry into human knowledge structures. find
  42. Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593–607. Springer. find
  43. Damien Sileo, Tim Van-De-Cruys, Camille Pradel, and Philippe Muller. 2019. Mining discourse markers for unsupervised sentence representation learning. arXiv preprint arXiv:1903.11850. find our card
  44. Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. 2015. Line: Large-scale information network embedding. In Proceedings of the 24th international conference on world wide web, pages 1067–1077. find
  45. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. find our card
  46. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lee-2020-weakly-supervised-modeling-contextualized-event/