Elamur Research⌕
☰
home / papers / 2020 / A Neural Approach to Discourse Relation Signal Detection

A Neural Approach to Discourse Relation Signal Detection

Paper · 2020 · Amir Zeldes and Yang Liu

Summary

Discourse relations such as CONDITION, CONCESSION or CAUSE are often marked by connectives. Corpus-wide frequency or PMI counts give each word type a single score. They cannot tell whether a given occurrence of "and" or "to" actually carries a relation, and they cannot flag words that point the wrong way. Zeldes and Liu address this by training a distantly supervised neural relation classifier, an RNN over contextualized character and word embeddings, on the English GUM corpus, which is annotated with 20 RST relations. The model sees only relation labels and never the signal annotations. To score a token, they mask it and measure how the softmax probability of the gold relation changes. This metric, ∆s ("delta-softmax"), runs from -1 to 1, and negative values mark distractors. They compare the scores with human token-anchored signal annotations using recall@k. The model separates discourse-relevant uses of frequent, ambiguous words from irrelevant ones. It misses relational cues and signals that lie outside the related unit pair. The work provides graded signal strength per instance and a diagnostic of what discourse classifiers still lack.

Key points

  • Data: GUM, a multi-genre English web corpus with 20 RST relations. A subset carries human signal annotations anchored to specific tokens, and the authors propose a taxonomy of anchorable signal types.
  • ∆s works like permutation importance at the token level. It is the shift in gold-relation probability when a word is masked, bounded between -1 and 1, and it yields both signals and "distractors."
  • Evaluation is framed as information retrieval: recall@k against human anchors. It is complemented by a tokenwise signal-strength visualization and an error analysis of disagreements in both directions.
  • ∆s distributions separate uses of the same word:
    • "to" is bimodal, with prepositional uses scoring low and infinitival PURPOSE uses scoring high.
    • Capitalized "If" almost always marks CONDITION, while lowercase "if" sometimes heads unsegmented object clauses such as "see if."
  • Limitations found:
    • The model ignores antecedents of pronouns, lexical relations (synonymy, meronymy, antonymy) and count comparisons.
    • It captures only endocentric signals.
    • The many exocentric signals suggest a natural accuracy ceiling for classifying relations from head EDU pairs alone.

Interesting ideas

  • I1. Mask each input token and read off the change in the gold label's probability, giving a signed, per-instance importance score instead of a per-type average.
  • I2. Treat negative contributions as first-class "distractors," words whose presence makes the correct label harder to identify.
  • I3. Learn which words signal a relation using only relation labels (distant supervision), then validate against a small human-anchored subset with recall@k.
  • I4. Plot the per-type distribution of instance scores (boxplots) to expose ambiguity, bimodality and surface cues such as capitalization of "If" vs "if".
  • I5. Use the gap between model and human signals as a list of missing information: antecedents, lexical relations and quantity comparisons.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Apply the same masking probe to gparse attachments and labels, so that every parse decision can be explained by the words that supported or undermined it.
  • I2 (answer): Score words in maths word problems and fable questions for distractor strength, to locate misleading phrasing and build harder negative-control test items.
  • I3 (frames): Induce cue inventories for discourse relations and speech acts from relation-labelled text, and check them against a few hand-anchored examples before trusting them.
  • I4 (tokens): Keep case and sentence position as tagger features for connective-like words, and use score distributions to decide which tokens need contextual disambiguation rules.
  • I5 (world): Feed explicit coreference links, WordNet relations and compared measures from the text world into relation and event-order decisions, since a pure neural classifier does not recover them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgetown UniversityWashington · United States69 works · 4,417 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPhilosophy · 211 works

🧭 Topics 1Research subjects and languages the work is about.Discourse analysis · 394 works

Cite

Cite as Amir Zeldes and Yang Liu. 2020. A Neural Approach to Discourse Relation Signal Detection. Dialogue & Discourse.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.5087/dad.2020.201 · arXiv 2001.02380

@inproceedings{zeldes2020neural,
  title = {{A Neural Approach to Discourse Relation Signal Detection}},
  author = {Amir Zeldes and Yang Liu},
  booktitle = {Dialogue \& Discourse},
  year = {2020},
  doi = {10.5087/dad.2020.201},
  eprint = {2001.02380},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.5087/dad.2020.201}
}

References

56 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Akbik, Tanja Bergmann, and Roland Vollgraf. Pooled contextualized embeddings for named entity recognition. In Proceedings of NAACL 2019, pages 724–728, Minneapolis, MN, 2019.
  2. Farah Benamara, Maite Taboada, and Yannick Mathieu. Evaluative language beyond bags of words: Linguistic insights and computational applications. Computational Linguistics, 43(1):201–264, 2017. ↗
  3. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Current and New Directions in Discourse and Dialogue, Text, Speech and Language Technology 22, pages 85–112. Kluwer, Dordrecht, 2003. ↗
  4. Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl. Visualizing the feature importance for black box models. In Michele Berlingerio, Francesco Bonchi, Thomas Gärtner, Neil Hurley, and Georgiana Ifrim, editors, Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2018, Lecture Notes in Computer Science 11051. find
  5. Iria da Cunha. A symbolic corpus-based approach to detect and solve the ambiguity of discourse markers. Research in Computing Science, 70:95–106, 2013. ↗
  6. Debopam Das and Maite Taboada. Signalling of coherence relations in discourse, beyond discourse markers. Discourse Processes, pages 1–29, 2017. ↗
  7. Debopam Das and Maite Taboada. RST Signalling Corpus: A corpus of signals of coherence relations. Language Resources and Evaluation, 52(1):149–184, 2018. ↗
  8. Debopam Das, Maite Taboada, and Paul McFetridge. RST Signalling Corpus. LDC2015T10, 2019. find
  9. Eldadio Duque. Signaling causal coherence relations. Discourse Studies, 16(1):25–46, 2014. ↗
  10. Gill Francis. Labelling discourse: An aspect of nominal-group lexical cohesion. In Malcolm Coulthard, editor, Advances in Written Text Analysis, pages 83–101. Routledge, 1994. find
  11. Bruce Fraser. What are discourse markers? Journal of Pragmatics, 31:931–952, 1999. ↗
  12. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. AllenNLP: A deep semantic natural language processing platform. In Proceedings of ACL 2018, pages 1–6, Melbourne, 2018. ↗
  13. Luke Gessler, Yang Liu, and Amir Zeldes. A discourse signal annotation system for RST trees. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking (DISRPT 2019), pages 56–61, Minneapolis, MN, 2019.
  14. Reza Ghaeini, Xiaoli Z. Fern, and Prasad Tadepalli. Interpreting recurrent and attention-based neural models: A case study on natural language inference. In Proceedings of EMNLP 2018, pages 4952–4957, Brussels, 2018. find
  15. Eduard H. Hovy and Elisabeth Maier. Parsimonious or profligate: How many and which discourse structure relations? Technical report, Information Sciences Institute, USC, 1993. ↗
  16. Patrick Huber and Giuseppe Carenini. Predicting discourse structure using distant supervision from sentiment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2306–2316, Hong Kong, China, November 2019. ↗
  17. Alistair Knott. A Data-Driven Methodology for Motivating a Set of Coherence Relations. PhD thesis, University of Edinburgh, 1996. find
  18. Alistair Knott and Robert Dale. Using linguistic phenomena to motivate a set of coherence relations. Discourse Processes, 18(1):35–52, 1994. ↗
  19. Alistair Knott and Ted Sanders. The classification of coherence relations and their linguistic markers: An exploration of two languages. Journal of Pragmatics, 30(2):135–175, 1998. ↗
  20. Geoffrey Leech, Tony McEnery, and Martin Weisser. SPAAC speech-act annotation scheme. Technical report, Lancaster University, 2003. find
  21. Yang Liu. Beyond the Wall Street Journal: Anchoring and comparing discourse signals across genres. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking (DISRPT 2019), pages 72–81, Minneapolis, MN, 2019.
  22. Yang Liu and Amir Zeldes. Discourse relations and signaling information: Anchoring discourse signals in RST-DT. Proceedings of the Society for Computation in Linguistics, 2(1):314–317, 2019. ↗
  23. William C. Mann and Sandra A. Thompson. Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8(3):243–281, 1988. ↗
  24. Daniel Marcu. Building up rhetorical structure trees. In Proceedings of AAAI-96, pages 1069–1074, Portland, OR, 1996. find
  25. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Special Issue on Using Large Corpora, Computational Linguistics, 19(2):313–330, 1993. ↗
  26. Jiřı́ Mı́rovský, Pavlı́na Synková, Magdaléna Rysová, and Lucie Poláková. CzeDLex a lexicon of Czech discourse connectives. The Prague Bulletin of Mathematical Linguistics, 109:61–91, 2017. find
  27. Mathieu Morey, Philippe Muller, and Nicholas Asher. How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT. In Proceedings of EMNLP 2017, pages 1319–1324, Copenhagen, Denmark, 2017.
  28. Philippe Muller, Chloé Braud, and Mathieu Morey. ToNy: Contextual embeddings for accurate multilingual discourse segmentation of full documents. In Proceedings of Discourse Relation Treebanking and Parsing (DISRPT 2019), pages 115–124, Minneapolis, MN, 2019. find
  29. Karthik Narasimhan and Regina Barzilay. Machine comprehension with discourse relations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL/IJCNLP 2015), pages 1253–1262, Beijing, China, 2015.
  30. Joakim Nivre, Željko Agić, Lars Ahrenberg, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Riyaz Ahmad Bhat, Eckhard Bick, Cristina Bosco, Gosse Bouma, Sam Bowman, Marie Candito, Gülşen Cebiroğlu Eryiğit, Giuseppe G. A. Celano, Fabricio Chalub, Jinho Choi, Çağrı Çöltekin, Miriam Connor, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Arantza Diaz de Ilarraza, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Marhaba Eli, Tomaž Erjavec, Richárd Farkas, Jennifer Foster, Cláudia Freitas, Katarı́na Gajdošová, Daniel Galbraith, Marcos Garcia, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Memduh Gökırmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta Gonzáles Saavedra, Matias Grioni, Normunds Grūzı̄tis, Bruno Guillaume, Nizar Habash, Jan Hajič, Linh Hà Mỹ, Dag Haug, Barbora Hladká, Petter Hohle, Radu Ion, Elena Irimia, Anders Johannsen, Fredrik Jørgensen, Hüner Kaşıkara, Hiroshi Kanayama, Jenna Kanerva, Natalia Kotsyba, Simon Krek, Veronika Laippala, Lê H`ông, Alessandro Lenci, Nikola Ljubešić, Olga Lyashevskaya, Teresa Lynn, Aibek Makazhanov, Christopher Manning, Cătălina Mărănduc, David Mareček, Héctor Martı́nez Alonso, André Martins, Jan Mašek, Yuji Matsumoto, Ryan McDonald, Anna Missilä, Verginica Mititelu, Yusuke Miyao, Simonetta Montemagni, Amir More, Shunsuke Mori, Bohdan Moskalevskyi, Kadri Muischnek, Nina Mustafina, Kaili Müürisep, Luong Nguy˜ên Thi., Huy`ên Nguy˜ên Thi. Minh, Vitaly Nikolaev, Hanna Nurmi, Stina Ojala, Petya Osenova, Lilja Øvrelid, Elena Pascual, Marco Passarotti, Cenel-Augusto Perez, Guy Perrier, Slav Petrov, Jussi Piitulainen, Barbara Plank, Martin Popel, Lauma Pretkalniņa, Prokopis Prokopidis, Tiina Puolakainen, Sampo Pyysalo, Alexandre Rademaker, Loganathan Ramasamy, Livy Real, Laura Rituma, Rudolf Rosa, Shadi Saleh, Manuela Sanguinetti, Baiba Saulı̄te, Sebastian Schuster, Djamé Seddah, Wolfgang Seeker, Mojgan Seraji, Lena Shakurova, Mo Shen, Dmitry Sichinava, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simkó, Mária Šimková, Kiril Simov, Aaron Smith, Alane Suhr, Umut Sulubacak, Zsolt Szántó, Dima Taji, Takaaki Tanaka, Reut Tsarfaty, Francis Tyers, Sumire Uematsu, Larraitz Uria, Gertjan van Noord, Viktor Varga, Veronika Vincze, Jonathan North Washington, Zdeněk v Zabokrtský, Amir Zeldes, Daniel Zeman, and Hanzhi Zhu. Universal dependencies 2.0. Technical report, LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University, 2017. URL http://hdl.handle.net/11234/1-1983. ↗
  31. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In Proceedings of EMNLP 2014, pages 1532–1543, Doha, Qatar, 2014.
  32. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL 2018, pages 2227–2237, New Orleans, LA, 2018.
  33. Emily Pitler, Annie Louis, and Ani Nenkova. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 683–691, Suntec, Singapore, 2009.
  34. Massimo Poesio, Barbara Di Eugenio, and Gerard Keohane. Discourse structure and anaphora: An empirical study. NLE Technical Note TN-02-02, University of Essex, Department of Computer Science, NLE Group, 2002. find
  35. Richard Power, Christine Doran, and Donia Scott. Generating embedded discourse markers from rhetorical structure. In Proceedings of EWNLG 1999, pages 30–38, Toulouse, 1999. find
  36. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. The Penn Discourse Treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008), pages 2961–2968, Marrakesh, Morocco, 2008.
  37. Rashmi Prasad, Bonnie Webber, and Aravind Joshi. Reflections on the Penn Discourse TreeBank, comparable corpora, and complementary annotation. Computational Linguistics, 40(4):921–950, 2014. ↗
  38. Rashmi Prasad, Bonnie Webber, Alan Lee, and Aravind Joshi. Penn Discourse Treebank Version 3.0. LDC2019T05, 2019. ↗
  39. Craige Roberts. Information structure in discourse: Towards an integrated formal theory of pragmatics. Semantics and Pragmatics, 5(6):1–69, December 2012. ↗
  40. Tatjana Scheffler and Manfred Stede. Adding semantic relations to a large-coverage connective lexicon of German. In Proceedings of LREC 2016, pages 1008–1013, Reykjavik, Iceland, 2016. find
  41. Manfred Stede. Discourse Processing. Synthesis Lectures on Human Language Technologies 4. Morgan & Claypool, [San Rafael, CA], 2012. find
  42. Manfred Stede and Carla Umbach. DiMLex: A lexicon of discorse markers for text generation and understanding. In Proceedings of COLING-ACL 1998, pages 1238–1242, Montreal, 1998. find
  43. Maite Taboada. Discourse markers as signals (or not) of rhetorical relations. Journal of pragmatics, 38(4):567–592, 2006. ↗
  44. Maite Taboada and Debopam Das. Annotation upon annotation: Adding signalling information to a corpus of discourse relations. Dialogue and Discourse, 4(2):249–281, 2013. ↗
  45. Maite Taboada and Marı́a de los Ángeles Gómez-González. Discourse markers and coherence relations: Comparison across markers, languages and modalities. Linguistics and the Human Sciences, 6(1–3):17–41, 2012. ↗
  46. Svetlana Toldova, Dina Pisarevskaya, Margarita Ananyeva, Maria Kobozeva, Alexander Nasedkin, Sofia Nikiforova, Irina Pavlova, and Alexey Shelepov. Rhetorical relation markers in Russian RST Treebank. In Proceedings of the 6th Workshop Recent Advances in RST and Related Formalisms, pages 29–33, Santiago de Compostela, Spain, 2017. find
  47. Fatemeh Torabi Asr and Vera Demberg. On the information conveyed by discourse markers. In Proceedings of the Fourth Annual Workshop on Cognitive Modeling and Computational Linguistics (CMCL), pages 84–93, Sofia, Bulgaria, 2013.
  48. Yannick Versley and Anna Gastel. Linguistic tests for discourse relations in the TüBa-D/Z corpus of written German. Dialogue and Discourse, 4(2):142–173, 2013.
  49. Bonnie Lynn Webber and Aravind K Joshi. Anchoring a lexicalized Tree-Adjoining Grammar for discourse. In Proceedings of SIGDIAL 1998, pages 86–92, Montreal, 1998. ↗
  50. Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of EMNLP 2019, pages 11–20, Hong Kong, China, 2019.
  51. Yue Yu, Yilun Zhu, Yang Liu, Yan Liu, Siyao Peng, Mackenzie Gong, and Amir Zeldes. GumDrop at the DISRPT2019 shared task: A model stacking approach to discourse unit segmentation and connective detection. In Proceedings of Discourse Relation Treebanking and Parsing (DISRPT 2019), pages 133–143, Minneapolis, MN, 2019.
  52. Amir Zeldes. The GUM Corpus: Creating Multilayer Resources in the Classroom. Language Resources and Evaluation, 51(3):581–612, 2017. ↗
  53. Amir Zeldes. Multilayer Corpus Studies. Routledge Advances in Corpus Linguistics 22. Routledge, London, 2018a. ↗
  54. Amir Zeldes. A neural approach to discourse relation signaling. In Georgetown University Round Table (GURT) 2018: Approaches to Discourse, Washington, DC, 2018b. find
  55. Deniz Zeyrek, Işin Demirşahin, Ayışığı B. Sevdik-Çallı, and Ruket Çakıcı. Turkish Discourse Bank: Porting a discourse annotation style to a morphologically rich language. Dialogue and Discourse, 4(2):174–184, 2013. ↗
  56. Yuping Zhou, Jill Lu, Jennifer Zhang, and Nianwen Xue. LDC2014T21, 2014. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeldes-2020-neural-approach-discourse-relation-signal/