Elamur Research⌕
☰
home / papers / 2017 / Neural Discourse Structure for Text Categorization

Neural Discourse Structure for Text Categorization

Summary

The paper asks whether the rhetorical organization of a document can guide a text classifier toward its most informative parts. The authors parse each document with DPLP, an open-source RST discourse parser, and convert the resulting tree into a discourse dependency structure in which central EDUs sit near the root. A recursive neural network is then built over this tree. A bidirectional LSTM encodes each EDU, and the children are merged into their parent using relation-specific matrices and a new attention weight. This weight is a sigmoid score rather than a softmax, so siblings do not compete for salience. The model is compared with simpler variants (UNLABELED, ROOT, ADDITIVE) across five categorization tasks. Variants of the approach beat prior work on four of the five tasks. The exception is legislative bills, a genre whose discourse conventions differ from the parser's training data. Further experiments indicate that better discourse parses lead to better classification. The work shows that automatically derived discourse structure can serve as a useful inductive bias for document-level classification.

Key points

  • Pipeline:
    • RST trees come from DPLP (Ji and Eisenstein, 2014).
    • They are deterministically converted to dependency trees using nucleus–satellite information (following Yoshida et al., 2014).
    • EDUs are encoded with a BiLSTM, and the document label is predicted with a softmax over the root vector.
  • Four model variants:
    • FULL uses relation-specific composition matrices plus attention.
    • UNLABELED uses tree structure and attention only, with far fewer parameters.
    • ROOT represents the document by its root EDU alone.
    • ADDITIVE averages all EDUs and ignores structure; it resembles HN-AVE of Yang et al. (2016).
  • Attention ablation: On Yelp, replacing the unnormalized attention with softmax-normalized attention gives 70.3% accuracy. This is 1.5 points absolute below the FULL model.
  • Results across tasks: The approach outperforms earlier systems on four of five tasks but not on congressional bill prediction. The authors attribute this to domain mismatch with the parser's training data.
  • Implementation:
    • The model is built in DyNet with norm clipping (τ = 5.0) and dropout of 0.3.
    • A grid search covers embedding and hidden sizes, learning rate, and SGD vs. Adam.
    • Pretrained GloVe embeddings are used for smaller datasets.
    • The vocabulary is cut until about 5% of tokens become UNK.
    • Code is released at github.com/jiyfeng/disco4textcat.

Interesting ideas

  • I1. Sigmoid instead of softmax attention: Each child gets an independent sigmoid gate, so several parallel children (for example, two explanations of the same claim) can all count as important. This could help wherever multiple pieces of evidence add up rather than compete, such as aggregating reviews or multiple supporting sentences.
  • I2. Constituency to dependency conversion: Turning a nucleus/satellite tree into a dependency tree places the most central unit at the root. This conversion trick transfers to any hierarchical annotation where "heads" can be defined.
  • I3. Soft linguistic bias: Building the network on the parse favours central EDUs while still letting learned weights override that preference. This middle ground between hand-crafted weighting and fully unstructured attention could be reused with other noisy parsers, including those for lower-resource languages.
  • I4. Ladder of ablations: The ROOT → ADDITIVE → UNLABELED → FULL progression separates the value of structure, relation labels and composition while holding the rest of the implementation constant. This is a clean evaluation template for testing whether any external linguistic resource actually helps.
  • I5. Diagnosing pipeline dependencies: Measuring how downstream accuracy changes with parse quality and genre shows when a pipeline component becomes the bottleneck. This analysis is worth repeating before relying on a parser trained on news for other domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Discourse analysis

 Institutions

🎓University of WashingtonSeattle · United States206 works · 158,423 citations

Cite

Cite as Yangfeng Ji and Noah A. Smith. 2017. Neural Discourse Structure for Text Categorization. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1092

@inproceedings{ji2017neural,
  title = {{Neural Discourse Structure for Text Categorization}},
  author = {Yangfeng Ji and Noah A. Smith},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1092},
  url = {https://doi.org/10.18653/v1/p17-1092}
}

References

28 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  2. Moshe Azar. 1999. Argumentative text as rhetorical structure: An application of rhetorical structure theory. Argumentation 13(1):97–114. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. our card
  4. David Bamman, Brendan O’Connor, and Noah A Smith. 2014. Learning latent personas of film characters. In ACL. our card
  5. James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D. Cox. 2015. Hyperopt: a Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. ArXiv:1404.2188. our card
  6. Youngjoong Ko, Jinwoo Park, and Jungyun Seo. 2004. Improving text categorization using the importance of sentences. Information Processing & Management 40(1):65–79. find
  7. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In EACL. find
  8. William Mann and Sandra Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text 8(3):243–281. our card
  9. Daniel Marcu. 1999. Discourse trees are good indicators of importance in text. In Inderjeet Mani and Mark T. Maybury, editors, Advances in Automatic Text Summarization, pages 123–136. find
  10. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 19(2):313–330. our card
  11. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH. find
  12. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. ArXiv:1701.03980. our card
  13. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics. Association for Computational Linguistics, page 271. our card
  14. Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. In ICML. find
  15. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In NAACL. our card
  16. Tae Yano, Noah A. Smith, and John D. Wilkerson. 2012. Textual predictors of bill survival in congressional committees. In NAACL. find
  17. Ainur Yessenalina, Yisong Yue, and Claire Cardie. 2010. Multi-level structured models for document sentiment classification. In EMNLP. find
  18. Dani Yogatama and Noah A. Smith. 2014. Linguistic structured sparsity in text categorization. In ACL. find
  19. Yasuhisa Yoshida, Jun Suzuki, Tsutomu Hirao, and Masaaki Nagata. 2014. Dependency-based discourse parser for single-document summarization. In EMNLP. our card
  20. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS. our card
  21. Romain Paulus, Richard Socher, and Christopher D Manning. 2014. Global belief recursive neural networks. In NIPS. find
  22. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In EMNLP. our card
  23. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse Treebank 2.0. In LREC. our card
  24. Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D. Manning. 2009. Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora. In EMNLP. find
  25. Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics 2:207–218. our card
  26. Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the vote: Determining support or opposition from Congressional floor-debate transcripts. In EMNLP. our card
  27. Yijun Xiao and Kyunghyun Cho. 2016. Efficient character-level document classification by combining convolution and recurrent layers. ArXiv:1602.00367. — [arXiv:1602.00367](https://arxiv.org/abs/1602.00367) find
  28. H. Yamada and Y. Matsumoto. 2003. Statistical dependency analysis with support vector machines. In IWPT. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ji-2017-neural-discourse-structure-text-categorization/