Elamur Research⌕
☰
home / papers / 2017 / Neural Net Models of Open-domain Discourse Coherence

Neural Net Models of Open-domain Discourse Coherence

Summary

Discourse coherence, meaning how sentences connect into a sensible whole, matters for both understanding and generating text. Earlier coherence models usually captured one phenomenon, such as entity transitions or lexical overlap. They worked in narrow domains and could only score existing text. Li and Jurafsky propose domain-independent neural models in two families:

  • Discriminative model: LSTM sentence embeddings for a window of sentences feed a classifier. Original windows are positive examples, and windows whose center sentence was randomly replaced are negatives.
  • Generative models:
    • SEQ2SEQ scorers that predict a sentence from its neighbor, in forward, bidirectional, and language-model-corrected (mutual-information-like) variants.
    • HMM-LDA-GM, a pipeline that feeds Markov topic states into the decoder.
    • VLV-GM, an end-to-end model with Markov-structured variational latent variables that carry discourse state across sentences.

The discriminative model essentially solves the classic small-domain sentence-pair ordering benchmark. However, it degrades in open domains because negative sampling covers too little of the meaning space. The generative models do best in open-domain settings, including full-paragraph ordering, and VLV-GM yields the most human-like generated text. The authors urge the field to adopt harder open-domain evaluations.

Key points

  • The discriminative clique model, a scalable extension of Li and Hovy (2014), reaches about 96% accuracy on the Barzilay and Lapata (2008) sentence-pair-ordering dataset.
  • The generative scorers include uni, bi, and a variant that subtracts LSTM language-model probability. This variant rewards information gained from context rather than fluency alone.
  • HMM-LDA-GM conditions each decoding step on a topic vector z_n. Its topics proved too coarse to capture fine-grained coherence relations, which motivated the end-to-end VLV-GM.
  • In adversarial evaluation (AdverSuc), VLV-GM produces the least distinguishable text. Still, the best adver-3 score is only 0.081, and the evaluator exceeds 90% accuracy against every model.
  • Qualitative probes suggest the model is sensitive to lexical, temporal, causal, and referential (centering) coherence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

35 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 33 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1019

@inproceedings{li2017neural,
  title = {{Neural Net Models of Open-domain Discourse Coherence}},
  author = {Li, Jiwei and Jurafsky, Dan},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1019},
  url = {https://doi.org/10.18653/v1/d17-1019}
}

References

53 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ernst Althaus, Nikiforos Karamanis, and Alexander Koller. 2004. Computing locally coherent discourses. In Proceedings of ACL 2004. find
  2. Kannan Anjuli and Oriol Vinyals. 2016. Adversarial evaluation of dialogue models. find
  3. NIPS 2016 Workshop on Adversarial Training . find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of the International Conference on Learning Representations (ICLR). find our card
  5. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics 34(1):1–34. find our card
  6. Regina Barzilay and Lillian Lee. 2004. Catching the drift: Probabilistic content models, with applications to generation and summarization. In HLT-NAACL. pages 113–120. find
  7. Regina Barzilay and Kathleen R McKeown. 2005. Sentence fusion for multidocument news summarization. Computational Linguistics 31(3):297–328. find
  8. Justin Bayer and Christian Osendorfer. 2014. Learning stochastic recurrent networks. arXiv preprint arXiv:1411.7610 . — [arXiv:1411.7610](https://arxiv.org/abs/1411.7610) find
  9. Gillinder Bedi, Facundo Carrillo, Guillermo A Cec-chi, Diego Fernández Slezak, Mariano Sigman, Natália B Mota, Sidarta Ribeiro, Daniel C Javitt, Mauro Copelli, and Cheryl M Corcoran. 2015. Automated analysis of free speech predicts psychosis onset in high-risk youths. npj Schizophrenia 1. find
  10. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research 3(Jan):993–1022. find our card
  11. Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 . find our card
  12. Jill Burstein, Joel Tetreault, and Slava Andreyev. 2010. Using entity-based features to model coherence in student essays. In Human language technologies: The 2010 annual conference of the North American chapter of the Association for Computational Linguistics. Association for Computational Linguist… find
  13. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In Advances in neural information processing systems. pages 2980–2988. find
  14. Carl Doersch. 2016. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 . — [arXiv:1606.05908](https://arxiv.org/abs/1606.05908) find
  15. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research 12:2121–2159. find
  16. Micha Eisner and Eugene Charniak. 2011. Extending the entity grid with entity-specific features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. Association for Computational Linguistics, pages 125–129. find
  17. Micha Elsner, Joseph L Austerweil, and Eugene Charniak. 2007. A unified local and global model for discourse coherence. In HLT-NAACL. pages 436–443. find our card
  18. Micha Elsner and Eugene Charniak. 2008. Coreference-inspired coherence modeling. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers. Association for Computational Linguistics, pages 41–44. find our card
  19. Brita Elvevåg, Peter W Foltz, Daniel R Weinberger, and Terry E Goldberg. 2007. Quantifying incoherence in speech: An automated methodology and novel application to schizophrenia. Schizophrenia research 93(1):304–316. find
  20. Vanessa Wei Feng and Graeme Hirst. 2012. Extending the entity-based coherence model with multiple ranks. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, pages 315–324. find
  21. Peter W Foltz. 2007. Discourse coherence and lsa. Handbook of latent semantic analysis pages 167– 184. find
  22. Peter W Foltz, Walter Kintsch, and Thomas K Landauer. 1998. The measurement of textual coherence with latent semantic analysis. Discourse processes 25(2-3):285–307. find
  23. Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and Larry Heck. 2016. Contextual lstm (clstm) models for large scale nlp tasks. arXiv preprint arXiv:1602.06291 . — [arXiv:1602.06291](https://arxiv.org/abs/1602.06291) find
  24. Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. 2007. Hidden topic markov models. In AISTATS. volume 2, pages 163–170. find
  25. Camille Guinaudeau and Michael Strube. 2013. Graph-based local coherence modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. pages 93–103. find
  26. M. A. K. Halliday and Ruqaiya Hasan. 1976. Cohesion in English. Longman. find
  27. Eduard H Hovy. 1988. Planning coherent multisenten-tial text. In Proceedings of the 26th annual meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 163– 169. find
  28. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. arXiv preprint arXiv:1603.01913 . — [arXiv:1603.01913](https://arxiv.org/abs/1603.01913) find
  29. Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114 . find our card
  30. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems. pages 3276–3284. find
  31. Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1. Association for Computational Linguistics, pages 545–552. find
  32. Mirella Lapata. 2006. Automatic evaluation of information ordering: Kendall’s tau. Computational Linguistics 32(4):471–484. find
  33. Alex Lascarides and Nicholas Asher. 1991. Discourse relations and defeasible knowledge. In Proceedings of the 29th annual meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 55–62. find our card
  34. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015a. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055 . find our card
  35. Jiwei Li and Eduard Hovy. 2014. A model of coherence based on distributed sentence representation. In Proceedings of Empirical Methods in Natural Language Processing. find
  36. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015b. A hierarchical neural autoencoder for paragraphs and documents. arXiv preprint arXiv:1506.01057 . find
  37. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically evaluating text coherence using discourse relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, pages 997–10… find our card
  38. Annie Louis and Ani Nenkova. 2012. A coherence model based on syntactic patterns. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, pages 1157–1168. find
  39. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. EMNLP . find
  40. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text 8(3):243–281. find our card
  41. Daniel Marcu. 1997. From local to global coherence: A bottom-up approach to text planning. In AAAI/IAAI. Citeseer, pages 629–635. find
  42. Danielle S. McNamara, Max M. Louwerse, Philip M. McCarthy, and Arthur C. Graesser. 2010. Coh-metrix: Capturing linguistic features of cohesion. Discourse Processes 47(4):292–330. find
  43. Hongyuan Mei, Mohit Bansal, and Matthew R Walter. 2016. Coherent dialogue with attention-based language models. arXiv preprint arXiv:1611.06997 . — [arXiv:1611.06997](https://arxiv.org/abs/1611.06997) find
  44. Eleni Miltsakaki and Karen Kukich. 2004. Evaluation of text coherence for electronic essay scoring systems. Natural Language Engineering 10(01):25–55. find
  45. ACM SIGIR conference on Research and development in information retrieval. ACM, pages 735–736. find
  46. J. Morris and G. Hirst. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics 17(1):21–48. find our card
  47. Hongning Wang, Duo Zhang, and ChengXiang Zhai. 2011. Structural topic model for latent topical structure analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, pages 1526–1… find
  48. Michael Paul and Roxana Girju. 2010. A two-dimensional topic-aspect model for discovering multi-faceted topics. Urbana 51(61801):36. find
  49. Michael J Paul. 2012. Mixed membership markov models for unsupervised conversation modeling. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, pages 94–104. find
  50. Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2016. Topic augmented neural response generation with a joint attention mechanism. arXiv preprint arXiv:1606.08340 . — [arXiv:1606.08340](https://arxiv.org/abs/1606.08340) find
  51. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP. volume 14, pages 1532– 1543. find
  52. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 . find our card
  53. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082 . find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/li-2017-neural-net-models-open-domain/