Elamur Research⌕
☰
home / papers / 2011 / Automatically Evaluating Text Coherence Using Discourse Relations

Automatically Evaluating Text Coherence Using Discourse Relations

Summary

The paper addresses automatic judgment of local text coherence through the text ordering ranking task: a system sees an original document and a sentence-shuffled version and must identify the source. The authors argue that well-written text prefers certain orderings of discourse relations, both within a single relation and between neighbouring ones. They apply the freely available PDTB-style parser of Lin et al. (2010) and keep the PDTB Level-1 types plus EntRel and NoRel. A pilot model based on plain relation-transition n-grams suffered from sparse data. To fix this, they build a discourse role matrix. Its rows are sentences and its columns are stemmed open-class terms. Each cell records the term's roles, meaning a relation type combined with an argument tag (e.g. Comp.Arg1). Probabilities of per-term role sub-sequences across adjacent sentences become features for preference ranking. On WSJ, Earthquakes and Accidents, the model beats the Barzilay and Lapata (2005) entity grid and complements it. This is the first evidence that automatic discourse parser output improves coherence modelling.

Key points

  • Compared with the entity-based model, the error rate drops by 29% on average across the three data sets, measured against human upper bounds. The paper also gives the first estimate of human performance on this task.
  • Merging the features of the discourse-role model and the entity grid cuts error by 73%, which shows that the two signals are complementary.
  • The best single configuration is Type+Arg+Sal. On WSJ, learning curves level off after about 8,000 training pairs. Roughly half of the training data is enough to reach peak performance.
  • Texts dense in the four "real" relations (Temp, Cont, Comp, Exp) are easier to rank. The ratio of these relations to text length is 0.58 for correctly ranked permutations and 0.48 for misranked ones.
  • Parser quality is a bottleneck. Explicit relations are identified with about 93% accuracy, but implicit Level-2 relations reach only about 40%. Modelling longer-range transitions is suggested as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

73 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 60 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lin2011automatically,
  title = {{Automatically Evaluating Text Coherence Using Discourse Relations}},
  author = {Lin, Ziheng and Ng, Hwee Tou and Kan, Min-Yen},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1100/}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: an entity-based approach. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pages 141–148, Morristown, NJ, USA. Association for Computational Linguistics. find
  2. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34:1–34, March. find our card
  3. Regina Barzilay and Lillian Lee. 2004. Catching the drift: Probabilistic content models, with applications to generation and summarization. In Proceedings of the Human Language Technology Conference / North American Chapter of the Association for Computational Linguistics Annual Meeting 2004. find
  4. Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007. A unified local and global model for discourse coherence. In Proceedings of the Conference on Human Language Technology and North American Chapter of the Association for Computational Linguistics (HLT-NAACL 2007), Rochester, New York, USA,… find our card
  5. Robert Elwell and Jason Baldridge. 2008. Discourse connective argument identification with connective specific rankers. In Proceedings of the IEEE International Conference on Semantic Computing (ICSC 2010), Washington, DC, USA. find
  6. Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995. Centering: a framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203–225, June. find our card
  7. Thorsten Joachims. 1999. Making large-scale support vector machine learning practical. In Bernhard Schlkopf, Christopher J. C. Burges, and Alexander J. Smola, editors, Advances in Kernel Methods – Support Vector Learning, pages 169–184. MIT Press, Cambridge, MA, USA. find
  8. Nikiforos Karamanis. 2007. Supplementing entity coherence with local rhetorical relations for information ordering. Journal of Logic, Language and Information, 16:445–464, October. find
  9. Mirella Lapata and Regina Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In Leslie Pack Kaelbling and Alessandro Saf-fiotti, editors, Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, UK. find
  10. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP 2009), Singapore. find our card
  11. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2010. A PDTB-styled end-to-end discourse parser. Technical Report TRB8/10, School of Computing, National University of Singapore, August. find
  12. William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8(3):243–281. find our card
  13. Daniel Marcu. 1996. Distinguishing between coherent and incoherent texts. In The Proceedings of the Student Conference on Computational Linguistics in Montreal, pages 136–143. find
  14. Jane Morris and Graeme Hirst. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics, 17:21– 48, March. find our card
  15. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, Singapore. find our card
  16. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING 2008) Short Papers, Manchester, UK. find our card
  17. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP (ACL-IJCNLP 2009)… find our card
  18. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse Treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). find our card
  19. Radu Soricut and Daniel Marcu. 2006. Discourse generation using utility-trained coherence models. In Proceedings of the COLING/ACL Main Conference Poster Sessions, pages 803–810, Morristown, NJ, USA. Association for Computational Linguistics. find
  20. WenTing Wang, Jian Su, and Chew Lim Tan. 2010. Kernel based discourse relation recognition with temporal ordering information. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010), Uppsala, Sweden, July. find our card
  21. Bonnie Webber. 2004. D-LTAG: Extending lexicalized TAG to discourse. Cognitive Science, 28(5):751–779. find
  22. Ben Wellner and James Pustejovsky. 2007. Automatically identifying the arguments of discourse connectives. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2007), Prague, Czech Republic. find

↑ back to top

This page in Markdown →