Elamur Research⌕
☰
home / papers / 2013 / Disambiguating Explicit Discourse Connectives without Oracles

Disambiguating Explicit Discourse Connectives without Oracles

Summary

Before any PDTB-style discourse parser can work, it has to decide whether a candidate connective such as "and" or "because" is actually signalling a discourse relation. Errors at this stage carry over to every later step. The widely cited 96.29% accuracy of Pitler and Nenkova (2009) relies on gold-standard parse trees and is averaged over tokens, so a handful of frequent connectives dominate it. The authors re-evaluate the task on the Penn Discourse Treebank (PDTB) and WSJ under more realistic conditions: features come from automatically predicted parses and tags, and scores are macro-averaged across connective types as well as micro-averaged across instances. They compare the P&N feature set, its extension by Lin et al. (2010), and a new lightweight POS+LEX model that uses only words and predicted part-of-speech tags. Under realistic evaluation, performance drops sharply for the parse-based models. POS+LEX beats P&N and is statistically on par with Lin. Cheap, robust features are therefore sufficient, and the rare connectives that matter for applications such as text simplification still need more work.

Key points

  • On predicted features, macro-averaged F1 is 80.7 for P&N, 83.6 for Lin and 83.5 for POS+LEX, against a majority baseline of 53.9. A Wilcoxon signed-rank test found the difference between POS+LEX and Lin not significant at p < 0.01.
  • Under oracle conditions, P&N reaches 93.0 micro-F1, which falls to 85.3 when macro-averaged and to roughly 80.7 when predicted parses replace gold ones as well.
  • Connective frequencies in the PDTB (about 100 explicit types) follow a power law: the top 10 cover over 75% of occurrences and the top 20 over 90%. Still, 70% of WSJ articles with explicit connectives contain at least one outside the top 10, rising to 87% when very short texts are excluded.
  • Setup:
    • Parses come from the Stanford Parser, trained in a leave-one-section-out fashion, with an average F1 of 85.87%.
    • The classifier is maximum entropy, trained on WSJ sections 2–22 with ten-fold cross-validation.
  • Without an oracle, the average per-feature conditional entropy of P&N rises by 8.8%, while POS features barely change. For 58 connective types, Lin's parse features introduce at least as many new errors as they fix relative to POS+LEX.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{johannsen2013disambiguating,
  title = {{Disambiguating Explicit Discourse Connectives without Oracles}},
  author = {Johannsen, Anders and Søgaard, Anders},
  booktitle = {Proceedings of the Sixth International Joint Conference on Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/I13-1134/}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amal Alsaif and Katja Markert. 2011. Modelling discourse relations for Arabic. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 736–747, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  2. Michaela Atterer and Hinrich Schütze. 2007. Prepositional phrase attachment without oracles. Computational Linguistics, 33(4):469–476. find
  3. Jill Burstein and Martin Chodorow. 2010. Progress and New Directions in Technology for Automated Essay Evaluation. In R. Kaplan, editor, The Oxford Handbook of Applied Linguistics, 2nd Edition, pages 487–497. Oxford University Press. find
  4. Sucheta Ghosh. 2012. End-to-End Discourse Parse using Cascaded Structured Prediction. Phd thesis, University of Trento. find
  5. Hen-Hsen Huang and Hsin-Hsi Chen. 2011. Chinese discourse relation recognition. In Proceedings of 5th International Joint Conference on Natural Language Processing (IJCNLP 2011), pages 1442–1446. find our card
  6. J.-D. Kim, T. Ohta, Y. Tateisi, and J. Tsujii. 2003. GE-NIA corpus–a semantically annotated corpus for bio-textmining. Bioinformatics, 19(Suppl 1):i180–i182, July. find
  7. Ziheng Lin, Hwee Tou Ng, and Min-yen Kan. 2010. find
  8. A PDTB-Styled End-to-End Discourse Parser. Technical Report 2004, School of Computing, National University of Singapore, Singapore. find
  9. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically evaluating text coherence using discourse relations. pages 997–1006, June. find our card
  10. Christopher D. Manning. 2011. Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In AlexanderF. Gelbukh, editor, Computational Linguistics and Intelligent Text Processing SE - 14, volume 6608 of Lecture Notes in Computer Science, pages 171–189. Springer Berlin Heidelberg. find our card
  11. Thomas Meyer and Andrei Popescu-Belis. 2012. Using Sense-labeled Discourse Connectives for Statistical Machine Translation. In Proceedings of the Joint Workshop on Exploiting Synergies Between Information Retrieval and Machine Translation (ESIRMT) and Hybrid Approaches to Machine Translation (HyTra)… find our card
  12. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, ACLShort ’09, pages 13–16, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  13. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of LREC. find our card
  14. Rashmi Prasad, Susan McRoy, Nadya Frid, Aravind Joshi, and Hong Yu. 2011. The biomedical discourse relation bank. BMC Bioinformatics, 12(1):188. find
  15. Balaji Polepalli Ramesh and Hong Yu. 2010. Identifying discourse connectives in biomedical text. In AMIA Annual Symposium Proceedings, volume 2010, page 657. American Medical Informatics Association. find
  16. Valentin I Spitkovsky, Hiyan Alshawi, Angel X Chang, and Daniel Jurafsky. 2011. Unsupervised dependency parsing without gold part-of-speech tags. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1281–1290. Association for Computational Linguistics. find
  17. Gian Lorenzo Thione, Martin Van Den Berg, Livia Polanyi, and Chris Culy. 2004. Hybrid text summarization: Combining external relevance measures with structural analysis. In Proceedings ACL Workshop Text Summarization Branches Out. Barcelona. find
  18. Sara Tonelli and Elena Cabrio. 2012. Hunting for Entailing Pairs in the Penn Discourse Treebank. In Proceedings of COLING 2012, pages 2653–2668, Mumbai, India, December. find
  19. The COLING 2012 Organizing Committee. find
  20. Bonnie Webber and Aravind Joshi. 2012. Discourse Structure and Computation: Past, Present and Future. In Association for Computational Linguistics, page 42. find
  21. Zhi Min Zhou, Man Lan, Zheng Yu Niu, Yu Xu, and Jian Su. 2010. The effects of discourse connectives prediction on implicit discourse relation recognition. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL ’10, pages 139–146, Stroudsburg, PA, U… find our card

↑ back to top