Improving Implicit Discourse Relation Recognition Through Feature Set Optimization

Summary

Recognizing implicit discourse relations, where no connective such as "because" signals the link between two text spans, is much harder than handling explicit ones. Many feature types had been proposed for this task, but they had never been combined systematically, and differing setups made results hard to compare. Park and Cardie re-implemented the established features (word pairs, first/last words, polarity, General Inquirer tags, Levin verb classes, modality, context, and production rules) and trained binary one-vs-rest Naive Bayes classifiers on the Penn Discourse Treebank 2.0 (PDTB). They followed the experimental protocol of Pitler et al. (2009). Greedy forward selection then picked the best feature subset for each of the four top-level senses. The tuned combinations gave the strongest reported F1-scores for the four PDTB classes under this protocol. The authors also found that preprocessing choices, especially stemming before lexicon lookup, strongly affect results, while word pairs add almost nothing once richer features are present. Because they released code and described the setup in detail, the work serves as a reproducible baseline.

Key points

Data and setup: The experiments use PDTB annotations on WSJ text, with gold parses taken from the Penn Treebank. Training sets were drawn from sections 2–20 and balanced by down-sampling, and sections 21–22 served as the test set. EntRel instances were counted as EXPANSION.
Best F1 per sense, compared with Pitler '09:
- COMPARISON: 31.32 vs. 21.96
- CONTINGENCY: 49.82 vs. 47.13
- EXPANSION: 79.22 vs. 76.42
- TEMPORAL: 26.57 vs. 16.76
Most useful feature: Production rules were the strongest single feature, ranking first or second for every sense. The optimal subsets combined them with First-Last-First3, Verbs, Polarity, and/or Inquirer Tags.
Stemming: Applying Porter stemming before lexicon lookup raised F1 by roughly 10%. Hand-written matching rules improved lexicon coverage further.
Binning: Representing numeric counts, such as polarity counts, through cumulative binning noticeably lowered performance. Binary classifiers also outperformed 4-way classifiers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

55 papers of our library cite it (of the 140673 whose bibliographies we read); 32 of them have a page here, 23 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{park2012improving,
  title = {{Improving Implicit Discourse Relation Recognition Through Feature Set Optimization}},
  author = {Park, Joonsuk and Cardie, Claire},
  booktitle = {Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2012},
  url = {https://aclanthology.org/W12-1614/}
}

References

16 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL on Interactive presentation sessions, COLING-ACL ’06, pages 69–72, Stroudsburg, PA, USA. Association for Computational Linguistics. find
Sasha Blair-Goldensohn, Kathleen McKeown, and Owen Rambow. 2007. Building and refining rhetorical-semantic relation models. In HLT-NAACL, pages 428– 435. find
G. John, R. Kohavi, and K. Pfleger. 1994. Irrelevant Features and the Subset Selection Problem. In W. Cohen and H. Hirsh, editors, Proceedings of the Eleventh International Conference on Machine Learning, pages 121–129. Morgan Kaufmann. find
Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In IN PROCEEDINGS OF THE 41ST ANNUAL MEETING OF THE ASSOCIA-TION FOR COMPUTATIONAL LINGUISTICS, pages 423–430. find our card
Beth Levin and Harold Somers. 1993. English Verb Classes and Alternations: A Preliminary Investigation. University Of Chicago Press. find
Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In EMNLP, pages 343–351. find our card
Annie Louis, Aravind K. Joshi, Rashmi Prasad, and Ani Nenkova. 2010. Using entity features to classify implicit discourse relations. In SIGDIAL Conference, pages 59–62. find our card
Daniel Marcu and Abdessamad Echihabi. 2002. An unsupervised approach to recognizing discourse relations. In ACL, pages 368–375. find our card
M. Marcus, M. Marcinkiewicz, and B. Santorini. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkovak, Alan Lee, and Aravind K. Joshi. 2008. Easily identifiable discourse relations. In COLING (Posters), pages 87–90. find our card
Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In ACL/AFNLP, pages 683–691. find our card
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Bente Maegaard Joseph Mariani Jan Odjik Stelios Piperidis Daniel Tapias Nicoletta Calzolari (Conference Chair), Khalid Choukri, editor, Proceedings of… find our card
P J Stone, D C Dunphy, M S Smith, and D M Ogilvie. 1966. The General Inquirer: A Computer Approach to Content Analysis, volume 08. MIT Press. find
Ben Wellner, Lisa Ferro, Warren R. Greiff, and Lynette Hirschman. 2006. Reading comprehension tests for computer-based understanding evaluation. Natural Language Engineering, 12(4):305–334. find
Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In HLT/EMNLP. find
Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010. Predicting discourse connectives for implicit discourse relation recognition. In COLING (Posters), pages 1507–1514. find our card

↑ back to top