Elamur Research⌕
☰
home / papers / 2013 / Comparative Evaluation of Argument Extraction Algorithms in Discourse Relation Parsing

Comparative Evaluation of Argument Extraction Algorithms in Discourse Relation Parsing

Summary

The paper addresses argument span extraction for explicit discourse relations in the Penn Discourse Treebank (PDTB). In this subtask, a parser must mark which tokens belong to Arg1 and Arg2 of a connective. Arg1 may sit in the connective's sentence (intra-sentential, SS) or in an earlier one (inter-sentential, PS). This raises a design question: should one model handle both cases, or should each case get its own model?

The authors start from the method of Ghosh et al. (2011), which labels each token with a CRF and uses a single model over a ±2 sentence window. They compare it with a pipeline that first predicts argument position (SS vs. PS) and then applies dedicated CRF models. For PS relations, this pipeline adds an immediately-previous-sentence heuristic to select the Arg1 candidate.

When position-classifier errors are counted against the separate-model parser, it still significantly beats the single-model parser for both arguments. The gain is largest for inter-sentential Arg1. The results show that splitting the task reduces its complexity. Token-level labeling also performs roughly on par with tree-node and constituent-level parsers.

Key points

  • PDTB statistics: of 18,459 explicit relations, 60.9% are SS, 39.1% are PS, and only 8 have Arg1 in a following sentence. In 44.2% of inter-sentential cases, Arg1 is exactly the whole immediately previous sentence.
  • Predicting SS vs. PS from syntactic category and connective position alone gives 95.0% majority accuracy. The trained argument position classifier reaches 98.12%.
  • On all relations, F1 scores with error propagation are:
    • single ±2 window parser (W5P): Arg2 77.79, Arg1 53.40
    • separate CRF models (SMP): Arg2 84.67, Arg1 58.25
    • separate models with heuristics (hSMP): Arg2 83.06, Arg1 56.71
  • On PS relations, Arg1 F1 improves from 29.01 (W5P) to 40.86 (SMP). Compared with the heuristics, the CRFs help significantly for Arg2 but not significantly for Arg1.
  • With training on sections 02-21 and testing on section 23, SMP scores Arg2/Arg1 F1 of 82.35/57.26. Lin et al. (2012) report 82.23/59.15 and Xu et al. (2012) report 81.00/60.69, though the authors consider this comparison not definitive.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{stepanov2013comparative,
  title = {{Comparative Evaluation of Argument Extraction Algorithms in Discourse Relation Parsing}},
  author = {Stepanov, Evgeny and Riccardi, Giuseppe},
  booktitle = {Proceedings of the 13th International Conference on Parsing Technologies (IWPT 2013)},
  year = {2013},
  url = {https://aclanthology.org/W13-5704/}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sabine Buchholz. 2000. Readme for perl script chunklink.pl. find
  2. Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2005. Attribution and the (non)-alignment of syntactic and discourse arguments of connectives. In Proceedings of the ACL Workshop on Frontiers in Corpus Annotation II: Pie in the Sky. find
  3. Robert Elwell and Jason Baldridge. 2008. Discourse connective argument identification with connective specific rankers. In Proceedings of the IEEE International Conference on Semantic Computing (ICSC 2008). find
  4. Sucheta Ghosh, Richard Johansson, Giuseppe Riccardi, and Sara Tonelli. 2011. Shallow discourse parsing with conditional random fields. In Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP 2011). find our card
  5. Sucheta Ghosh, Richard Johansson, Giuseppe Riccardi, and Sara Tonelli. 2012a. Improving the recall of a discourse parser by constraint-based postprocessing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012). find
  6. Sucheta Ghosh, Giuseppe Riccardi, and Richard Johansson. 2012b. Global features for shallow discourse parsing. In Proceedings of the SIGDIAL 2012 Conference, The 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find our card
  7. Dan Klein and Christopher D. Manning. 2003. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems 15 (NIPS 2002). find
  8. Alistair Knott. 1996. A Data-Driven Methodology for Motivating a Set of Coherence Relations. Ph.D. thesis, University of Edinburgh. find
  9. John Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of 18th International Conference on Machine Learning. find
  10. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2012. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 1:1 – 35. find
  11. Guido Minnen, John Carroll, and Darren Pearce. 2001. Applied morphological processing of english. Natural Language Engineering. find
  12. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP Conference. find our card
  13. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). find our card
  14. Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2010. Exploiting scope for shallow discourse parsing. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC-2010). find our card
  15. Robert E. Schapire and Yoram Singer. 2000. Boostexter: A boosting-based system for text categorization. Machine Learning, 39(2/3):135–168. find
  16. Maite Taboada and William C. Mann. 2006. Applications of rhetorical structure theory. Discourse Studies, (8):567–88. find
  17. Bonnie L. Webber, Markus Egg, and Valia Kordoni. 2011. Discourse structure and language technology. Natural Language Engineering, pages 1 – 54. find
  18. Ben Wellner and James Pustejovsky. 2007. Automatically identifying the arguments of discourse connectives. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2007). find
  19. Fan Xu, Qiao Ming Zhu, and Guo Dong Zhou. 2012. A unified framework for discourse argument identification via shallow semantic parsing. In Proceedings of 24th International Conference on Computational Linguistics (COLING 2012): Posters. find
  20. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proceedings of 8th International Workshop on Parsing Technologies. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/stepanov-2013-comparative-evaluation-argument-extraction-algorithms/