Elamur Research⌕
☰
home / papers / 2022 / Label distributions help implicit discourse relation classification

Label distributions help implicit discourse relation classification

Summary

Implicit discourse relations often allow more than one reading, yet most classifiers are trained as if each argument pair had a single correct sense. The authors use DiscoGeM, an English corpus covering Europarl, novels and Wikipedia, in which every implicit relation was labelled by ten crowdworkers. They address two questions. First, they measure how sense pairs co-occur using NPMI over each instance's top two labels. Pairs such as CONTRAST/CONCESSION and INSTANTIATION/LEVEL-OF-DETAIL/REASON appear in all genres, but other pairings depend on genre. Second, they train BMGF-RoBERTa, with a genre token added to the input, on PDTB 3.0 combined with DiscoGeM. They compare three regimes: majority label only, instances duplicated in proportion to the crowd vote distribution, and an equally upsampled majority-label control. The distribution-based model matches the single-label model on accuracy and macro F1. It achieves lower cross entropy against the crowd distributions, including on PDTB. This indicates that its predicted distributions are closer to human interpretations, which matters for discourse parsing that must handle ambiguity.

Key points

  • Data: DiscoGeM provides 6,505 inter-sentential implicit relations with 10 crowd labels each, split 70/20/10 into train/test/dev with some held-out unseen texts. PDTB 3.0 contributes 15,544 inter-sentential and 6,188 intra-sentential implicit relations, using the Ji and Eisenstein (2015) split.
  • Genre effects in co-occurrence: In Wikipedia, relations co-occur with CONJUNCTION more often than in other genres. In novels, RESULT tends to pair with PRECEDENCE. In Europarl, RESULT more often stands alone.
  • Training setup: For the multi-label model, labels with under 20% of votes were dropped before upsampling. All results are averaged over five runs in an 11-way setting.
  • Accuracy (single vs. multi): On all of DiscoGeM, accuracy was 48.66 vs. 49.91. On the PDTB 3.0 test set, it was 55.28 vs. 54.82.
  • Cross entropy (lower is better): The multi model improved over the single model, from 1.86 to 1.79 on DiscoGeM and from 1.48 to 1.36 on PDTB. The uncertainty control (1.83 / 1.48) did not match these gains, suggesting the benefit comes from the label distribution itself rather than from reweighting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yung2022label,
  title = {{Label distributions help implicit discourse relation classification}},
  author = {Yung, Frances and Anuranjana, Kaveri and Scholman, Merel and Demberg, Vera},
  booktitle = {Proceedings of the 3rd Workshop on Computational Approaches to Discourse},
  year = {2022},
  doi = {10.22028/d291-38834},
  url = {https://doi.org/10.22028/d291-38834}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Current and new directions in discourse and dialogue, pages 85–112. Springer. find our card
  2. Vera Demberg, Merel CJ Scholman, and Fatemeh Torabi Asr. 2019. How compatible are our discourse annotation frameworks? insights from mapping rst-dt and pdtb annotations. Dialogue & Discourse, 10(1):87– 135. find
  3. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. find
  4. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. find our card
  5. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The Penn Discourse Treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania. find
  6. Jianhua Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory, 37(1):145–151. find
  7. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2021. On the importance of word and sentence representation learning in implicit discourse relation classification. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3830–38… find
  8. Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. 2019. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617–9626. find
  9. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Asso… find our card
  10. Rashmi Prasad, Eleni Miltsakaki, Nikhil Dinesh, Alan Lee, Aravind Joshi, Robaldo Robaldo, and Bonnie Webber. 2007. The penn discourse treebank 2.0 annotation manual. Philadelphia, University of Pennsylvania. find
  11. Livio Robaldo and Eleni Miltsakaki. 2014. Corpus-driven semantics of concession: Where do expectations come from? Dialogue & Discourse, 5(1):1–36. find

↑ back to top

This page in Markdown →