A Unified Approach to Discourse Relation Classification in nine Languages

Summary

This paper is a submission to Task 3 of the DISRPT 2021 shared task, which asks systems to label the discourse relation between two text units. The authors restricted their work to the RST-based corpora, which span nine languages: Spanish, Russian, Portuguese, German, Farsi, English, Dutch, Chinese and Basque. Most other datasets had their text masked with underscores. Across these corpora they found 57 distinct relation labels, and each corpus uses between 16 and 29 of them. To make the task more manageable, they manually merged the labels into five broad groups, taking inspiration from the hierarchical scheme of PDTB.

The system uses only two features: the Euclidean distance between multilingual sBERT embeddings of the two units, and the relation direction. A stacked random forest first predicts the broad group, and then a separate model for each group assigns the fine-grained label.

Accuracy on the RST sets ranged from 35% for German to 70% for Chinese, and performance depended strongly on how often each label appeared in the training data. The work shows that a single, very simple pipeline can be applied across languages and frameworks.

Key points

Data choices: For Spanish, spa.rst.rststb was used. For English (eng.rst.gum), rows with hidden text were removed. Only four labels appear in every language: background, condition, contrast and elaboration.
Model setup: The random forest uses a one-vs-all scheme with balanced class weights. Training and dev data were merged and re-split with 40% held out. For the sparse Chinese set, the training data was duplicated into the dev set.
Results by language: Chinese reached 70% accuracy on 20 test labels, even though half of those labels were never predicted correctly. Elaboration still scored F1 0.93. Russian, the largest corpus, reached 61% and missed only 2 labels.
Organizers' evaluation: The organizers scored the system on all 16 datasets, including non-RST ones. Average accuracy was 54%, with the best score of 89% on zho.pdtb.cdtb.
Future work: The authors suggest clustering-based label grouping, fine-tuning the embeddings, and tuning hyperparameters separately for each language.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.disrpt-1.5

@inproceedings{varachkina2021unified,
  title = {{A Unified Approach to Discourse Relation Classification in nine Languages}},
  author = {Varachkina, Hanna and Pannach, Franziska},
  booktitle = {Proceedings of the 2nd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2021)},
  year = {2021},
  doi = {10.18653/v1/2021.disrpt-1.5},
  url = {https://doi.org/10.18653/v1/2021.disrpt-1.5}
}

References

7 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Farah Benamara, Maite Taboada, and Yannick Mathieu. 2016. Evaluative language beyond bags of words: Linguistic insights and computational applications. Computational Linguistics, 43:1–103. find
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the 6th Language Resources and Evaluation Conference. find our card
Hugo Hernault, Danushka Bollegala, and Mitsuru Ishizuka. 2010. Towards semi-supervised classification of discourse relations using feature correlations. In Proceedings of the SIGDIAL 2010 Conference, pages 55–58. find
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. find our card
Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. 2020. Implicit discourse relation classification: We need to talk about evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5404– 5414, Online. Association for Computational Lingui… find our card
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual universal sentence encoder for semantic retrieval. In Proceedings of the 58th Annual Meeting of the Association… find our card
Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/varachkina-2021-unified-approach-discourse-relation-classification/

📄 A Unified Approach to Discourse Relation Classification in nine Languages