Elamur Research⌕
☰
home / papers / 2023 / Overview of PragTag-2023: Low-Resource Multi-Domain Pragmatic Tagging of Peer Reviews

Overview of PragTag-2023: Low-Resource Multi-Domain Pragmatic Tagging of Peer Reviews

Summary

Most NLP work on peer review has used reviews from machine learning conferences. This raises two concerns: models may fail when applied to other research fields, and labeled review data is scarce and costly to produce. The PragTag-2023 shared task, held at the 10th Workshop on Argument Mining, tested both problems through pragmatic tagging. In this task, each sentence of a review is assigned an argumentative role such as Strength, Weakness or Todo.

Participants trained systems on labeled F1000RD reviews from five scientific domains. They worked under progressively harder data budgets (full, low and zero labeled data) and could optionally draw on unlabeled auxiliary corpora. Scoring used macro-F1 computed per domain and then averaged across domains. A surprise test set of computational linguistics reviews checked generalization to an unseen domain.

Five teams took part. Most fine-tuned BERT-family models, often with ensembling, pseudo-labeling or extra pre-training on the auxiliary data. Some recast the task, for example as question answering. The organizers argue that the competition's main value is this range of explored strategies, and they have released all code and data.

Key points

  • Label set: six pragmatic tags from Kuznetsov et al. (2022): Recap, Strength, Weakness, Todo, Other and Structure. Inter-annotator agreement in the original study was about 0.7 Krippendorff's α.
  • Core data (F1000RD): 4911 sentences from 224 reviews across five domains (diso, iscb, case, rpkg, scip). The data was split by review into 2326 training and 2585 test sentences. The low-data setting used 33 reviews (739 sentences).
  • Secret test domain: 255 sentences from 10 COLING-20 reviews taken from the NLPeer corpus, annotated with α = 0.65 and adjudicated by an expert.
  • Auxiliary unlabeled data: F1000Raw contains about 10k reviews of 4.8k papers (3.8M words). ARR-22 contains 684 semi-structured reviews of 476 papers (about 266k words), whose form fields could serve as distant supervision.
  • Setup and resources: the task ran on CodaLab with roberta-base and majority-class baselines. Only open-weight models trained without extra pre-training data were allowed. Code and data are available at github.com/UKPLab/pragtag2023.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.argmining-1.21

@inproceedings{dycke2023overview,
  title = {{Overview of PragTag-2023: Low-Resource Multi-Domain Pragmatic Tagging of Peer Reviews}},
  author = {Dycke, Nils and Kuznetsov, Ilia and Gurevych, Iryna},
  booktitle = {Proceedings of the 10th Workshop on Argument Mining},
  year = {2023},
  doi = {10.18653/v1/2023.argmining-1.21},
  url = {https://doi.org/10.18653/v1/2023.argmining-1.21}
}

References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7000–7011, Online. Association for Computational Linguistics. find
  2. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. arXiv:2210.11416. ↗
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Yuning Ding, Marie Bexte, and Andrea Horbach. 2023. CATALPA_EduNLP at PragTag-2023. In Proceedings of the 10th Workshop on Argument Mining, Singapore. Association for Computational Linguistics. find
  6. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey on in-context learning. arXiv:2301.00234. ↗ our card
  7. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2022. Yes-yes-yes: Proactive data collection for acl rolling review and beyond. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 300–318. https://intertext.ukp-lab.de https://argmining-org.github.io/2023/index. html Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023. NLPeer: A unified resource for the computational study of peer review. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5049– 5073, Toronto, Canada. Association for Computational Linguistics. ↗
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. ↗
  9. Sujatha Das Gollapalli, Yixin Huang, and See-Kiong Ng. 2023. NUS-IDS at PragTag-2023: Improving pragmatic tagging of peer reviews through unlabeled data. In Proceedings of the 10th Workshop on Argument Mining, Singapore. Association for Computational Linguistics. find
  10. Zhipeng Luo, Jiahui Wang, and Yihao Guo. 2023. Deep-BlueAI at PragTag-2023: Ensemble-based text classification approaches under limited data resources. In Proceedings of the 10th Workshop on Argument Mining, Singapore. Association for Computational Linguistics. find
  11. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv:2111.09543. ↗ our card
  12. Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019. Argument mining for understanding peer reviews. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2131–2137, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  13. Rob Johnson, Anthony Watkinson, and Michael Mabe. 2018. The STM Report: An overview of scientific and scholarly publishing. International Association of Scientific, Technical and Medical Publishers, The Hague, Netherlands. find
  14. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1647–1661, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  15. Neha Kennard, Tim O’Gorman, Rajarshi Das, Akshay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, and Andrew McCallum. 2022. DISAPERE: A dataset for discourse structure in peer review discussions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1234–1249, Seattle, United States. Association for Computational Linguistics. find
  16. Ilia Kuznetsov, Jan Buchmann, Max Eichler, and Iryna Gurevych. 2022. Revise and resubmit: an intertex-tual model of text-based collaboration in peer review. Computational Linguistics, 48(4):949–986. find
  17. Yoonsang Lee, Dongryeol Lee, and Kyomin Jung. 2023. MILAB at PragTag-2023: Enhancing cross-domain generalization through data augmentation with reduced uncertainty. In Proceedings of the 10th Workshop on Argument Mining, Singapore. Association for Computational Linguistics. GPT-4 technical report. find our card
  18. Adrien Pavao, Isabelle Guyon, Anne-Catherine Letour-nel, Dinh-Tuan Tran, Xavier Baro, Hugo Jair Es-calante, Sergio Escalera, Tyler Thomas, and Zhen Xu. 2023. Codalab competitions: An open source platform to organize scientific challenges. Journal of Machine Learning Research, 24(198):1–6. find
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683v1. ↗
  20. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. ↗
  21. Anna Rogers and Isabelle Augenstein. 2020. What can we do to improve peer review in NLP? In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1256–1262, Online. ACL. find
  22. Ivan Stelmakh, Nihar B Shah, and Aarti Singh. 2020. Catch me if i can: Detecting strategic behaviour in peer assessment. In ICML Workshop on Incentives in Machine Learning. ↗
  23. Kunal Suri, Prakhar Mishra, and Albert Nanda. 2023. SuryaKiran at PragTag 2023 - benchmarking domain adaptation using masked language modeling in natural language processing for specialized data. In Proceedings of the 10th Workshop on Argument Mining, Singapore. Association for Computational Linguistics. find
  24. Simone Teufel, Advaith Siddharthan, and Colin Batchelor. 2009. Towards domain-independent argumentative zoning: Evidence from chemistry and computational linguistics. In Proceedings of the 2009 conference on empirical methods in natural language processing, pages 1493–1502. find
  25. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. ↗
  26. Jingyan Wang and Nihar B Shah. 2018. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. arXiv:1806.05085. ↗
  27. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. arXiv:1904.09675. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dycke-2023-overview-pragtag-2023-low-resource/