Elamur Research⌕
☰
home / papers / 2023 / NUS-IDS at PragTag-2023: Improving Pragmatic Tagging of Peer Reviews through Unlabeled Data

NUS-IDS at PragTag-2023: Improving Pragmatic Tagging of Peer Reviews through Unlabeled Data

Paper · 2023 · Sujatha Das Gollapalli, Yixin Huang and See-Kiong Ng

Summary

The paper tackles PragTag-2023, a shared task in which each sentence of a peer review receives one pragmatic category: Recap, Strength, Weakness, Todo, Other or Structure. The data come from the F1000RD corpus, with unlabeled review corpora supplied as auxiliary material. Systems compete under three regimes: no labels, roughly a fifth of the labels, and the full training split. Scores are macro-F1 averaged over domains, and the final test uses an unseen "secret" domain. For the zero-label case the authors try two things: sentence-embedding semantic search driven by reviewer-style questions, and a multiple-choice prompt given to FlanT5. With labels, they fine-tune RoBERTa, T5 and FlanT5. They then pseudo-label the unlabeled reviews by comparing the three models' outputs, keeping only sentences where T5-large disagrees with the other two. This silver set is used either to pretrain T5-large before fine-tuning (PT) or merged with the gold data (Combined). On F1000RD, PT with 10% of the labels approaches full-data models. In the competition the team placed in the top two in every condition, although scores fell on the secret domain.

Key points

  • No-data results on the F1000RD test split: semantic search with QA-MPNet (multi-qa-mpnet-base-dot-v1) reached macro-F1 0.32. The MC-Prompt with FlanT5-XL reached 0.42, about half of what the full-data models achieve.
  • Low-data condition (10%, 338 labeled sentences): T5-large rose from 0.74 to 0.80 macro-F1 with PT, using about 14.7k pseudo-labeled sentences. This approaches full-data RoBERTa (0.82) and T5-large (0.82).
  • Full-data condition (2691 sentences): T5-large improved from 0.82 to 0.85 with PT and to 0.83 with Combined. PT raised F1 for five of the six classes, with the largest gain on Other (0.63 → 0.70).
  • Competition Phase 1:
    • The team ranked first in low-data (mean 0.813).
    • It tied the top full-data score (0.850, T5-large Combined).
    • On the secret domain in Phase 2, full-data F1 dropped to 0.741, which points to weak cross-domain generalization.
  • Adding every majority-voted unlabeled sentence lengthened training considerably without improving validation scores.

Interesting ideas

  • I1. The authors kept only the unlabeled examples where two auxiliary models agreed and the target model disagreed, so the silver data targets the target model's mistakes rather than repeating what it already knows.
  • I2. The paper compares two ways of using silver data: pretraining on it first and then fine-tuning on gold, or mixing both in one pool. The staged version worked better, especially with few labels.
  • I3. For zero-shot labeling, each category was described by typical questions a reviewer might ask, and sentences were matched to those questions by embedding search. This works as a cheap prototype-based labeler.
  • I4. Evaluation averages macro-F1 per domain and adds a hidden held-out domain, which exposed a generalization gap that in-domain tests hid.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When several witness parsers produce silver trees, Mova could prioritize sentences where the witnesses agree against the current parser. This would shrink the silver set and concentrate training on real errors.
  • I2 (learn): Mova's parsers and taggers could train on silver data first and then on gold treebank data as a separate stage. This should be compared against simple mixing before settling on one recipe.
  • I3 (frames): Speech-act or rhetorical-role labels in prag could be seeded with prototype questions or definitions per act. These would give first-pass silver tags that explicit rules then confirm or reject.
  • I4 (learn): Mova's evaluation could always hold out one genre, such as fables versus school texts. This would catch repair rules or trained models that overfit the genres seen in training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations🎓Institut Polytechnique de ParisPalaiseau · France15 works · 1,612 citations🎓Télécom ParisPalaiseau · France11 works · 826 citations🔬Institute of Data ScienceSingapore · Singapore2 works · 4 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Sujatha Das Gollapalli, Yixin Huang and See-Kiong Ng. 2023. NUS-IDS at PragTag-2023: Improving Pragmatic Tagging of Peer Reviews through Unlabeled Data. Proceedings of the 10th Workshop on Argument Mining.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.argmining-1.25

@inproceedings{gollapalli2023nus,
  title = {{NUS-IDS at PragTag-2023: Improving Pragmatic Tagging of Peer Reviews through Unlabeled Data}},
  author = {Sujatha {Das Gollapalli}, Yixin Huang and See-Kiong Ng},
  booktitle = {Proceedings of the 10th Workshop on Argument Mining},
  year = {2023},
  doi = {10.18653/v1/2023.argmining-1.25},
  url = {https://doi.org/10.18653/v1/2023.argmining-1.25}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peng Bao, Weihui Hong, and Xuanya Li. 2021. Predicting paper acceptance via interpretable decision sets. In Companion Proceedings of the Web Conference 2021, WWW ’21, page 461–467, New York, NY, USA. Association for Computing Machinery. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Razvan C. Bunescu and Oseremen O. Uduehi. 2022. Distribution-based measures of surprise for creative language: Experiments with humor and metaphor. In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 68–78, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  4. Alessandro Checco, Lorenzo Bracciale, Pierpaolo Loreti, Stephen Pinfield, and Giuseppe Bianchi. 2021. Ai-assisted peer review. Humanities and Social Sciences Communications, 8. ↗
  5. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299–4307. ↗
  6. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. ↗
  7. Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, and Dan Weld. 2019. Pretrained language models for sequential sentence classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3693–3699, Hong Kong, China. Association for Computational Linguistics. find
  8. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023a. NLPeer: A unified resource for the computational study of peer review. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5049– 5073, Toronto, Canada. Association for Computational Linguistics. ↗
  9. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023b. Overview of PragTag-2023: Low-resource multi-domain pragmatic tagging of peer reviews. In Proceedings of the 10th Workshop on Argument Mining, Singapore. Association for Computational Linguistics.
  10. Tirthankar Ghosal, Kamal Kaushik Varanasi, and Valia Kordoni. 2022. Hedgepeer: A dataset for uncertainty detection in peer reviews. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries, JCDL ’22, New York, NY, USA. Association for Computing Machinery. find
  11. Ketevan Glonti, Daniel Cauchi, Erik Cobo, Isabelle Boutron, David Moher, and Darko Hren. 2019. A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMC medicine, 17:1–14. ↗
  12. Christoph J Griessenauer and Michelle K Roach. 2019. Scientific peer review. A Guide to the Scientific Career: Virtues, Communication, Research and Academic Writing, pages 403–406. ↗
  13. Zihao He, Leili Tavabi, Kristina Lerman, and Mohammad Soleymani. 2021. Speaker turn modeling for dialogue act classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2150–2157, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  14. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL, pages 328–339. find
  15. Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019. Argument mining for understanding peer reviews. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2131–2137, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  16. Janine Huisman and Jeroen Smits. 2017. Duration and quality of the peer review process: the author’s perspective. Scientometrics, 113(1):633–650. ↗
  17. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1647–1661, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  18. Ilia Kuznetsov, Jan Buchmann, Max Eichler, and Iryna Gurevych. 2022. Revise and resubmit: An intertex-tual model of text-based collaboration in peer review. Computational Linguistics, 48(4):949–986. find
  19. Xinzhe Li, Qianru Sun, Yaoyao Liu, Shibao Zheng, Qin Zhou, Tat-Seng Chua, and Bernt Schiele. 2019. Learning to Self-Train for Semi-Supervised Few-Shot Classification. ↗
  20. Khalid Nassiri and Moulay Akhloufi. 2023. Transformer models used for text-based question answering systems. Applied Intelligence, 53(9):10602– 10635. ↗
  21. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. ↗
  22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1–67. ↗
  24. Robert Schulz, Adrian Barnett, René Bernard, Nicholas J.L. Brown, Jennifer A. Byrne, Peter Eckmann, Małgorzata A. Gazda, Halil Kilicoglu, Eric M. Prager, Maia Salholz-Hillel, Gerben ter Riet, Timothy Vines, Colby J. Vorland, Han Zhuang, Anita Bandrowski, and Tracey L. Weissgerber. 2022. Is the future of peer review automated? BMC Research Notes, 15(1). ↗
  25. Xiaohui Song, Longtao Huang, Hui Xue, and Songlin Hu. 2022. Supervised prototypical contrastive learning for emotion recognition in conversation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5197– 5206, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  26. Tiberiu Sosea and Cornelia Caragea. 2022. Leveraging training dynamics and self-training for text classification. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4750–4762, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  27. Richard Walker and Pascal Rocha da Silva. 2015. Emerging trends in peer review—a survey. Frontiers in Neuroscience, 9. ↗
  28. Jiquan Wang, Lin Sun, Yi Liu, Meizhi Shao, and Zeng-wei Zheng. 2022. Multimodal sarcasm target identification in tweets. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8164–8175, Dublin, Ireland. Association for Computational Linguistics.
  29. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA. Curran Associates Inc. find
  30. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771. ↗
  31. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? J. Artif. Int. Res., 75. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gollapalli-2023-nus-ids-pragtag-2023-improving/