Elamur Research⌕
☰
home / papers / 2026 / OPINE: A Prior-calibrated Scoring Framework for LLM-based Multi-label Scientific Opinion Classification

OPINE: A Prior-calibrated Scoring Framework for LLM-based Multi-label Scientific Opinion Classification

Paper · 2026 · Mengting Zhang, Gaofeng Pan, Zhixiong Zhang et al.

Summary

The paper addresses how to classify opinion sentences in scientific articles by the discourse role they play, such as interpreting results, judging methods, or noting limitations. This is a multi-label task, because one sentence can serve several roles. Labels also cluster unevenly across AIMRaD sections, which produces a long tail. Standard LLM prompting outputs labels directly, so the model's confidence stays hidden and section structure serves only as background context.

The authors propose OPINE, which treats the LLM as a scorer rather than a label generator. The pipeline works in four steps:

  • The LLM rates each candidate label separately.
  • These affinity scores are fused with AIMRaD-based section priors.
  • A quantile-based threshold converts the fused scores into label sets.
  • A pairwise step separates easily confused categories.

The authors also release a benchmark with an 18-class taxonomy. OPINE generally beats strong baselines. The results suggest that discourse structure helps more as an explicit decision-level prior than as plain prompt context. This is useful for downstream work such as finding research gaps or generating hypotheses.

Key points

  • Taxonomy and benchmark: The 18 opinion types are split into two groups: opinions about the scientific community (Group A) and opinions about the authors' own research (Group B). Each instance includes context metadata and AIMRaD section labels. Code and data are released at github.com/znoodle63/OPINE.
  • Headline results: The best reported F1 scores are 63.20% (Micro), 53.68% (Macro) and 63.22% (Example-based).
  • Scoring strategy: Label-wise scoring beat list-wise and path-wise scoring. For Phi-4, Micro-F1 was 39.09%, compared with 36.89% and 33.89% respectively.
  • Ablations: Experiments covered Qwen-3-14B, Phi-4 and DeepSeek-V3.2.
    • Removing the prior caused the largest drop, a 5.20-point Micro-F1 loss for Qwen-3-14B.
    • Pairwise refinement still helped when the prior was absent.
    • The prior/score fusion weight α performed best in the range 0.6–0.9.
  • Limitations:
    • The quantile threshold is tuned heuristically on validation data.
    • The method assumes a standard paper structure, so it may not transfer to non-linear or short formats.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China246 works · 24,951 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,743 citations🎓Zhengzhou UniversityZhengzhou · China4 works · 71 citations🔬National Science LibraryBeijing · China4 works · 16 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Mengting Zhang, Gaofeng Pan, Zhixiong Zhang, Yang Li and Guangyin Zhang. 2026. OPINE: A Prior-calibrated Scoring Framework for LLM-based Multi-label Scientific Opinion Classification. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1617

@inproceedings{zhang2026opine,
  title = {{OPINE: A Prior-calibrated Scoring Framework for LLM-based Multi-label Scientific Opinion Classification}},
  author = {Mengting Zhang, Gaofeng Pan, Zhixiong Zhang, Yang Li and Guangyin Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1617},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1617}
}

References

40 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Waleed Abdeen, Michael Unterkalmsteiner, Krzysztof Wnuk, Alessio Ferrari, and Panagiota Chatzipetrou. 2025. Language models to support multi-label classification of industrial data. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 45–55. find
  2. Amjad Abu-Jbara, Jefferson Ezra, and Dragomir R. Radev. 2013. Purpose and polarity of citation: Towards nlp-based bibliometrics. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 596–606. find
  3. Atilla Kaan Alkan, Shashwat Sourav, Maja Jablonska, Simone Astarita, Rishabh Chakrabarty, Nikhil Garuda, Pranav Khetarpal, Maciej Píoro, Dimitrios Tanoglidis, Kartheik G. Iyer, Mugdha S. Polimera, Michael J. Smith, Tirthankar Ghosal, Marc Huertas-Company, Sandor Kruk, Kevin Schawinski, and Ioana Ciu… — [arXiv:2504.05496](https://arxiv.org/abs/2504.05496) find
  4. Margaret Cargill and Patrick D. T. O’Connor. 2013. Writing Scientific Research Articles: Strategy and Steps. John Wiley & Sons. find
  5. Yi-Pei Chen, KuanChao Chu, and Hideki Nakayama. 2024. Llm as a scorer: The impact of output order on dialogue evaluation. arXiv preprint arXiv:2406.02863. — [arXiv:2406.02863](https://arxiv.org/abs/2406.02863) find
  6. Cheng-Han Chiang, Wei-Chih Chen, Chun-Yi Kuan, Chienchou Yang, and Hung yi Lee. 2024. Large language model as an assignment evaluator: Insights, feedback, and challenges in a 1000+ student course. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 2489–… find
  7. Kripa Jagannathan, Geniffer Emmanuel, James Arnott, Katharine J. Mach, Aparna Bamzai-Dodson, Kristen Goodrich, Ryan Meyer, Mark Neff, K. Dana Sjostrom, Kristin M. F. Timm, Esther Turnhout, Gabrielle Wong-Parodi, Angela T. Bednarek, Alison Meadow, Art Dewulf, Christine J. Kirchhoff, Richard H. Moss,… find
  8. Noriko Kando. 1997. Text-level structure of research papers: Implications for text-based information processing systems. In Proceedings of the 19th Annual BCS-IRSG Colloquium on IR Research. BCS Learning & Development. find
  9. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213.
  10. Jieran Li, Xiuyuan Hu, Yang Zhao, Shengyao Zhuang, and Hao Zhang. 2025. Leveraging reference documents for zero-shot ranking via large language models. arXiv preprint arXiv:2506.11452. — [arXiv:2506.11452](https://arxiv.org/abs/2506.11452) find
  11. Marcus Ma, Georgios Chochlakis, Niyantha Maruthu Pandiyan, Jesse Thomason, and Shrikanth Narayanan. 2025. Large language models do multi-label classification differently. arXiv preprint arXiv:2505.17510. find
  12. Michael J. Moravcsik and Poovanalingam Murugesan. 1975. Some results on the function and quality of citations. Social Studies of Science, 5(1):86–92. find
  13. Jingwei Ni, Minjing Shi, Dominik Stammbach, Mrinmaya Sachan, Elliott Ash, and Markus Leipold. 2024. Afacta: Assisting the annotation of factual claim detection with reliable LLM annotators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap… find
  14. Mohammad Rabiei, Seyyed-Mahdi Hosseini-Motlagh, and Abdorrahman Haeri. 2017. Using text mining techniques for identifying research gaps and priorities: a case study of the environmental science in iran. Scientometrics, 110(2):815–842. find
  15. Andrew Curtis. 2012. The science of subjectivity. Geology, 40(1):95–96. find
  16. Wanda Safira, Benedictus Prabaswara, Andrea Stevens Karnyoto, and Bens Pardamean. 2025. Leveraging ALBERT for sentiment classification of long-form ChatGPT reviews on twitter. International Journal of Computing and Digital Systems, 17(1):1–12. find
  17. Wenlong Hu, Qiang Fan, Hao Yan, Xinyao Xu, Shan Huang, and Ke Zhang. 2025. A survey of multi-label text classification under few-shot scenarios. Applied Sciences, 15(16):8872. find
  18. Hajar Sakai and Sarah S Lam. 2025. Quad-llm-mltc: Large language models ensemble learning for healthcare text multi-label classification. arXiv preprint arXiv:2502.14189. — [arXiv:2502.14189](https://arxiv.org/abs/2502.14189) find
  19. Edward J. Huth. 1987. Structured abstracts for papers reporting clinical trials. Annals of Internal Medicine, 106(4):626–627. find
  20. Ingeborg Spiegel-Rüsing. 1977. Science studies: Bibliometric and content analysis. Social Studies of Science, 7(1):97–113. find
  21. Shuoqi Sun, Shengyao Zhuang, Shuai Wang, and Guido Zuccon. 2025. An investigation of prompt variations for zero-shot llm-based rankers. In European Conference on Information Retrieval, pages 185–201. Springer. find
  22. John Swales. 2004. Research Genres: Explorations and Applications. Cambridge University Press. find
  23. John Swales. 2014. Create a research space (cars) model of research introductions. In Writing About Writing: A College Reader, pages 12–15. find
  24. Simone Teufel. 1999. Argumentative Zoning: Information Extraction from Scientific Text. Phd thesis, University of Edinburgh, Edinburgh, UK. find
  25. Simone Teufel. 2010. The Structure of Scientific Articles: Applications to Citation Indexing and Summarization. Center for the Study of Language and Information. find
  26. Simone Teufel, Jean Carletta, and Marc Moens. 1999. An annotation scheme for discourse-level argumentation in research articles. In Proceedings of the Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 110–117. find
  27. Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 103– 110.
  28. Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. 2022. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7):5731–5780. find
  29. Mingxuan Xia, Zhijie Jiang, Haobo Wang, Junbo Zhao, Tianlei Hu, and Gang Chen. 2025. Ensembling prompting strategies for zero-shot hierarchical text classification with large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025). find
  30. Ruichao Yang, Jing Ma, Wei Gao, and Hongzhan Lin. 2025. LLM-enhanced multiple instance learning for joint rumor and stance detection with social context information. ACM Transactions on Intelligent Systems and Technology, 16(3):1–27. 10th International Conference on Big Data Analytics, pages 300–308… find
  31. Mengting Zhang, Zhixiong Zhang, Yajiao Wang, Yang Li, Xin Lin, and Meng Wang. 2025b. Explain before classify: Contrastive rationale distillation for academic opinion recognition. In Proceedings of the International Conference on Advanced Data Mining and Applications, pages 311–325, Singapore. Spring… find
  32. Qian Zhang, Qinliang Su, Wei Zhu, and Yachun Pang. 2025c. Hierprompt: Zero-shot hierarchical text classification with llm-enhanced prototypes. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 3846–3859. find
  33. Yunyi Zhang, Ruozhen Yang, Xueqiang Xu, Rui Li, Jinfeng Xiao, Jiaming Shen, and Jiawei Han. 2025d. Teleclass: Taxonomy enrichment and llm-enhanced hierarchical text classification with minimal supervision. In Proceedings of the ACM on Web Conference 2025 (WWW ’25), pages 2032–2042. find
  34. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, and 1 others. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
  35. Yaxin Zhu and Hamed Zamani. 2024. Icxml: An in-context learning framework for zero-shot extreme multi-label classification. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2086–2098. find
  36. Yilun Zhu, Joel Ruben Antony Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, and Bo-Hsiang Tseng. 2024. Can large language models understand context? arXiv preprint arXiv:2402.00858. find
  37. Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2023. Beyond yes and no: Improving zero-shot llm rankers via scoring fine-grained relevance labels. arXiv preprint arXiv:2310.14122. find
  38. Kosuke Yoshimura and Hisashi Kashima. 2025. Hierarchical text classification using black box large language models. arXiv preprint arXiv:2508.04219. — [arXiv:2508.04219](https://arxiv.org/abs/2508.04219) find
  39. Jinze Yu and Guanghui Wang. 2025. Reasoning-enhanced prompt strategies for multi-label classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6065–6070. find
  40. Mengting Zhang, Yajiao Wang, Yufei Wang, and Zhixiong Zhang. 2025a. Sentiment-enhanced opinion sentence recognition using AgriBERT-SentiDPCNN and multi-opinion summarization via LLMs for agricultural scientific literature. In Proceedings of the find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-opine-prior-calibrated-scoring-framework/