Elamur Research⌕
☰
home / papers / 2026 / Evaluating Generative Large Language Models for Portuguese Scientific Information Extraction

Evaluating Generative Large Language Models for Portuguese Scientific Information Extraction

Paper · 2026 · Tomás Pinto, Catarina Silva and Hugo Goncalo Oliveira

Summary

Scientific knowledge graphs (SciKGs) depend on reliable extraction of entities and relations from research papers. However, LLM-based scientific information extraction (IE) has been studied mostly for English. This paper tests how well mid-sized generative LLMs (8–12B parameters) perform this task for European Portuguese.

The authors translated and curated SciERC, a corpus of annotated computer science abstracts, into Portuguese to create a controlled benchmark. They compared several open models, including the government-backed sovereign Portuguese model AMALIA. The comparison covered two extraction setups: a pipeline with separate NER and RE steps, and joint triple extraction. It also covered two adaptation strategies: prompting and fine-tuning.

Performance was moderate overall. How a model is adapted mattered more than which model was used. Prompting gave unstable results with especially weak RE, while fine-tuning improved both NER and RE and reduced differences between models. The authors conclude that prompting alone is not enough for SciKG construction at this model scale. They also provide an error analysis and directions for Portuguese scientific IE.

Key points

  • New resource: a European Portuguese version of SciERC with 500 computer science abstracts, split 70/10/20 into train/dev/test.
  • Annotation schema:
    • 6 entity types, covering methods, tasks, and materials, along with Evaluation Metric, Other Sci Terms, and a Generic category.
    • 7 relation types, given in Portuguese (e.g., Usado-para, Parte-de, Hipónimo-de, Comparado-com, Conjunção).
  • Models: generative LLMs in the 8–12B range, including AMALIA, benchmarked against comparably sized multilingual open models.
  • Setups compared:
    • Pipelined NER→RE versus joint extraction.
    • Zero-shot and few-shot prompting (the joint few-shot prompt uses 3 examples with JSON triple output) versus fine-tuning.
  • Main finding: fine-tuning consistently beats prompting on both NER and RE and narrows cross-model variability. Prompted RE scores are poor.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CoimbraCoimbra · Portugal2 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksInformation extraction · 167 works

Cite

Cite as Tomás Pinto, Catarina Silva and Hugo Goncalo Oliveira. 2026. Evaluating Generative Large Language Models for Portuguese Scientific Information Extraction. Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5inv25yddbgv

@inproceedings{pinto2026evaluating,
  title = {{Evaluating Generative Large Language Models for Portuguese Scientific Information Extraction}},
  author = {Tomás Pinto, Catarina Silva and Hugo Goncalo Oliveira},
  booktitle = {Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/5inv25yddbgv},
  url = {https://doi.org/10.63317/5inv25yddbgv}
}

References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’Arcy, et al. 2026. Synthesizing scientific literature with retrieval-augmented language models. Nature, pages 1–7. find
  2. Sören Auer, Dante AC Barone, Cassiano Bartz, Eduardo G Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, et al. 2023. The sciqa scientific question answering benchmark for scholarly knowledge. Scientific Reports, 13(1):7240. find
  3. Sören Auer, Viktor Kovtun, Manuel Prinz, Anna Kasprzik, Markus Stocker, and Maria Esther Vidal. 2018. Towards a knowledge graph for science. In Proceedings of the 8th international conference on web intelligence, mining and semantics, pages 1–6. find
  4. Mykhailo Bondarenko, Sviatoslav Lushnei, Yurii Paniv, Oleksii Molchanovsky, Mariana Romanyshyn, Yurii Filipchuk, and Artur Kiulian. 2025. Sovereign large language models: Advantages, strategy and regulations. arXiv preprint arXiv:2503.04745. — [arXiv:2503.04745](https://arxiv.org/abs/2503.04745) find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Bruno Cabral, Daniela Claro, and Marlo Souza. 2024. Exploring open information extraction for Portuguese using large language models. In Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. 1, pages 127–136, Santiago de Compostela, Galicia/Spain. Associat… find
  7. John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S Rosen, Gerbrand Ceder, Kristin A Persson, and Anubhav Jain. 2024. Structured information extraction from scientific text with large language models. Nature communications, 15(1):1418.
  8. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient fine-tuning of quantized LLMs. Advances in neural information processing systems, 36:10088–10115. find
  9. Jose A Diaz-Garcia and Julio Amador Diaz Lopez. 2025. A survey on cutting-edge relation extraction techniques based on language models. Artificial Intelligence Review, 58(9):287. find
  10. Keyan Ding, Zhihui Zhu, Yuqi Tang, Kehua Feng, Xiang Zhuang, Hongwei Wang, Yi Yang, Huifang Du, Zhangkai Ni, Shiqi Wang, Xiaohui Fan, Huabin Xing, Lei Bai, Qi Liu, Haofen Wang, Qiang Zhang, and Huajun Chen. 2025. Bridging data and discovery: A survey on knowledge graphs in ai for science. find
  11. Markus Eberts and Adrian Ulges. 2020. Span-based joint entity and relation extraction with transformer pre-training. In ECAI 2020, pages 2006–2013. IOS Press. find
  12. Kata Gábor, Davide Buscaldi, Anne-Kathrin Schumann, Behrang QasemiZadeh, Haïfa Zargayouna, and Thierry Charnois. 2018. SemEval-2018 task 7: Semantic relation extraction and classification in scientific papers. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 679–688, N…
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Mark A Hanson, Pablo Gómez Barreiro, Paolo Crosetto, and Dan Brockington. 2024. The strain on scientific publishing. Quantitative Science Studies, 5(4):823–843. find
  15. Zhi Hong, Logan Ward, Kyle Chard, Ben Blaiszik, and Ian Foster. 2021. Challenges and advances in information extraction from scientific literature: a review. Jom, 73(11):3383–3400. find
  16. Devvrat Joshi and Islem Rekik. 2025. Dependency parsing-based syntactic enhancement of relation extraction in scientific texts. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 24888–24897. find
  17. Tanara Zingano Kuhn, José Matos, Bruno Neves, Daniela Pereira, Elisabete Cação, Ivo Simões, Jacinto Estima, Delfim Leão, and Hugo Gonçalo Oliveira. 2026. CorEGe-PT: Compiling a Large Corpus of Academic Texts in Portuguese. In Proceedings of 15th Language Resources and Evaluation Conference, LREC 202… find
  18. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3219– 3232, Brussels, Bel…
  19. Chuangtao Ma, Yongrui Chen, Tianxing Wu, Arijit Khan, and Haofen Wang. 2025. Large language models meet knowledge graphs for question answering: Synthesis and opportunities. arXiv preprint arXiv:2505.20099. find
  20. Sérgio Nunes, Alípio Mario Jorge, Evelin Amorim, Hugo Sousa, António Leal, Purificação Moura Silvano, Inês Cantante, and Ricardo Campos. 2024. Text2story lusa: A dataset for narrative analysis in european portuguese news articles. In Proceedings of the 2024 Joint International Conference on Computat… find
  21. Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2025. Graph retrieval-augmented generation: A survey. ACM Transactions on Information Systems, 44(2):1–52.
  22. Ciyuan Peng, Feng Xia, Mehdi Naseriparsa, and Francesco Osborne. 2023. Knowledge graphs: Opportunities and challenges. Artificial intelligence review, 56(11):13071–13102. find
  23. Miguel Moura Ramos, Duarte M Alves, Hippolyte Gisserot-Boukhlef, João Alves, Pedro Henrique Martins, Patrick Fernandes, José Pombal, Nuno M Guerreiro, Ricardo Rei, Nicolas Boizard, et al. 2026. Eurollm-22b: Technical report. arXiv preprint arXiv:2602.05879. — [arXiv:2602.05879](https://arxiv.org/abs/2602.05879) find
  24. Rodrigo Santos, João Silva, Luís Gomes, João Rodrigues, and António Branco. 2024. Advancing generative ai for portuguese with open decoder gervásio pt. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages@ LREC-COLING 2024, pages 16–26. find
  25. Tokala Yaswanth Sri Sai Santosh, Prantika Chakraborty, Sudakshina Dutta, Debarshi Kumar Sanyal, and Partha Pratim Das. 2021. Joint entity and relation extraction from scientific documents: Role of linguistic information and entity types. EEKE@ JCDL, 21:15–19. find
  26. Gabriel Silva, Mário Rodrigues, António Teixeira, and Marlene Amorim. 2024. Advancing open information extraction for portuguese by leveraging graph structures and large language models. In Proc. find
  27. IberSPEECH 2024, pages 61–65. find
  28. Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, Dav… find
  29. Hugo Sousa, Alipio Mario Jorge, Arian Pasquali, Catarina Santos, and Mario Lopes. 2023. A biomedical entity extraction pipeline for oncology health records in portuguese. In Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing, pages 950–956. find
  30. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  31. Trang Tran, Trung Hoang Le, Huiping Cao, and Tran Cao Son. 2025. An LLM+ ASP workflow for joint entity-relation extraction. arXiv preprint arXiv:2508.12611. — [arXiv:2508.12611](https://arxiv.org/abs/2508.12611) find
  32. Shilpa Verma, Rajesh Bhatia, Sandeep Harit, and Sanjay Batish. 2023. Scholarly knowledge graphs through structuring scholarly communication: a review. Complex & intelligent systems, 9(1):1059–1095. find
  33. Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. 2024. Large language models for generative information extraction: A survey. Frontiers of Computer Science, 18(6):186357.
  34. Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Kexin Xu, Yuqi Ye, and Hanwen Gu. 2025. A survey on multilingual large language models: Corpora, alignment, and bias. Frontiers of Computer Science, 19(11):1911362. find
  35. Zhaohui Yan, Zixia Jia, and Kewei Tu. 2022. An empirical study of pipeline vs. joint approaches to entity and relation extraction. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural… find
  36. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  37. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 18(6):1–32.
  38. Rui Yang, Boming Yang, Xinjie Zhao, Fan Gao, Aosong Feng, Sixun Ouyang, Moritz Blum, Tianwei She, Yuang Jiang, Freddy Lecue, et al. 2025b. Graphusion: A rag framework for scientific knowledge graph construction with a global perspective. In Companion Proceedings of the ACM on Web Conference 2025, pa… find
  39. Zhenyao Yang, Sha Yuan, Zhou Shao, Wenfa Li, and Runzhou Liu. 2025c. A review on synergizing knowledge graphs and large language models. Computing, 107(6):143. find
  40. Qi Zhang, Zhijia Chen, Huitong Pan, Cornelia Caragea, Longin Jan Latecki, and Eduard Dragut. 2024. Scier: An entity and relation extraction dataset for datasets, methods, and tasks in scientific documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pag… find
  41. Qinggang Zhang, Shengyuan Chen, Yuanchen Bei, Zheng Yuan, Huachi Zhou, Zijin Hong, Hao Chen, Yilin Xiao, Chuang Zhou, Junnan Dong, et al. 2025a. A survey of graph retrieval-augmented generation for customized large language models. arXiv preprint arXiv:2501.13958.
  42. Zikang Zhang, Wangjie You, Tianci Wu, Xinrui Wang, Juntao Li, and Min Zhang. 2025b. A survey of generative information extraction. In Proceedings of the 31st International conference on computational linguistics, pages 4840–4870. find
  43. Dong Zhao, Yadong Wang, Xiang Chen, Chenxi Wang, Hongliang Dai, Chuanxing Geng, Shengzhong Zhang, Shaoyuan Li, and Sheng-Jun Huang. 2025. Reflect then learn: Active prompting for information extraction guided by introspective confusion. arXiv preprint arXiv:2508.10036. — [arXiv:2508.10036](https://arxiv.org/abs/2508.10036) find
  44. Xiaoyan Zhao, Yang Deng, Min Yang, Lingzhi Wang, Rui Zhang, Hong Cheng, Wai Lam, Ying Shen, and Ruifeng Xu. 2024. A comprehensive survey on relation extraction: Recent advances and new frontiers. ACM Computing Surveys, 56(11):1–39. find
  45. Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. 2023. A comprehensive survey on automatic knowledge graph construction. ACM Computing Surveys, 56(4):1–62. find
  46. Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 50–61.
  47. Zhihui Zhu, Yuqi Tang, Qiang Zhang, and Keyan Ding. 2025. Synergizing large language models and knowledge graphs in science: A survey. In NeurIPS 2025 AI for Science Workshop. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pinto-2026-evaluating-generative-large-language-models/