Elamur Research⌕
☰
home / papers / 2026 / Benchmarking Portuguese Open Information Extraction

Benchmarking Portuguese Open Information Extraction

Summary

Portuguese Open Information Extraction (OIE) has had no common evaluation standard. English has OIE2016 and CaRB, but Portuguese systems could not be compared against a shared baseline. The authors, from the University of Aveiro, address this with a reproducible benchmarking framework rather than a new dataset. They surveyed six Portuguese corpora and thirteen systems. From these, they kept three human-validated corpora covering distinct domains: WIKI200 (Wikipedia), CETEN200 (news) and Gamalho (ecology-focused scientific text). They then scored eight rule-based and neural extractors, released over roughly a decade, using the CaRB methodology. No system leads on every corpus. Rule-based tools do well on the two general-domain sets but decline on the specialised Gamalho texts. Neural systems are steadier across corpora but not stronger overall. Average F1 is close to 40%, so the task remains far from solved. The benchmark gives future Portuguese OIE work a common reference point, and the authors call for a larger, fully hand-annotated gold standard.

Key points

  • The benchmark contains 506 sentences and 1151 extractions in total:
    • Gamalho: 103 sentences, 358 extractions
    • CETEN200: 202 sentences, 366 extractions
    • WIKI200: 201 sentences, 427 extractions
  • Selection criteria were human involvement in annotation, free availability, domain diversity, coverage of language variant, and at least 100 sentences. All three chosen corpora are Pt-Br, so European Portuguese is not covered.
  • Three corpora were excluded:
    • IBERLEF: only 25 sentences, with domain overlap.
    • TransAlign: automatically generated, one triple per sentence, no human validation.
    • OIEC-PT: the download link was inactive.
  • Thirteen candidate systems (2015–2024) were considered, including ArgOE (via Linguakit), RePort, DptOIE, PortNOIE, Multi²OIE, DetIE and PortOIE-Llama. Eight of them were evaluated.
  • The evaluation framework is public at github.com/gabrielrsilva11/PT-OIE-Benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2wuxjj5bo7ax

@inproceedings{silva2026benchmarking,
  title = {{Benchmarking Portuguese Open Information Extraction}},
  author = {Silva, Gabriel and Rodrigues, Mário and Teixeira, António and Amorim, Marlene},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2wuxjj5bo7ax},
  url = {https://doi.org/10.63317/2wuxjj5bo7ax}
}

References

36 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sangnie Bhardwaj, Samarth Aggarwal, and Mausam Mausam. 2019. CaRB: A crowdsourced benchmark for open IE. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6263–6268… find
  2. Bruno Cabral, Daniela Claro, and Marlo Souza. 2024. Exploring open information extraction for portuguese using large language models. In Proceedings of the 16th International Conference on Computational Processing of Portuguese-Vol. 1, pages 127–136. find
  3. Bruno Cabral, Marlo Souza, and Daniela Barreiro Claro. 2022. Portnoie: A neural framework for open information extraction for the portuguese language. In International Conference on Computational Processing of the Portuguese Language, pages 243–255. Springer. find
  4. Bruno Souza Cabral, Rafael Glauber, Marlo Souza, and Daniela Barreiro Claro. 2020. Crossoie: Cross-lingual classifier for open information extraction. In International conference on computational processing of the Portuguese language, pages 368–378. Springer. find
  5. Sandra Collovini, Joaquim Francisco Santos Neto, Bernardo Scapini Consoli, Juliano Terra, Renata Vieira, Paulo Quaresma, Marlo Souza, Daniela Barreiro Claro, and Rafael Glauber. 2019. find
  6. Iberlef 2019 portuguese named entity recognition and relation extraction tasks. In Iber-LEF@ SEPLN, pages 390–410. find
  7. Leandro Souza de Oliveira, Rafael Glauber, and Daniela Barreiro Claro. 2017. Dependentie: An open information extraction system on portuguese by a dependence analysis. Encontro Nacional de Inteligência Artificial e Computacional. find
  8. Luciano Del Corro and Rainer Gemulla. 2013. Clausie: clause-based open information extraction. In Proceedings of the 22nd international conference on World Wide Web, pages 355–366. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics. find
  10. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the Conference of Empirical Methods in Natural Language Processing (EMNLP ’11), Edinburgh, Scotland, UK. find
  11. Pablo Gamallo and Marcos Garcia. 2015. Multilingual open information extraction. In Progress in Artificial Intelligence, pages 711–722, Cham. Springer International Publishing. find
  12. Pablo Gamallo, Marcos Garcia, César Piñeiro, Rodrigo Martinez-Castaño, and Juan C. Pichel. 2018. Linguakit: A big data-based multilingual tool for linguistic analysis and information extraction. In 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS), page… find
  13. Kiril Gashteovski, Mingying Yu, Bhushan Kotnis, Carolin Lawrence, Mathias Niepert, and Goran Glavaš. 2022. BenchIE: A framework for multifaceted fact-based open information extraction evaluation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… find
  14. Keshav Kolluru, Vaibhav Adlakha, Samarth Aggarwal, Mausam, and Soumen Chakrabarti. 2020. OpenIE6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3748–3761, Onl… find
  15. William Lechelle, Fabrizio Gotti, and Phillippe Langlais. 2019. WiRe57 : A fine-grained benchmark for open information extraction. In Proceedings of the 13th Linguistic Annotation Workshop, pages 6–15, Florence, Italy. Association for Computational Linguistics. find
  16. Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Unified structure generation for universal information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5755–5772, Dublin… find
  17. Mausam Mausam. 2016. Open information extraction systems and downstream applications. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJ-CAI’16, page 4074–4077. AAAI Press. find
  18. Ritwik Mishra, Simranjeet Singh, Rajiv Ratn Shah, Ponnurangam Kumaraguru, and Pushpak Bhattacharyya. 2023. IndIE: A multilingual open information extraction tool for Indic languages. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 312–326, Nusa Dua, B… find
  19. Dan Nielsen. 2023. ScandEval: A benchmark for Scandinavian natural language processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185–201, Tórshavn, Faroe Islands. University of Tartu Library. find
  20. Leandro Oliveira, Daniela Barreiro Claro, and Marlo Souza. 2023. Dptoie: a portuguese open information extraction based on dependency analysis. Artificial Intelligence Review, 56(7):7015–7046. find
  21. Liu Pai, Wenyang Gao, Wenjie Dong, Lin Ai, Ziwei Gong, Songfang Huang, Li Zongsheng, Ehsan Hoque, Julia Hirschberg, and Yue Zhang. 2024. find
  22. A survey on open information extraction from rule-based model to large language model. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9586–9608, Miami, Florida, USA. Association for Computational Linguistics. find
  23. Kevin Pei, Ishan Jindal, Kevin Chen-Chuan Chang, ChengXiang Zhai, and Yunyao Li. 2023. When to use what: An in-depth comparative empirical analysis of OpenIE systems for downstream applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  24. Victor Pereira and Vládia Pinheiro. 2015. Report-um sistema de extração de informações aberta para língua portuguesa. In Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL), pages 191–200. SBC. find
  25. Alan Rios, Bruno Cabral, Daniela Claro, Rerisson Cavalcante, and Marlo Souza. 2024. Transalign: an automated corpus generation through cross-linguistic data alignment for open information extraction. In Proceedings of the 16th International Conference on Computational Processing of Portuguese-Vol. 1… find
  26. Youngbin Ro, Yukyung Lee, and Pilsung Kang. 2020. Multi^2OIE: Multilingual open information extraction based on multi-head attention with BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1107– 1117, Online. Association for Computational Linguistics. find
  27. Cleiton Fernando Lima Sena and Daniela Barreiro Claro. 2019. Inferportoie: A portuguese open information extraction system with inferences. Natural Language Engineering, 25(2):287–306. find
  28. Cleiton Fernando Lima Sena and Daniela Barreiro Claro. 2020. Pragmaticoie: a pragmatic open information extraction for portuguese language. Knowledge and Information Systems, 62(9):3811–3836. find
  29. Gabriel Silva, Mário Rodrigues, António Teixeira, and Marlene Amorim. 2024. Advancing open information extraction for portuguese by leveraging graph structures and large language models. In Proc. find
  30. IberSPEECH 2024, pages 61–65. find
  31. Marlo Souza, Bruno Cabral, Daniela Claro, and Lais Salvador. 2025. Challenges in expanding portuguese resources: A view from open information extraction. find
  32. Gabriel Stanovsky and Ido Dagan. 2016. Creating a large benchmark for open information extraction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2300–2305. find
  33. Michael Vasilkovsky, Anton Alekseev, Valentin Malykh, Ilya Shenbin, Elena Tutubalina, Dmitriy Sa-likhov, Mikhail Stepnov, Andrey Chertok, and Sergey Nikolenko. 2022. Detie: Multilingual open information extraction inspired by object detection. In Proceedings of the AAAI Conference on Artificial Inte… find
  34. Chenguang Wang, Xiao Liu, and Dawn Song. 2022. IELM: An open information extraction benchmark for pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8417– 8437, Abu Dhabi, United Arab Emirates. Association for Computational L… find
  35. Alexander Yates, Michele Banko, Matthew Broadhead, Michael Cafarella, Oren Etzioni, and Stephen Soderland. 2007. TextRunner: Open information extraction on the web. In Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational… find
  36. Junlang Zhan and Hai Zhao. 2020. Span model for open information extraction on accurate corpus. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9523–9530. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/silva-2026-benchmarking-portuguese-open-information-extraction/