Elamur Research⌕
☰
home / papers / 2026 / Twenty Years of HAREM: A Reproducible Audit and Reassessment of Portuguese Named Entity Recognition

Twenty Years of HAREM: A Reproducible Audit and Reassessment of Portuguese Named Entity Recognition

Paper · 2026 · Rafael O. Nunes, André Spritzer, Carla M. D. S. Freitas et al.

Summary

For about two decades, Portuguese NER has been benchmarked with one fixed protocol: train on First HAREM, test on Mini-HAREM. No one had checked whether these two sets share data. The authors audited the split and found 153 sentences that appear in both. They also found a few duplicates inside each corpus. They built a decontaminated version of the benchmark and retrained 13 NER systems on both the original and cleaned data. The systems ranged from a CRF baseline and static-embedding BiLSTM-CRF models to Transformers such as BERTimbau, Albertina, XLM-R, mBERT and ModernBERT. Evaluation used stratified five-fold cross-validation on First HAREM plus the traditional holdout on Mini-HAREM. Wilcoxon, Friedman and Nemenyi tests assessed the results. Removing the overlap changed scores significantly for most models, and usually improved them. Macro-F1, which reflects rare entity classes, showed the largest gains. Large Transformers still clearly outperformed the baselines. The work matters because even a small leak can distort comparisons and state-of-the-art claims. The cleaned benchmark and scripts are released for future evaluations.

Key points

  • Overlap found: 153 sentences are shared between the sets. Of these, 142 come from the First HAREM training partition and 11 from its development partition. Internal duplicates number 15 in First HAREM and 25 in Mini-HAREM, with no conflicting BIO tags.
  • Effect on entity counts: Cleaning reduced entity counts by roughly 1–3%. First HAREM went from 4,950 to 4,830 entities and Mini-HAREM from 3,481 to 3,437. The biggest drops were in LOC, PER and ORG.
  • Significance: Decontamination produced statistically significant changes (p < 0.05) for 9 of the 13 models.
  • Holdout micro-F1 gains: Albertina (900M BrWaC) rose from 76.55 to 79.28, and BERTimbau (large) from 74.73 to 78.74. BERTimbau (large) macro-F1 jumped from 64.11 to 74.00. A few models, such as XLM-R (base) and NILC FastText, changed slightly negatively.
  • Released resources: The decontaminated corpus, the overlap list and the cleaning script are available at github.com/RafaelOleques/harem_decontaminated. Models were built with FLAIR/PyTorch, and the CRF baseline used sklearn_crfsuite.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.Named entity recognition · 214 works

Cite

Cite as Rafael O. Nunes, André Spritzer, Carla M. D. S. Freitas and Dennis G. Balreira. 2026. Twenty Years of HAREM: A Reproducible Audit and Reassessment of Portuguese Named Entity Recognition. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nunes2026twenty,
  title = {{Twenty Years of HAREM: A Reproducible Audit and Reassessment of Portuguese Named Entity Recognition}},
  author = {Rafael O. Nunes, André Spritzer, Carla M. D. S. Freitas and Dennis G. Balreira},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.35/}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hidelberg O Albuquerque, Ellen Souza, Carlos Gomes, Matheus Henrique de C Pinto, Ricardo PS Filho, Rosimeire Costa, Vinícius Teixeira de M Lopes, Nádia FF da Silva, André CPLF de Carvalho, and Adriano LI Oliveira. 2023. Named entity recognition: a survey for the portuguese language. Procesamiento de… find
  2. Aymé Arango, Jorge Pérez, and Barbara Poblete. 2022. Hate speech detection is not as easy as you may think: A closer look at model validation (extended version). Information Systems, 105:101584. find
  3. Diedre Carmo, Marcos Piau, Israel Campiotti, Rodrigo Nogueira, and Roberto Lotufo. 2020. Ptt5: Pretraining and validating the t5 model on brazilian portuguese data. arXiv preprint arXiv:2008.09144. — [arXiv:2008.09144](https://arxiv.org/abs/2008.09144) find
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo…
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Cicero dos Santos and Victor Guimarães. 2015. Boosting named entity recognition with neural character embeddings. In Proceedings of the Fifth Named Entity Workshop, pages 25–33.
  7. Milton Friedman. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675–701. find
  8. Diana Santos, Nuno Cardoso, Nuno Seco, and Rui Vilela. 2007. Breve introdução ao harem. HAREM, a primeira avaliaçao conjunta de sistemas de reconhecimento de entidades mencionadas para português: documentaçao e actas do encontro, Linguateca. find
  9. Diana Santos, Nuno Seco, Nuno Cardoso, and Rui Vilela. 2006. Harem: An advanced ner evaluation contest for portuguese. In quot; In Nicoletta Calzolari; Khalid Choukri; Aldo Gangemi; Bente Maegaard; Joseph Mariani; Jan Odjik; Daniel Tapias (ed) Proceedings of the 5 th International Conference on Lang… find
  10. Joaquim Santos, Bernardo Consoli, Cicero dos Santos, Juliano Terra, Sandra Collonini, and Renata Vieira. 2019. Assessing the impact of contextual embeddings for portuguese named entity recognition. In 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), pages 437–442. IEEE. find
  11. Konstantinos Sechidis, Grigorios Tsoumakas, and Ioannis Vlahavas. 2011. On the stratification of multi-label data. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22, pages 145–158. Springer. find
  12. Samuel Sanford Shapiro and Martin B Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika, 52(3-4):591–611. find
  13. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: pretrained bert models for brazilian portuguese. In Brazilian conference on intelligent systems, pages 403–417. Springer. find
  14. Nathan Hartmann, Erick Fonseca, Christopher Shulby, Marcos Treviso, Jessica Rodrigues, and Sandra Aluisio. 2017. Portuguese word embeddings: Evaluating on word analogies and natural language tasks. arXiv preprint arXiv:1708.06025. find
  15. Guilherme Tapajós, Tiago de Melo, Elloá B Guedes, and Fábio Santos. 2025. Reconhecimento de entidades nomeadas em português: Comparação de modelos pré-treinados com fine-tuning. Revista Eletrônica de Iniciação Científica em Computação, 23:111–117. find
  16. Peter Bjorn Nemenyi. 1963. Distribution-free multiple comparisons. Princeton University. find
  17. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, and 1 others. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning an…
  18. Pedro Vitor Quinta de Castro, Nádia Félix Felipe da Silva, and Anderson da Silva Soares. 2018. Portuguese named entity recognition using lstm-crf. In International Conference on Computational Processing of the Portuguese Language, pages 83–92. Springer. find
  19. João Rodrigues, Luís Gomes, João Silva, António Branco, Rodrigo Santos, Henrique Lopes Cardoso, and Tomás Osório. 2023. Advancing neural encoding of portuguese with transformer albertina pt. In EPIA Conference on Artificial Intelligence, pages 441–453. Springer. find
  20. F Wilcoxon. 1945. Individual comparisons by ranking methods. biom. bull., 1, 80–83. find
  21. Susanna Rücker and Alan Akbik. 2023. Cleanconll: A nearly noise-free named entity recognition dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8628–8645. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nunes-2026-twenty-years-harem-reproducible-audit/