Elamur Research⌕
☰
home / papers / 2026 / NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus

NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus

Paper · 2026 · Enzo S. N. Silva, Pablo B. Costa, Raphael C. Vlasman et al.

Summary

Portuguese encoder models trail English ones in two ways: the scale of curated monolingual pre-training data and the adoption of current encoder designs. The authors, from Itaú Unibanco and ICTi, address both. They build Aurora-PT, a Brazilian Portuguese corpus drawn from web sources and existing multilingual collections. It was cleaned, deduplicated and filtered for toxic and low-quality text, and it reaches 331 billion GPT-2 tokens. The authors describe it as the largest openly available monolingual Portuguese corpus. On this data they pre-train NorBERTo from scratch in base and large sizes, using the ModernBERT architecture with RoPE, long-context attention and efficient implementations. They compare it with BERTimbau, Albertina PT-BR, mmBERT and others on PLUE and ASSIN 2. NorBERTo-large wins every PLUE task tested and has the top ASSIN 2 entailment F1. Older large models still do better on semantic similarity. The study shows that large monolingual data combined with a modern architecture can produce strong encoders that are cheap to serve. Such "right-sized" models can complement LLMs, for example as a retrieval backbone in RAG.

Key points

  • Aurora-PT corpus: about 873 GB of text and roughly 275 million documents. It counts 331B tokens with the GPT-2 tokenizer and about 226B with the tokenizer used for training. That is far larger than brWaC (2.68B tokens), Aroeira (15B) and GigaVerbo (780 GB).
  • PLUE results for NorBERTo-large: MRPC 0.9191 versus 0.8873 for BERTimbau-large, RTE 0.7689 and WNLI 0.5774. All three are the best among the encoders compared.
  • ASSIN 2 results: NorBERTo-large reaches about 0.904 entailment F1, the highest among the evaluated encoders. On similarity (Pearson), BERTimbau-large and Albertina-900M remain ahead.
  • Model size: NorBERTo-base has about 150M parameters and beats the 340M-parameter BERTimbau-large on several tasks, such as MRPC (0.8926 vs 0.8873).
  • Limitations and future work: the authors note that pre-training used far less data than the original ModernBERT, which was trained on trillions of tokens. Planned work includes 1–2B-parameter variants, contrastive training to improve similarity, and evaluations on QA, cross-lingual and long-document tasks. Both the model and the corpus are listed on Hugging Face.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Enzo S. N. Silva, Pablo B. Costa, Raphael C. Vlasman, Rosimeire P. Costa, Henrique L. P. Silva, Lucas F. A. O. Pellicer, Guilherme Rinaldo, Renato A. Almeida, Darian S. R. Rabbani, Cinthya O. Oestreich and Vinicius F. Caridá. 2026. NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{silva2026norberto,
  title = {{NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus}},
  author = {Enzo S. N. Silva, Pablo B. Costa, Raphael C. Vlasman, Rosimeire P. Costa, Henrique L. P. Silva, Lucas F. A. O. Pellicer, Guilherme Rinaldo, Renato A. Almeida, Darian S. R. Rabbani, Cinthya O. Oestreich and Vinicius F. Caridá},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.18/}
}

References

42 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. Preprint, arXiv:1907.10902.
  2. Nicholas Kluge Corrêa, Aniket Sen, Sophia Falk, and Shiza Fatimah. 2024. Tucano: Advancing Neural Text Generation for Portuguese. Preprint, arXiv:2411.07854. — [arXiv:2411.07854](https://arxiv.org/abs/2411.07854) find
  3. Maria Clara Ramos Morales Crespo, Maria Lina de Souza Jeannine Rocha, Mariana Lourenço Sturzeneker, Felipe Ribas Serras, Guilherme Lamar-tine de Mello, Aline Silva Costa, Mayara Feliciano Palma, Renata Morais Mesquita, Raquel de Paula Guets, Mariana Marques da Silva, Marcelo Finger, Maria Clara Paix… — [arXiv:2303.16098](https://arxiv.org/abs/2303.16098) find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT.
  5. Paulo Finardi, José D. Viegas, and et al. 2021. Bertaú: Itaú bert for digital customer service. arXiv preprint arXiv:2101.12015. — [arXiv:2101.12015](https://arxiv.org/abs/2101.12015) find
  6. Gabriel Lino Garcia, Pedro Henrique Paiola, Danilo Samuel Jodas, Luis Afonso Sugi, and João Paulo Papa. 2024. Text summarization and temporal learning models applied to Portuguese fake news detection in a novel Brazilian corpus dataset. In Proceedings of the 16th International Conference on Computat… find
  7. Juliana Resplande S. Gomes. 2020. Plue: Portuguese language understanding evaluation. https://github.com/ju-resplande/PLUE. ↗
  8. Henrico Brum and Maria das Graças Volpe Nunes. 2018. Building a sentiment corpus of tweets in brazilian portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  9. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  10. Marc Brysbaert and Boris New. 2009. Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english. Behavior research methods, 41(4):977–990. find
  11. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
  12. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.
  13. Amir Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. Glotlid: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, page 6155–6218. Association for Computational Linguistics.
  14. Aida Kostikova, Zhipin Wang, and et al. 2025. Lllms: A data-driven survey of evolving research on limitations of large language models. arXiv preprint arXiv:2505.19240. — [arXiv:2505.19240](https://arxiv.org/abs/2505.19240) find
  15. Batia Laufer and Paul Nation. 1995. Vocabulary size and use: Lexical richness in l2 written production. Applied linguistics, 16(3):307–322. find
  16. Thiago Lira, Flávio Cação, Cinthia Souza, João Valentini, Edson Bollis, Otavio Oliveira, Renato Almeida, Marcio Magalhães, Katia Poloni, Andre Oliveira, and Lucas Pellicer. 2025. Aroeira: A curated corpus for the portuguese language with a large number of tokens. In A. Paes and F. A. N. Verri, edito… find
  17. BRACIS 2024, volume 15412 of Lecture Notes in Computer Science. Springer, Cham. find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  19. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. Preprint, arXiv:2509.06888.
  20. Philip M. McCarthy and Scott Jarvis. 2010. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42:381–392. find
  21. Guilherme Mello, Marcelo Finger, and et al. 2024. Pelle: Encoder-based language models for brazilian portuguese based on open data. arXiv preprint arXiv:2402.19204. — [arXiv:2402.19204](https://arxiv.org/abs/2402.19204) find
  22. Ian SP Nation and ISP Nation. 2001. Learning vocabulary in another language, volume 10. Cambridge university press Cambridge. find
  23. Felipe Oliveira, Victoria Reis, and Nelson Ebecken. 2023. Tupy-e: detecting hate speech in brazilian portuguese social media with a novel dataset and comprehensive analysis of models. arXiv preprint arXiv:2312.17704. — [arXiv:2312.17704](https://arxiv.org/abs/2312.17704) find
  24. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. Preprint, arXiv:2406.17557.
  25. Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all – adapting pre-training data processing to every language. Preprint, arXiv:2506.2…
  26. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  27. Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S. Yu. 2025. Large language models meet nlp: A survey. Preprint, arXiv:2405.12819. — [arXiv:2405.12819](https://arxiv.org/abs/2405.12819) find
  28. Livy Real, Erick Fonseca, and Hugo Gonçalo Oliveira. 2020. The assin 2 shared task: a quick overview. In International Conference on Computational Processing of the Portuguese Language, pages 406–412. Springer. find
  29. Brian Richards. 1987. Type/token ratios: what do they really tell us? Journal of Child Language, 14(2):201–209. find
  30. João Rodrigues, Luís Gomes, and et al. 2023. Advancing neural encoding of portuguese with transformer albertina pt-*. arXiv preprint arXiv:2305.06721. find
  31. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  32. Rodrigo Santos, João Rodrigues, Luís Gomes, João Silva, António Branco, Henrique Lopes Cardoso, Tomás Freitas Osório, and Bernardo Leite. 2024. Fostering the ecosystem of open neural encoders for portuguese with albertina pt* family. In Proceedings of the 3rd Annual Meeting of the Special Interest G… find
  33. Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2008. Introduction to information retrieval, volume 39. Cambridge University Press Cambridge. find
  34. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilingual. arXiv preprint.
  35. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: Pretrained bert models for brazilian portuguese. In Brazilian Conference on Intelligent Systems, pages 403–417. find
  36. Mildred C. Templin. 1957. Certain language skills in children: Their development and interrelationships. University of Minnesota Press, Minneapolis, MN. find
  37. Jean Ure. 1971. Lexical density and register differentiation. Applications of linguistics, 23(7):443–452. find
  38. Francielle Vargas, Isabelle Carvalho, Fabiana Rodrigues de Góes, Thiago Pardo, and Fabrício Benevenuto. 2022. Hatebr: A large expert annotated corpus of brazilian instagram comments for offensive language and hate speech detection. In Proceedings of the Thirteenth Language Resources and Evaluation C… find
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  40. Jorge A Wagner Filho, Rodrigo Wilkens, Marco Idiart, and Aline Villavicencio. 2018. The brwac corpus: A new open resource for brazilian portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  41. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.
  42. Benjamin Warner, Antoine Chaffin, and et al. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13663.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/silva-2026-norberto-modernbert-model-trained-portuguese/