Elamur Research⌕
☰
home / papers / 2026 / JabuticaBERT: Modern Portuguese Encoders from Scratch with RTD and Long-Context Training

JabuticaBERT: Modern Portuguese Encoders from Scratch with RTD and Long-Context Training

Paper · 2026 · Thiago Porto, Gabriel Gomes, Alexandre Bender et al.

Summary

Brazilian Portuguese lacks modern monolingual encoders. Widely used models such as BERTimbau rely on the original BERT design, with a 512-token limit, and many models use multilingual vocabularies that split Portuguese words into too many subwords. The authors pre-train two families of encoders from scratch on the Jabuticaba corpus, each with its own Portuguese tokenizer:

  • JabuticaBERT uses the DeBERTa architecture with Replaced Token Detection (RTD).
  • modernJabuticaBERT uses ModernBERT with MLM, FlashAttention and RoPE, trained on contexts of up to 8,192 tokens.

The models were fine-tuned under the same hyperparameter search and tested on ASSIN 2 (RTE and STS), LeNER and ToldBR. JabuticaBERT Large performs about as well as Albertina 900M with roughly half its parameters. Its tokenizer also produces the fewest tokens per word overall. The ModernBERT variants score lower but steadily, and their long-context ability was not tested on downstream tasks.

The work matters because it shows that a monolingual tokenizer and a sample-efficient objective can stand in for sheer model size. Because objective and architecture are bundled together, the RTD vs. MLM effect cannot be cleanly separated.

Key points

  • Released models: JabuticaBERT-XSmall, JabuticaBERT-Large, and modernJabuticaBERT Base/Large in 1k and 8k context versions, along with their tokenizers.
  • JabuticaBERT Large vs. Albertina 900M:
    • ASSIN2 RTE F1: 0.920 vs. 0.921
    • LeNER F1: 0.915 vs. 0.914
    • ASSIN2 STS Pearson: 0.861 vs. 0.865
  • ToldBR: JabuticaBERT Large reaches 0.757 accuracy, below BERTimbau Large's 0.768.
  • Tokenizer fertility (tokens per word, overall):
    • JabuticaBERT: 1.495
    • BERTimbau: 1.774
    • modernJabuticaBERT: 1.814
    • Albertina: 2.060
    • English BERT: 2.158
    • BERTimbau is best only on the Wikipedia domain, which its tokenizer was trained on.
  • Stated limitations:
    • No cross-over ablations, such as ModernBERT+RTD or DeBERTa+MLM.
    • No long-context downstream benchmark.
    • No efficiency measurements comparing 1k and 8k contexts.
    • No decontamination check against the test sets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Thiago Porto, Gabriel Gomes, Alexandre Bender, Ulisses Corrêa, Larissa Freitas, William Cruz and Marcellus Amadeus. 2026. JabuticaBERT: Modern Portuguese Encoders from Scratch with RTD and Long-Context Training. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{porto2026jabuticabert,
  title = {{JabuticaBERT: Modern Portuguese Encoders from Scratch with RTD and Long-Context Training}},
  author = {Thiago Porto, Gabriel Gomes, Alexandre Bender, Ulisses Corrêa, Larissa Freitas, William Cruz and Marcellus Amadeus},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.93/}
}

References

23 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marcellus Amadeus, William Alberto Cruz Castañeda, José Roberto Homeli da Silva, and Rodrigo Scotti. 2024. Jabuticaba: The largest commercial corpus for llms in portuguese. Preprint, arXiv:2404.13680. — [arXiv:2404.13680](https://arxiv.org/abs/2404.13680) find
  2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. Preprint, arXiv:2004.05150.
  3. Michael Brenndoerfer. 2025. Attention complexity: Quadratic scaling, memory limits & efficient alternatives. In Language AI Handbook, chapter 78. Michael Brenndoerfer. Chapter 78 of 380 in the Language AI Handbook. Published May 26 2025; updated May 29 2025. find
  4. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR). find
  5. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS). find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  7. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543.
  8. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. Preprint, arXiv:2006.03654.
  9. Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021. Ammus : A survey of transformer-based pretrained models in natural language processing. Preprint, arXiv:2108.05542. — [arXiv:2108.05542](https://arxiv.org/abs/2108.05542) find
  10. João Augusto Leite, Diego Silva, Kalina Bontcheva, and Carolina Scarton. 2020. Toxic language detection in social media for Brazilian Portuguese: New dataset and multilingual analysis. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics a… find
  11. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. Preprint, arXiv:2005.11401.
  12. Conglong Li, Minjia Zhang, and Yuxiong He. 2022. The stability-efficiency dilemma: investigating sequence length warmup for training gpt models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. find
  13. Pedro H. Luz de Araujo, Teófilo E. de Campos, Renato R. R. de Oliveira, Matheus Stauffer, Samuel Couto, and Paulo Bermejo. 2018. LeNER-Br: a dataset for named entity recognition in Brazilian legal text. In International Conference on the Computational Processing of Portuguese (PROPOR), Lecture Notes… find
  14. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203– 7219,…
  15. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? Preprint, arXiv:1906.01502.
  16. Livy Real, Erick Fonseca, and Hugo Goncalo Oliveira. 2020. The assin 2 shared task: a quick overview. In International Conference on Computational Processing of the Portuguese Language, pages 406–412. Springer. find
  17. João Rodrigues, Luís Gomes, João Silva, António Branco, Rodrigo Santos, Henrique Lopes Cardoso, and Tomás Osório. 2023. Advancing neural encoding of portuguese with transformer albertina pt-*. Preprint, arXiv:2305.06721. find
  18. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: Pretrained bert models for brazilian portuguese. In Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I, page 403–417, Berlin, Heidelberg. Springer-Verlag. find
  19. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024a. Roformer: Enhanced transformer with rotary position embedding. Neurocomput., 568(C).
  20. Jianlin Su, Yujie Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2024b. Rotary positional embedding: A method for relative position encoding in transformers. In Proceedings of the Conference. ArXiv preprint arXiv:2104.09864.
  21. Jiajia Wang, Jimmy X. Huang, Xinhui Tu, Junmei Wang, Angela J. Huang, Md Tahmid Rahman Laskar, and Amran Bhuiyan. 2024. Utilizing bert for information retrieval: Survey, applications, resources, and challenges. Preprint, arXiv:2403.00784. find
  22. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast,…
  23. Ricardo Zago and Luciane Agnoletti dos Santos Pedotti. 2024. Bertugues: A novel bert transformer model pre-trained for brazilian portuguesebertugues: Um modelo transformer bert inovador pré-treinado para o português brasileiro. Semina: Ciências Exatas e Tecnológicas, 45:e50630. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/porto-2026-jabuticabert-modern-portuguese-encoders-scratch/