Elamur Research⌕
☰
home / papers / 2026 / Global vs. Local Sentence Embeddings for Brazilian Portuguese: Revisiting Monolingual Models in the Age of Foundation Models

Global vs. Local Sentence Embeddings for Brazilian Portuguese: Revisiting Monolingual Models in the Age of Foundation Models

Paper · 2026 · Matheus Peixoto, Guilherme Silva, Giacomo Figueredo et al.

Summary

The paper asks whether large multilingual ("global") foundation models, lightly adapted, can replace Portuguese-specialized ("local") models as sources of sentence embeddings for Brazilian Portuguese (PT-BR). The authors benchmark Portuguese models (BERTimbau, Albertina, Serafim, Tucano) against multilingual ones (Gemma, Qwen, a multilingual MiniLM sentence-transformer) on classification, clustering, NLI and STS. They evaluate each model in two settings: frozen embeddings (linear probing, plus intrinsic STS) and lightweight supervised fine-tuning.

With frozen representations, both groups perform at a comparable level. After adaptation, the local encoders pull ahead. Fine-tuning raises classification accuracy for every model, but the global models often lose STS correlation compared with their frozen baseline. The authors call this adaptation-induced representation drift.

A tokenization analysis adds nuance. Multilingual tokenizers segment PT-BR words with unexpected morphological sense and handle English loanwords well, whereas BERTimbau's vocabulary breaks such loanwords apart. For practitioners, the work shows that choosing a PT-BR embedding model depends on the task and means balancing vocabulary coverage against adaptability.

Key points

  • Evaluation data visible in the paper include ASSIN2 (STS) and the classification sets Hate-BR, TuPy and AG NEWS PT.
  • The models cover both encoder-only and decoder-only architectures:
    • Albertina PT-BR encoders (100M and 900M)
    • BERTimbau base and large
    • Serafim 335M
    • Tucano
    • Gemma
    • Qwen
    • paraphrase-multilingual-MiniLM-L12-v2
  • Sentence vectors come from attention-masked mean pooling over last-layer hidden states, including for decoder-only models (no text generation). Fine-tuning uses a fixed budget of 10 epochs for all models.
  • In the tokenization analysis, BERTimbau splits "fake news" into fragments such as 'fa', '##ke', 'ne', '##ws', while several multilingual tokenizers keep both words whole. The authors conclude that no tokenizer is universally best.
  • The authors acknowledge several limitations:
    • Embedding-specialized multilingual models (E5/BGE-style) were not included.
    • Recent PT decoders (Cabrita, GlórIA, Gervásio) were not benchmarked.
    • Some STS effects may come from architecture or embedding-extraction choices rather than multilinguality itself.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Matheus Peixoto, Guilherme Silva, Giacomo Figueredo, Pedro Silva and Eduardo J. S. Luz. 2026. Global vs. Local Sentence Embeddings for Brazilian Portuguese: Revisiting Monolingual Models in the Age of Foundation Models. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{peixoto2026global,
  title = {{Global vs. Local Sentence Embeddings for Brazilian Portuguese: Revisiting Monolingual Models in the Age of Foundation Models}},
  author = {Matheus Peixoto, Guilherme Silva, Giacomo Figueredo, Pedro Silva and Eduardo J. S. Luz},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.52/}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Andrey Kutuzov, Barry Haddow, and Kenneth Heafield. 2024. Monolingual or multilingual instruction tuning: Which makes a better alpaca. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1347–1356, St. Julian’s, Malta. Associa… find
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th annual meeting of the association fo…
  3. Nicholas Kluge Corrêa, Aniket Sen, Sophia Falk, and Shiza Fatimah. 2025. Tucano: Advancing Neural Text Generation for Portuguese. Patterns. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  5. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891.
  6. Giacomo Figueredo, Gabriel Gomes Botelho Freitas, Laura Scaramussa Azevedo, and Lucas Alexandre Damasceno. 2024. Considerações sobre a organização do texto e da instanciação sob a perspectiva sistêmico-funcional. DELTA: Documentação de Estudos em Lingüística Teórica e Aplicada, 40(1):202440159824. find
  7. E Fonseca, L Santos, Marcelo Criscuolo, and S Aluisio. 2016. Assin: Avaliacao de similaridade semantica e inferencia textual. In Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal, pages 13–15. find
  8. Gabriel Lino Garcia, Pedro Henrique Paiola, Luis Henrique Morelli, Giovani Candido, Arnaldo Cândido Júnior, Danilo Samuel Jodas, Luis C. S. Afonso, Ivan Rizzo Guilherme, Bruno Elias Penteado, and João Paulo Papa. 2024. Introducing bode: A fine-tuned large language model for portuguese prompt-based t… — [arXiv:2401.02909](https://arxiv.org/abs/2401.02909) find
  9. Luís Gomes, António Branco, João Silva, João Rodrigues, and Rodrigo Santos. 2024. Open sentence embeddings for portuguese with the serafim pt* encoders family. Preprint, arXiv:2407.19527. — [arXiv:2407.19527](https://arxiv.org/abs/2407.19527) find
  10. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR).
  11. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, and 1 others. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(1… find
  12. Celio Larcher, Marcos Piau, Paulo Finardi, Pedro Gengo, Piero Esposito, and Vinicius Caridá. 2023. Cabrita: closing the gap for foreign languages. Preprint, arXiv:2308.11878. — [arXiv:2308.11878](https://arxiv.org/abs/2308.11878) find
  13. Ricardo Lopes, Joao Magalhaes, and David Semedo. 2024. GlórIA: A generative and open large language model for Portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese, pages 441–453, Santiago de Compostela, Galicia/Spain. Association for Computational… find
  14. Maicon Domingues. 2023. 08fdd2b). canarim-7b (revision Maritaca-AI. 2023. ag_news_pt. find
  15. Christian M.I.M. Matthiessen. 2015. Register in the round: registerial cartography. Functional Linguistics, 2(1):1–48. find
  16. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.04884. find
  17. Felipe Oliveira, Victoria Reis, and Nelson Ebecken. 2023. Tupy-e: detecting hate speech in brazilian portuguese social media with a novel dataset and comprehensive analysis of models. arXiv preprint arXiv:2312.17704. — [arXiv:2312.17704](https://arxiv.org/abs/2312.17704) find
  18. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. In Advances in Neural Information Processing Systems, volume 36, pages 36963–36990. Curran Associates, Inc.
  19. Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023. Sabiá: Portuguese Large Language Models, page 226–240. Springer Nature Switzerland. find
  20. Livy Real, Erick Fonseca, and Hugo Goncalo Oliveira. 2020. The assin 2 shared task: a quick overview. In International Conference on Computational Processing of the Portuguese Language, pages 406–412. Springer. find
  21. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  22. Thirteenth Language Resources and Evaluation Conference, pages 7174–7183. find
  23. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076.
  24. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP (RepL4NLP). Association for Computational Linguistics.
  25. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024. Qwen2 technical report. arXiv preprint arXi… find
  26. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
  27. Rodrigo Santos, João Rodrigues, Luís Gomes, João Silva, António Branco, Henrique Lopes Cardoso, Tomás Freitas Osório, and Bernardo Leite. 2024a. Fostering the ecosystem of open neural encoders for portuguese with albertina pt* family. Preprint, arXiv:2403.01897. find
  28. Rodrigo Santos, João Silva, Luís Gomes, João Rodrigues, and António Branco. 2024b. Advancing generative ai for portuguese with open decoder gervásio pt-*. Preprint, arXiv:2402.18766. find
  29. Fabricio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: pretrained bert models for brazilian portuguese. In Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I, pages 403–417. Springer. find
  30. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. find
  31. Francielle Vargas, Isabelle Carvalho, Fabiana Rodrigues de Góes, Thiago Pardo, and Fabrício Benevenuto. 2022. Hatebr: A large expert annotated corpus of brazilian instagram comments for offensive language and hate speech detection. In Proceedings of the find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peixoto-2026-global-vs-local-sentence-embeddings/