Elamur Research⌕
☰
home / papers / 2026 / Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs

Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs

Paper · 2026 · Mariela M. Nina, Caio Veloso Costa, Lilian Berton et al.

Summary

The paper asks whether parameter-efficient adaptation and low-bit quantization, mostly validated on English, also work for extractive question answering in Brazilian Portuguese, where compute budgets are often limited. The authors fine-tune BERTimbau (Base, 110M; Large, 335M) on SQuAD-BR, a Portuguese translation of SQuAD v1. They test 40 configurations that cross LoRA, DoRA, QLoRA and QDoRA with model sizes and learning rates, using full fine-tuning as the reference. They score accuracy (F1, Exact Match), training time and peak GPU memory.

LoRA on the Large model gets close to the full fine-tuning baseline while cutting training time by roughly three quarters. PEFT needs much higher learning rates than usual, and 4-bit quantization hurts the larger model about half as much as the smaller one. A smaller exploratory test of the generative models Tucano and Sabiá shows they can reach similar F1 with LoRA but use far more resources.

The work gives practical, Green-AI-oriented guidance: compact encoders adapted with PEFT are a cheap, strong option for Portuguese extractive QA.

Key points

  • LoRA on BERTimbau-Large reached F1 = 81.32 against 84.86 for full fine-tuning, which is 95.8% of the baseline, with 73.5% less training time. DoRA matched LoRA but added about 28% training time, so it brought no practical benefit.
  • Learning rate: raising it to 2e-4 improved PEFT runs by up to +19.71 F1. At the same rate, full fine-tuning of the Large model collapsed to F1 = 3.02. The authors read this as low-rank updates acting like implicit regularization.
  • QLoRA and model size: compared with the baseline, QLoRA lost 9.56 F1 on Base but only 4.83 on Large, where it kept 94.3% of baseline performance. Peak memory fell by 86.9% on the Base model, to 1,897 MB, and by 81.9% on the Large model, to 3,281 MB, putting both within reach of consumer GPUs.
  • Generative models: Sabiá-7B with LoRA scored F1 = 78.10, roughly level with BERTimbau-Base at 78.01. It needed 4.2× the GPU memory (15,642 vs 3,687 MB) and about 3× the training time. BERTimbau-Large with LoRA outperformed every generative model tested, including Tucano.
  • Recommendation: prefer BERTimbau-Large with LoRA or QLoRA at high learning rates. Treat full fine-tuning only as a baseline, and avoid aggressive quantization of Base models unless memory is very tight.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Universidade Federal de São Paulo.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksQuestion answering · 441 works

Cite

Cite as Mariela M. Nina, Caio Veloso Costa, Lilian Berton and Didier A. Vega-Oliveros. 2026. Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nina2026efficient,
  title = {{Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs}},
  author = {Mariela M. Nina, Caio Veloso Costa, Lilian Berton and Didier A. Vega-Oliveros},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.91/}
}

References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. Preprint, arXiv:2012.13255.
  2. Gian Franco Condori-Luna, Didier Vega-Oliveros, and Marcelo da Silva Reis. 2026. Reducing dependence on labeled data: A self-supervised fine-tuning approach for low-resource language models. In Intelligent Systems, pages 395–409, Cham. Springer Nature Switzerland. find
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  4. Nicholas Kluge Corrêa, Aniket Sen, Sophia Falk, and Shiza Fatimah. 2025. Tucano: Advancing neural text generation for portuguese. Patterns, 6(11):101325. find
  5. E. da Silva, J. Laterza, and T. Faleiros. 2022. New state-of-the-art for question answering on portuguese squad v1.1. In Anais do X Symposium on Knowledge Discovery, Mining and Learning, pages 98–105, Porto Alegre, RS, Brasil. SBC. find
  6. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit optimizers via block-wise quantization. Preprint, arXiv:2110.02861.
  7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Preprint, arXiv:2305.14314.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.
  9. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. Gptq: Accurate post-training quantization for generative pre-trained transformers. Preprint, arXiv:2210.17323. find
  10. Christian Freitas, Didier A. Vega-Oliveros, and Lilian Berton. 2025. Enhancing industrial data access with text-to-sql using portuguese llms and langgraph. In 2025 12th International Conference on Soft Computing & Machine Intelligence (ISCMI), pages 278–282. find
  11. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning. Preprint, arXiv:2110.04366.
  12. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  13. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. Preprint, arXiv:1902.00751. find
  14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  15. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.
  16. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980.
  17. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. Preprint, arXiv:2104.08691.
  18. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. Preprint, arXiv:2101.00190. find
  19. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. Preprint, arXiv:2402.09353. find
  20. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. find
  21. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft. ↗
  22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. Preprint, arXiv:1606.05250.
  23. Tim Salimans and Diederik P. Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Preprint, arXiv:1602.07868. find
  24. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: Pretrained bert models for brazilian portuguese. In Intelligent Systems, pages 403– 417, Cham. Springer International Publishing. find
  25. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in nlp. Preprint, arXiv:1906.02243.
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762.
  27. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2020. Superglue: A stickier benchmark for general-purpose language understanding systems. Preprint, arXiv:1905.00537.
  28. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. Preprint, arXiv:1804.07461.
  29. Jing Yang, Didier Vega-Oliveros, Taís Seibt, and Anderson Rocha. 2022. Explainable fact-checking through question answering. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8952–8956. find
  30. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations (ICLR).
  31. Ricardo Mazza Zago and Luciane Agnoletti dos Santos Pedotti. 2024. Bertugues: A novel bert transformer model pre-trained for brazilian portuguese. Semina: Ciências Exatas e Tecnológicas, 45:e50630. find
  32. Mariela M. Nina and Didier A. Vega-Oliveros. 2025. Maximizing model adaptation for low-resource languages: A progressive unfreezing strategy for spanish-aymara translation. In 2025 12th International Conference on Soft Computing & Machine Intelligence (ISCMI), pages 259–263. find
  33. Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023. Sabiá: Portuguese Large Language Models, page 226–240. Springer Nature Switzerland. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nina-2026-efficient-fine-tuning-methods-portuguese/