Elamur Research⌕
☰
home / papers / 2026 / From Bones to Rocks: A Systematic Evaluation of Specialized Definition Generation for Portuguese

From Bones to Rocks: A Systematic Evaluation of Specialized Definition Generation for Portuguese

Summary

Definition generation (DG) means producing a natural-language definition for a given term. Most DG research targets English. The only earlier Portuguese benchmark, DORE, uses general dictionaries and does not cover technical vocabulary. The authors address this gap by evaluating Portuguese DG in two specialized domains, medicine and geology. Reference definitions come from the MeSH thesaurus (through DeCS/MeSH) and the SIGEP geological glossary. Real usage contexts come from two NER corpora, BETE and GeoCorpus-3. The authors linked glossary terms and their synonyms to corpus sentences, producing term–context–definition triples. They then prompted several open decoder-only LLMs, including one adapted to Portuguese, in zero-shot and few-shot settings, with and without a context sentence. Evaluation used 5-fold cross-validation and significance testing. Most models handled the task well, and the strongest ones scored close together. Their differences were frequently not statistically significant. This suggests that model size, multilingual coverage and Portuguese specialization do not decisively determine quality. The result is useful when choosing models for specialized NLP in lower-resource languages.

Key points

  • Gold-standard glossaries:
    • MeSH, 2018 version: 28,939 definitions and 121,318 terms.
    • SIGEP glossary, version of May 11, 2025: 1,514 definitions and 2,026 terms.
  • Context corpora:
    • GeoCorpus-3 covers Brazilian sedimentary basins: 5,272 sentences, 166,462 tokens, 16,694 unique tokens.
    • BETE covers diabetes texts: 439 sentences, 43,198 tokens, 2,844 unique tokens.
  • Term–sentence alignment: a reverse index maps synonyms to concept IDs. Matching uses lowercased, strict word-boundary regex. Stemming, lemmatization and embedding similarity were deliberately excluded to keep matches precise and reproducible.
  • Models: the smallest is GAIA (4B parameters), included as an open-source model with continued pretraining for Portuguese. The other models have roughly 7B parameters or more.
  • Prompting and code: the prompt template asks the model to act as a domain expert. Context and examples are optional slots, and the output must be a Portuguese definition only. Code is released at github.com/RafaelOleques/definition_generation_lrec2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4w8a83i8b3wy

@inproceedings{nunes2026bones,
  title = {{From Bones to Rocks: A Systematic Evaluation of Specialized Definition Generation for Portuguese}},
  author = {Nunes, Rafael Oleques and Balreira, Dennis Giovani and Carbonera, Joel Luís},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4w8a83i8b3wy},
  url = {https://doi.org/10.63317/4w8a83i8b3wy}
}

References

37 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2018. Health Sciences Descriptors: DeCS, 2018 edition. BIREME / PAHO / WHO, São Paulo. find
  2. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. find our card
  3. Hugo Abonizio, Thales Sales Almeida, Thiago Laitz, Roseval Malaquias Junior, Giovana Kerche Bonás, Rodrigo Nogueira, and Ramon Pires. 2024. Sabi\\’a-3 technical report. arXiv preprint arXiv:2410.12049. — [arXiv:2410.12049](https://arxiv.org/abs/2410.12049) find
  4. Thales Sales Almeida, Hugo Abonizio, Rodrigo Nogueira, and Ramon Pires. 2024. Sabi\\’a-2: A new generation of portuguese large language models. arXiv preprint arXiv:2403.09887. — [arXiv:2403.09887](https://arxiv.org/abs/2403.09887) find
  5. Mohannad AlMousa, Rachid Benlamri, and Richard Khoury. 2022. A novel word sense disambiguation approach using wordnet knowledge graph. Computer Speech & Language, 74:101337. find
  6. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. find our card
  7. C. G. CAMILO-JUNIOR, S. S. T. OLIVEIRA, L. A. PEREIRA, M. AMADEUS, D. FAZZIONI, A. M. A. NOVAIS, and S. A. A. JORDÃO. 2025. Gaia: An open language model for brazilian portuguese. [https://huggingface.co/CEIA-UFG/Gemma-3-Gaia-PT-BR-4b-it] (https://huggingface.co/CEIA-UFG/Gemma-3-Gaia-PT-BR-4b-it). find
  8. Ashish Chouhan and Michael Gertz. 2024. Lex-Drafter: Terminology drafting for legislative documents using retrieval augmented generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10448–10458, T… find
  9. Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan):1–30. find
  10. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383– 1392. find
  11. Anna Beatriz Dimas Furtado, Tharindu Ranasinghe, Frédéric Blain, and Ruslan Mitkov. 2024. Dore: A dataset for portuguese definition generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5315–532… find
  12. Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry Vetrov. 2018. Conditional generators of words definitions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 266– 271. find
  13. Gabriel Lino Garcia, Pedro Henrique Paiola, Luis Henrique Morelli, Giovani Candido, Arnaldo Cândido Júnior, Danilo Samuel Jodas, Luis CS Afonso, Ivan Rizzo Guilherme, Bruno Elias Penteado, and João Paulo Papa. 2024. Introducing bode: A fine-tuned large language model for portuguese prompt-based task… find
  14. Salvador Garcia and Francisco Herrera. 2008. An extension on" statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons. Journal of machine learning research, 9(12). find
  15. Mario Giulianelli, Iris Luden, Raquel Fernández, and Andrey Kutuzov. 2023. Interpretable word sense representations via definition generation: The case of semantic change analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag… find our card
  16. Diogo da Silva Magalhães Gomes, Fábio Corrêa Cordeiro, Bernardo Scapini Consoli, Nikolas Lacerda Santos, Viviane Pereira Moreira, Renata Vieira, Silvia Moraes, and Alexandre Gonçalves Evsukoff. 2021. Portuguese word embeddings for the oil and gas industry: Development and evaluation. Computers in In… find
  17. Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786–2791, Florence, Italy. Association for Computational Linguistics. find
  18. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  19. Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshinaga, Graham Neubig, Shoetsu Sato, Masashi Toyoda, and Masaru Kitsuregawa. 2019. Learning to describe unknown phrases with local and global contexts. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computati… find
  20. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  21. Jiahuan Li, Yu Bao, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2020. Explicit semantic decomposition for definition generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 708–717. find
  22. Alcides Lopes, Joel Carbonera, Fabricio Rodrigues, Luan Garcia, and Mara Abel. 2024. How to classify domain entities into top-level ontology concepts using large language models. Applied Ontology, (Preprint):1–29. find
  23. Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnini, Francesco Cecconi, et al. 2021. Ten years of babelnet: A survey. In IJ-CAI, pages 4559–4567. International Joint Conferences on Artificial Intelligence Organization. find
  24. Roberto Navigli and Paola Velardi. 2005. Structural semantic interconnections: a knowledge-based approach to word sense disambiguation. IEEE transactions on pattern analysis and machine intelligence, 27(7):1075–1086. find
  25. Ke Ni and William Yang Wang. 2017. Learning to explain non-standard english words and phrases. find
  26. Thanapon Noraset, Chen Liang, Larry Birnbaum, and Doug Downey. 2017. Definition modeling: Learning to define word embeddings in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31. find
  27. Rafael O Nunes, João E Soares, Henrique DP dos Santos, and Renata Vieira. 2018. Meshx-notes: web-system for clinical notes. In International Workshop on Artificial Intelligence in Health, pages 5–12. Springer. find
  28. Rafael O Nunes, Andre S Spritzer, Dennis G Balreira, Carla MDS Freitas, and Joel L Carbonera. 2024. An evaluation of large language models for geological named entity recognition. In 2024 IEEE 36th International Conference on Tools with Artificial Intelligence (ICTAI), pages 494– 501. IEEE. find
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. find our card
  30. Lucas Pavanelli, Yohan Bonescki Gumiel, Thiago Ferreira, Adriana Pagano, and Eduardo Laber. 2023. Bete: A brazilian portuguese dataset for named entity recognition and relation extraction in the diabetes healthcare domain. In Brazilian Conference on Intelligent Systems, pages 256– 267. Springer. find
  31. Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023. Sabiá: Portuguese large language models. In Brazilian Conference on Intelligent Systems, pages 226–240. Springer. find
  32. David Riaño, Mor Peleg, and Annette Ten Teije. 2019. Ten years of knowledge representation for health care (2009–2018): Topics, trends, and challenges. Artificial intelligence in medicine, 100:101713. find
  33. Hendrik Schuff, Lindsey Vanderlyn, Heike Adel, and Ngoc Thang Vu. 2023. How to do human evaluation: A brief introduction to user studies in nlp. Natural Language Engineering, 29(5):1199– 1222. find
  34. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… find our card
  36. Bowen Zhang and Harold Soh. 2024. Extract, define, canonicalize: An llm-based framework for knowledge graph construction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9820– 9836. find
  37. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nunes-2026-bones-rocks-systematic-evaluation-specialized/