Elamur Research⌕
☰
home / papers / 2026 / Prompt Engineering for Named Entity Extraction from Portuguese Legal Documents

Prompt Engineering for Named Entity Extraction from Portuguese Legal Documents

Paper · 2026 · Giovanni Maffeo, Catarina Silva and Hugo Gonçalo Oliveira

Summary

Supervised NER for Portuguese legal texts is limited because annotated corpora are scarce and expensive to build. The authors test whether LLMs guided only by prompts, through In-Context Learning, can do this work instead of fine-tuned models. They adapt a prompting framework from clinical NER (Zhang et al., 2023) to the LeNER-Br corpus. Models mark entities with HTML span tags. The study varies three factors:

  • one prompt per entity type versus a single prompt,
  • chunk length (300, 600 or 900 characters, or single sentences),
  • prompt content, adding layers step by step: guidelines, context-free examples, rules drawn from error analysis, and one- or five-shot examples.

Performance is measured with entity-level Exact Match Micro F1. Prompt content had a larger effect than splitting the task or shortening the input. The best setup also worked on the unseen test split, where GPT-5.2 reached 71.9% and a 4-bit Qwen-2.5:32B reached 57.9%. These scores remain well below a supervised LSTM-CRF (92.53%). Still, the results show that prompting is a workable option when labelled legal data is lacking.

Key points

  • Baseline setup: Qwen 3 (8.2B, Q4_K_M quantisation, temperature 0) run on 900-character chunks. RQ1–RQ3 used the LeNER-Br validation split and RQ4 used the test split. The authors dropped CDJUR-BR because its annotations were inconsistent; for example, "Ministério Público" was sometimes labelled Person and sometimes left unlabelled.
  • Adding instructions based on common errors (H3d) gave the largest single gain, raising Micro F1 from 41.8% to 48.0%. One-shot prompting (H3e) was the best configuration overall, and five examples did not improve on it.
  • Giving every entity type its own dedicated prompt outperformed the baseline. Among chunking options, single sentences worked best, followed by 300, 600 and 900 characters.
  • Among the small 4-bit models, Qwen-3:8B scored highest. An ensemble of the three small models beat the weaker individual ones but not the larger Qwen-2.5:32B or GPT-5.2.
  • A manual check of 5 GPT-5.2 output chunks found that most errors involved entity boundaries (50%, n=7). Missing entities accounted for 35.7% and spurious extractions for 14.3%, with no wrong-type errors. Code and prompts are published at github.com/giovannimaffeo/prompt-engineering-portuguese-legal-ner.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksLaw · 77 works

🧭 Topics 2Research subjects and languages the work is about.Named entity recognition · 200 worksLegal AI · 134 works

Cite

Cite as Giovanni Maffeo, Catarina Silva and Hugo Gonçalo Oliveira. 2026. Prompt Engineering for Named Entity Extraction from Portuguese Legal Documents. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{maffeo2026prompt,
  title = {{Prompt Engineering for Named Entity Extraction from Portuguese Legal Documents}},
  author = {Giovanni Maffeo, Catarina Silva and Hugo Gonçalo Oliveira},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.116/}
}

References

13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hidelberg O. Albuquerque, Rosimeire Costa, Gabriel Silvestre, Ellen Souza, Nádia F. F. Silva, Diego Vitório, and André C. P. L. F. de Carvalho. 2022. Ulyssesner-br: A corpus of brazilian legislative documents for named entity recognition. In Computational Processing of the Portuguese Language (PRO-POR). Springer. find
  2. Maurício Brito, Vládia Pinheiro, Vasco Furtado, João Araújo Monteiro Neto, Francisco das Chagas Jucá Bomfim, André C. F. Costa, Raquel Silveira, and Nilsiton Aragão. 2023. Cdjur-br: Uma coleção dourada do judiciário brasileiro com entidades nomeadas refinadas. In Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL). Sociedade Brasileira de Computação. find
  3. Cristina Mota, Diana Santos, Luís Miguel Cabral, and Nuno Silva. 2010. Segundo harem: Avaliação de reconhecimento de entidades mencionadas em português. In Actas do Encontro Nacional da Associação Portuguesa de Linguística. find
  4. Fei Yu, Linda Quartey, and Frank Schilder. 2023. Exploring the effectiveness of prompt engineering for legal reasoning tasks. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics. find
  5. Hao Zhang, Jingyan Wang, Shuai Yu, and 1 others. 2023. Improving large language models for clinical named entity recognition via prompt engineering. arXiv preprint. find
  6. Lucy Zheng, Neel Guha, Patrick Ivy, Daniel E. Ho, and Peter Henderson. 2021. When does pretraining help? assessing self-supervised learning for law and the case of casehold. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  7. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Nikos Aletras. 2020. Legal-bert: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT.
  9. Daniel Jurafsky and James H. Martin. 2023. Speech and Language Processing. Draft Edition, available online. find
  10. Daniel Martin Katz, Michael James Bommarito, and Josh Blackman. 2020. Complex societies and the growth of the law. arXiv preprint arXiv:2005.07646. ↗
  11. Junjie Li, Aixin Sun, Jie Han, and Chenliang Li. 2022. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering. find
  12. Pedro H. Luz de Araujo, Teófilo E. de Campos, Renato R. R. de Oliveira, Matheus Stauffer, Samuel Couto, and Paulo Bermejo. 2018. LeNER-Br: A dataset for named entity recognition in brazilian legal text. In Proceedings of the International Conference on the Computational Processing of Portuguese (PROPOR). find
  13. Chao Ma. 2023. Prompt engineering and calibration for zero-shot commonsense reasoning. arXiv preprint, arXiv:2304.06962. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/maffeo-2026-prompt-engineering-named-entity-extraction/