Elamur Research⌕
☰
home / papers / 2026 / Auditing the Evaluators: How Far Can Automatic Evaluation Go in Assessing Portuguese Financial Texts?

Auditing the Evaluators: How Far Can Automatic Evaluation Go in Assessing Portuguese Financial Texts?

Paper · 2026 · Marina Ramalhete Masid, Gabriel Assis, Daniela Vianna et al.

Summary

The paper asks whether automatic evaluators can be trusted to judge Portuguese financial commentaries, which are expert texts interpreting official corporate disclosures (material facts). In this setting, wrong company names or numbers can seriously mislead investors. The authors used 72 expert-written commentaries about 35 Brazilian companies and created a synthetic corpus by applying seven controlled noise types to them. Each noisy text was compared with its clean original by classical metrics and by several open LLMs acting as judges. The LLM judges rated five dimensions: facts, entities, numerical values, fluency and writing. Classical metrics reacted to surface damage but often missed structural or meaning-changing edits, especially entity swaps. LLM judges varied widely between models. Some smaller models frequently gave noisy texts high ratings. Other models judged more evenly, but none reliably caught entity substitutions or implicit value errors. The study concludes that neither approach is yet a safe stand-in for correctness checks in this high-stakes domain. It calls for evaluators that are aware of domain and context.

Key points

  • Data: 72 Portuguese financial commentaries covering 35 companies, collected from financial analysis firms' websites following Assis et al. (2025). Companies include Petrobras, Vale, Bradesco, Banco do Brasil, Sabesp and BTG Pactual.
  • Perturbations: seven noise types were applied, using spaCy and the Augmenty library:
    • swapping named entities for other ones
    • shuffling the order of sentences
    • introducing spelling mistakes
    • replacing words with synonyms
    • random insertion
    • random swapping
    • random deletion
  • Classical metrics: BLEU and ROUGE respond to surface-level disruptions but handle structural changes poorly. BERTScore and CTC frequently overlook domain-critical changes such as entity replacement.
  • LLM judges and overall ranking: the judges were llama3.1:8b, phi-4:14b, gemma-3:27b and deepseek-r1 (8b/14b/32b). deepseek-r1:32b and phi-4:14b gave the most balanced assessments, with phi-4:14b strong despite its size. gemma-3:27b is sensitive to fluency and writing but weak on facts. llama3.1:8b and deepseek-r1:8b were inconsistent.
  • Specific failures: most models did not treat sentence reordering as serious. Detection of entity replacement stayed well below 100%. Synonym substitutions that changed meaning, such as replacing the company name "Vale" with a geographic term, were caught by only some models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Marina Ramalhete Masid, Gabriel Assis, Daniela Vianna, Aline Paes and Altigran Soares da Silva. 2026. Auditing the Evaluators: How Far Can Automatic Evaluation Go in Assessing Portuguese Financial Texts?. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{masid2026auditing,
  title = {{Auditing the Evaluators: How Far Can Automatic Evaluation Go in Assessing Portuguese Financial Texts?}},
  author = {Marina Ramalhete Masid, Gabriel Assis, Daniela Vianna, Aline Paes and Altigran Soares {da Silva}},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.22/}
}

References

4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… — Phi-4 Technical Report
  2. Gabriel Assis, Hugo Dutra, Daniela Vianna, Júnior Meira, Wagner, Altigran Soares da Silva, and Aline Paes. 2025. Language Models for Automated Market Commentary from Corporate Disclosures. In Proceedings of the 6th ACM International Conference on AI in Finance, ICAIF ’25, page 727–735, New York, NY,… find
  3. Gabriel Assis, Daniela Vianna, Gisele L. Pappa, Alexandre Plastino, Wagner Meira Jr, Altigran Soares da Silva, and Aline Paes. 2024. Analysis of material facts on financial assets: A generative AI approach. In Proceedings of the Joint Workshop of the 7th Financial Technology and Natural Language Pro… find
  4. Ozan Caglayan, Pranava Madhyastha, and Lucia Specia. 2020. Curious case of language generation evaluation metrics: A cautionary tale. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2322–2328, Barcelona, Spain (Online). International Committee on Computational… find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/masid-2026-auditing-evaluators-how-far-can/