Elamur Research⌕
☰
home / papers / 2026 / Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News

Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News

Paper · 2026 · João Victor Assaoka Ribeiro, Thomas Pires Correia, José Vitor Souza Cardoso Requena et al.

Summary

Production pipelines that summarize Portuguese financial news rarely have expert-written reference summaries, so metrics like ROUGE cannot be used. Holistic LLM-as-a-Judge scoring is also opaque and may miss subtle factual errors. The authors test whether breaking evaluation into atomic question-answer checks works better. Summaries were produced with gpt-4.1-nano from articles in the Exame portal, which belong to a corpus being built for sentiment analysis. These summaries were scored with a reference-free QA pipeline run on gpt-5-nano. The pipeline uses three answer-scoring variants: Lexical (token F1), Binary, and Semantic (LLM-based). The authors compared it with a direct LLM-Judge baseline and checked all metrics against human ratings of Faithfulness and Completeness for 50 items. For Completeness, the QA metrics tracked human judgments more closely than the holistic judge, and QA-Binary was strongest. For Faithfulness, human scores clustered near the maximum, so correlations were not meaningful. Case studies showed the Semantic QA metric catching hallucinations that annotators missed. The work offers a transparent, scalable way to audit Portuguese financial summaries without references.

Key points

  • Corpus: 4,841 articles from the Brazilian portal Exame. Each was condensed into three sentences, with an average compression rate of about 5.41.
  • Human ground truth: 50 randomly sampled items, each rated by two of four annotators (100 annotations in total) on a 1–5 Likert scale. Faithfulness ratings matched exactly in 43 cases, while Completeness ratings differed in 21.
  • Completeness result: QA-Binary achieved the best rank correlation (ρ ≈ 0.49) under pessimistic aggregation of human scores, outperforming the 0–10 LLM-Judge prompted as a senior economic journalism editor.
  • Faithfulness result: A ceiling effect made most correlations non-significant (p > 0.05). Token-overlap QA-F1 penalized correct but differently worded answers. QA-LLM flagged a shift from "next year" to 2024 and an invented event, both of which humans overlooked.
  • Weighting: A plain arithmetic mean over QA items performed as well as or better than risk-aware importance weighting. The authors attribute this to the short summaries, where nearly every remaining fact is critical.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksSummarization · 232 works

Cite

Cite as João Victor Assaoka Ribeiro, Thomas Pires Correia, José Vitor Souza Cardoso Requena and Lilian Berton. 2026. Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ribeiro2026evaluating,
  title = {{Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News}},
  author = {João Victor Assaoka Ribeiro, Thomas Pires Correia, José Vitor Souza Cardoso Requena and Lilian Berton},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.89/}
}

References

14 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055– 5070, Online. Association for Computational Lingu…
  2. Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacteval: Improved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language technolo…
  3. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1–38.
  4. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  5. Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. FineSurE: Fine-grained summarization evaluation using LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 906–922, Bangkok, Thailand. Association for…
  6. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228. find
  7. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020b. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. find
  8. Minghao Wu and Alham Fikri Aji. 2025. Style over substance: Evaluation biases for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 297–312.
  9. Weijia Zhang, Mohammad Aliannejadi, Yifei Yuan, Jiahuan Pei, Jia-Hong Huang, and Evangelos Kanoulas. 2024. Towards fine-grained citation evaluation in generated text: A comparative analysis of faithfulness metrics. arXiv preprint arXiv:2406.15264. find
  10. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  11. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  12. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661.
  13. Arman Khadjeh Nassirtoussi, Saeed Aghabozorgi, Teh Ying Wah, and David Chek Ling Ngo. 2014. Text mining for market prediction: A systematic review. Expert Systems with Applications, 41(16):7653– 7670. find
  14. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. Questeval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 6594–6604.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ribeiro-2026-evaluating-reference-free-summarization-quality/