Elamur Research⌕
☰
home / papers / 2026 / Rating–Text Mismatch in Brazilian Portuguese Reviews: How Reliable Are Zero-Shot LLMs?

Rating–Text Mismatch in Brazilian Portuguese Reviews: How Reliable Are Zero-Shot LLMs?

Paper · 2026 · Emanuelle Marreira, Carlos M. S. Figueiredo and Tiago de Melo

Summary

Star ratings on e-commerce sites sometimes contradict the review text, for example an enthusiastic comment paired with a single star. Such mismatches distort satisfaction metrics and can mislead shoppers and recommender systems. The authors test whether off-the-shelf LLMs, without task-specific training, can reliably flag these contradictions in Brazilian Portuguese.

They crawled Amazon Brazil reviews from 2021–2024 and kept only 1- and 5-star items, where the expected polarity is clear. Ratings were balanced within ten product categories, yielding 20,586 reviews. Five models received one generic zero-shot prompt: GPT-5, DeepSeek-V3.2 and LLaMA-4-Scout, plus the Portuguese-oriented Sabiá-3.1 and Bode-3.1-8B. Each model ran three times with deterministic decoding and was scored against a human-annotated subset.

DeepSeek-V3.2 performed best, reaching an F1 close to 0.98 on the incoherent class. Repeated runs agreed almost perfectly (Fleiss' κ above 0.95). The work contributes an open Portuguese resource and shows that zero-shot LLMs can support review moderation and reputation systems.

Key points

  • Dataset: 20,586 Amazon Brazil reviews (10,293 per rating) across ten categories, with Books the largest and Computers the smallest. The data and annotations are released on Zenodo.
  • Evaluation subset: 389 human-labeled reviews, of which 152 (39.1%) are incoherent. Incoherence affected 43.5% of 1-star reviews versus 33.7% of 5-star ones.
  • Incoherent-class F1 by model:
    • DeepSeek-V3.2: 0.979
    • LLaMA-4-Scout: 0.967 (recall 1.0)
    • Sabiá-3.1: 0.938
    • GPT-5: 0.822 (precision 1.0 but recall about 0.70)
    • Bode-3.1-8B: 0.336
  • Category analysis with DeepSeek-V3.2: detected mismatches mostly hover around 10%. Computers is lowest (4.8%) and Cell Phones highest (12.9%).
  • Stability and future work: within-model agreement over three runs was measured with Fleiss' κ and Cohen's κ, using temperature 0 or do_sample=False. Planned extensions include few-shot prompting, chain-of-thought, fine-tuning and intermediate (2–4 star) ratings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Emanuelle Marreira, Carlos M. S. Figueiredo and Tiago de Melo. 2026. Rating–Text Mismatch in Brazilian Portuguese Reviews: How Reliable Are Zero-Shot LLMs?. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{marreira2026rating,
  title = {{Rating–Text Mismatch in Brazilian Portuguese Reviews: How Reliable Are Zero-Shot LLMs?}},
  author = {Emanuelle Marreira, Carlos M. S. Figueiredo and Tiago {de Melo}},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.96/}
}

References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hugo Abonizio, Thales Sales Almeida, Thiago Laitz, Roseval Malaquias Junior, Giovana Kerche Bonás, Rodrigo Nogueira, and Ramon Pires. 2024. Sabiá-3 technical report. arXiv preprint arXiv:2410.12049. — [arXiv:2410.12049](https://arxiv.org/abs/2410.12049) find
  2. Turki Aljrees, Muhammad Umer, Oumaima Saidani, Latifah Almuqren, Abid Ishaq, Shtwai Alsubai, Imran Ashraf, and 1 others. 2024. Contradiction in text review and apps rating: prediction using textual features and transfer learning. PeerJ Computer Science, 10:e1722. find
  3. Amal Almansour, Reem Alotaibi, and Hajar Alharbi. 2022. Text-rating review discrepancy (trrd): an integrative review and implications for research. Future Business Journal, 8(1):3. find
  4. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, and 1 others. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. find
  5. Michela Fazzolari, Vittoria Cozza, Marinella Petrocchi, and Angelo Spognardi. 2017. A study on text-score disagreement in online reviews. Cognitive Computation, 9(5):689–701. find
  6. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  7. Nan Hu, Noi Sian Koh, and Srinivas K Reddy. 2014. Ratings lead you to the product, reviews help you clinch it? the mediating role of online review sentiments on product sales. Decision support systems, 57:42–53. find
  8. Mir Riyanul Islam. 2014. Numeric rating of apps on google play store by sentiment analysis on user reviews. In 2014 international conference on electrical engineering and information & communication technology, pages 1–4. IEEE. find
  9. Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. 2023. Do llms understand user preferences? evaluating llms on user rating prediction. arXiv preprint arXiv:2305.06474. — [arXiv:2305.06474](https://arxiv.org/abs/2305.06474) find
  10. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  11. Shijie Liu, Ruixin Ding, Weihai Lu, Jun Wang, Mo Yu, Xiaoming Shi, and Wei Zhang. 2025. Coherency improved explainable recommendation via large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12201– 12209. find
  12. Emanuelle Marreira, Tiago de Melo, Miguel de Oliveira, and Carlos MS Figueiredo. 2025a. Rating prediction in brazilian portuguese: A benchmark of large language models. Journal of the Brazilian Computer Society, 31(1):827–838. find
  13. Emanuelle Marreira, Tiago de Melo, Miguel Oliveira, and Carlos Maurício. 2025b. Detectando incoerên-cias avaliativas em e-commerce com llms-um estudo de caso na amazon brasil. In Brazilian Symposium on Multimedia and the Web (WebMedia), pages 535–539. SBC. find
  14. Juan Pedro Mellinas, Juan L Nicolau, and Sangwon Park. 2019. Inconsistent behavior in online consumer reviews: The effects of hotel attribute ratings on location. Tourism Management, 71:421–427. find
  15. Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta.com/blog/ llama-4-multimodal-intelligence/. Accessed on: November 10, 2025. ↗
  16. Susan M Mudambi, David Schuff, and Zhewei Zhang. 2014. Why aren’t the stars aligned? an analysis of online review content and star ratings. In 2014 47th Hawaii International conference on system sciences, pages 3139–3147. IEEE. find
  17. OpenAI. 2025. Gpt-5 system card. https://openai.com/pt-BR/index/gpt-5-system-card/. Accessed on: November 10, 2025. ↗
  18. Denilson Alves Pereira. 2021. A survey of sentiment analysis in the portuguese language. Artificial Intelligence Review, 54(2):1087–1115. find
  19. Abhinav Sharma, Sangwon Park, and Juan L Nicolau. 2020. Testing loss aversion and diminishing sensitivity in review sentiment. Tourism Management, 77:104020. find
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  21. Xiaoyu Zhang, Yishan Li, Jiayin Wang, Bowen Sun, Weizhi Ma, Peijie Sun, and Min Zhang. 2024. Large language models as evaluators for recommendation explanations. In Proceedings of the 18th ACM Conference on Recommender Systems, pages 33–42. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/marreira-2026-rating-text-mismatch-brazilian-portuguese/