Elamur Research⌕
☰
home / papers / 2026 / Why is “Chicago” Predictive of Deceptive Reviews? Using LLMs to Discover Language Phenomena from Lexical Cues

Why is “Chicago” Predictive of Deceptive Reviews? Using LLMs to Discover Language Phenomena from Lexical Cues

Paper · 2026 · Jiaming Qu, Mengtian Guo and Yue Wang

Summary

Classifiers that flag fake online reviews often rely on word-level signals that seem meaningless to people. For example, the city name "Chicago" points toward deception. This limits how much users can learn from such models or trust them. The authors ask whether LLMs can turn these opaque lexical cues into readable descriptions of underlying language phenomena that are backed by data. Using hotel reviews as a case study, they first extract stable predictive words with logistic regression. LLMs then propose ("conjecture") phenomena that might explain those words. Finally, the conjectures are given to an LLM as classification guidance, without the original words. If they improve its accuracy, they are treated as real patterns rather than hallucinations. The resulting phenomena were predictive and carried over to reviews of hotels in other cities. They also outperformed phenomena produced from LLM prior knowledge alone or from in-context labeled examples. In a human study, they improved participants' understanding of deception cues. This helps people judge review credibility where no automated detector exists.

Key points

  • Data: D_Chicago (Ott et al., 2011, 2013) has 800 genuine and 800 deceptive Chicago hotel reviews. D_three-cities (Li et al., 2013) has 240 genuine and 240 deceptive reviews of hotels in Houston, New York, and Los Angeles.
  • Cue discovery used logistic regression on unigram TF-IDF features with 10-fold cross-validation, reaching an average F1 of 0.88. The top 25 words per class were taken from each fold and checked with a Wald test.
  • Only words predictive in all 10 folds were kept. This left 30 cues: 16 for genuine and 14 for deceptive reviews. Phenomena were conjectured from these cues and aggregated over four LLMs.
  • The authors call their approach a conjecture-then-validate framework. Validation is algorithmic, because no ground truth exists for the phenomena. It was complemented by a human evaluation.
  • Stated limitations:
    • Validation does not guarantee faithfulness to the model or to causal mechanisms.
    • The phenomena come from single-pass prompting.
    • The study covers only the hotel-review domain.
    • Human evaluation used lay participants.
    • The phenomena could be misused to write more convincing fake reviews.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North Carolina at Chapel HillChapel Hill · United States67 works · 16,474 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

Cite

Cite as Jiaming Qu, Mengtian Guo and Yue Wang. 2026. Why is “Chicago” Predictive of Deceptive Reviews? Using LLMs to Discover Language Phenomena from Lexical Cues. Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.trustnlp-main.41

@inproceedings{qu2026why,
  title = {{Why is “Chicago” Predictive of Deceptive Reviews? Using LLMs to Discover Language Phenomena from Lexical Cues}},
  author = {Jiaming Qu, Mengtian Guo and Yue Wang},
  booktitle = {Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.trustnlp-main.41},
  url = {https://doi.org/10.18653/v1/2026.trustnlp-main.41}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jiwei Li, Myle Ott, and Claire Cardie. 2013. Identifying manipulated offerings on review portals. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1933–1942. find
  2. About Amazon Team. 2023. Amazon, Booking.com, Expedia Group, Glassdoor, Tripadvisor, and Trustpilot launch first global Coalition for Trusted Reviews. find
  3. Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.
  4. David Martens, James Hinns, Camille Dams, Mark Ver-gouwen, and Theodoros Evgeniou. 2023. Tell me a story! narrative-driven xai with large language models. arXiv preprint arXiv:2309.17057. — [arXiv:2309.17057](https://arxiv.org/abs/2309.17057) find
  5. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In Proceedings of the AAAI conference on artificial intelligence, volume 32. find
  6. OpenAI. 2025. Introducing gpt-5. Accessed: 2025-11-04. find
  7. Hendrik Schuff, Alon Jacovi, Heike Adel, Yoav Goldberg, and Ngoc Thang Vu. 2022. Human interpretation of saliency-based explanation over text. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 611–636. find
  8. Myle Ott, Claire Cardie, and Jeffrey T Hancock. 2013. Negative deceptive opinion spam. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 497–501. find
  9. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T Hancock. 2011. Finding deceptive opinion spam by any stretch of the imagination. arXiv preprint arXiv:1107.4557.
  10. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319– 3328. PMLR. find
  11. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  12. Bo Pan, Zhen Xiong, Guanchen Wu, Zheng Zhang, Yifei Zhang, and Liang Zhao. 2024. Tagexplainer: Narrating graph explanations for text-attributed graph learning models. arXiv preprint arXiv:2410.15268. — [arXiv:2410.15268](https://arxiv.org/abs/2410.15268) find
  13. Himangshu Paul and Alexander Nikolaev. 2021. Fake review detection on online e-commerce platforms: a systematic literature review. Data Mining and Knowledge Discovery, 35(5):1830–1881. find
  14. Anthropic PBC. 2025. Introducing claude haiku 4.5. Accessed: 2025-11-04. find
  15. Jiaming Qu, Jaime Arguello, and Yue Wang. 2021. A study of explainability features to scrutinize faceted filtering results. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1498–1507. find
  16. Jiaming Qu, Jaime Arguello, and Yue Wang. 2023. Understanding the cognitive influences of interpretability features on how users scrutinize machine-predicted categories. In ACM SIGIR Conference On Human Information Interaction And Retrieval. find
  17. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. find
  18. Michael Tsang, Sirisha Rambhatla, and Yan Liu. 2020. How does this interaction affect me? interpretable attribution for feature interactions. Advances in neural information processing systems, 33:6147–6159. find
  19. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Computational Linguistics, 50(1):237–291.
  20. Alexandra Zytek, Sara Pidò, and Kalyan Veeramachaneni. 2024. Llms for xai: Future directions for explaining explanations. arXiv preprint arXiv:2405.06064. — [arXiv:2405.06064](https://arxiv.org/abs/2405.06064) find
  21. Jiaming Qu, Jaime Arguello, and Yue Wang. 2024. Why is" problems" predictive of positive sentiment? a case study of explaining unintuitive features in sentiment classification. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 161–172. find
  22. Jiaming Qu, Jaime Arguello, and Yue Wang. 2025. Understanding the effects of explaining predictive but unintuitive features in human-xai interaction. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 296–311. find
  23. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135– 1144.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qu-2026-why-is-chicago-predictive-deceptive/