Elamur Research⌕
☰
home / papers / 2026 / From Trust to Compromise: Outcome-Verified LLM Phishing Simulation and Real-Time Defense

From Trust to Compromise: Outcome-Verified LLM Phishing Simulation and Real-Time Defense

Paper · 2026 · Tulika Tewari, Nalin Arachchilage, Jagat Sesh Challa et al.

Summary

Prior LLM-based social-engineering simulators judge an attack successful only when the victim discloses PII inside the chat. Real compromises, however, often happen off-platform, for example when a persuaded user enters credentials on a malicious website. The authors address this gap with PhishSim. In this simulator, a Gemini-2.5-Flash attacker poses as a fake recruiter and talks with persona-conditioned victim agents. A victim's decision to follow a link is then carried out by a browser agent that submits credentials to a dummy portal, so compromise is verified by outcome.

The result is 1,833 multi-turn dialogues in both phishing and benign modes. On this data, the authors evaluate PhishGate, a real-time multi-agent risk scorer. It first filters messages for manipulation tactics, then estimates attack severity, and runs a RAG consistency check only when the evidence is ambiguous.

With each of the four LLM backends tested, PhishGate flags phishing across whole conversations better than a baseline that also runs in real time. It warns earlier, often before the URL appears, and depends less on link cues. The work offers an outcome-grounded testbed for conversational phishing defense, while showing that LLM detectors remain brittle.

Key points

  • The PhishSim dataset has 1,833 dialogues, 919 of them phishing. Victims come from 192 personas built from gender, career level (early, mid or senior) and high or low levels of each Big Five trait.
  • Attacker tactics follow Cialdini's principles. The most frequent in phishing chats are Urgency (31.0%), Authority (29.9%) and Likability/Rapport (29.4%).
  • Compared with a RAG-only baseline, PhishGate's gating lowers the false-positive rate to 18.5%. It also cuts average latency from 2.08 s to 1.58 s per turn with Mistral Small 24B, avoiding about 32% overhead.
  • Some generated dialogues contain unresolved placeholders, which could act as shortcut cues. A clean-subset ablation on 200 test dialogues still gives F1 0.81 and accuracy 0.80 on placeholder-free dialogues, versus 0.84 F1 overall.
  • In the authors' evaluation, many LLM detectors had high false-positive rates and relied too heavily on URLs. Proprietary models such as Gemini performed comparatively well.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Birla Institute of Technology and Science, PilaniPilāni · India16 works · 41,879 citations🎓RMIT UniversityMelbourne · Australia17 works · 703 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Tulika Tewari, Nalin Arachchilage, Jagat Sesh Challa and Dhruv Kumar. 2026. From Trust to Compromise: Outcome-Verified LLM Phishing Simulation and Real-Time Defense. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.543

@inproceedings{tewari2026trust,
  title = {{From Trust to Compromise: Outcome-Verified LLM Phishing Simulation and Real-Time Defense}},
  author = {Tulika Tewari, Nalin Arachchilage, Jagat Sesh Challa and Dhruv Kumar},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.543},
  url = {https://doi.org/10.18653/v1/2026.acl-long.543}
}

References

28 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Khalifa Afane, Wenqi Wei, Ying Mao, Junaid Farooq, and Juntao Chen. 2024. Next-generation phishing: How llm agents empower cyber attackers. Preprint, arXiv:2411.13874. — [arXiv:2411.13874](https://arxiv.org/abs/2411.13874) find
  2. Lin Ai, Tharindu Sandaruwan Kumarage, Amrita Bhattacharjee, Zizhou Liu, Zheng Hui, Michael S. Davin-roy, James Cook, Laura Cassani, Kirill Trapeznikov, Matthias Kirchner, Arslan Basharat, Anthony Hoogs, Joshua Garland, Huan Liu, and Julia Hirschberg. 2024. Defending against social engineering attack… find
  3. Sanket Badhe. 2025. Scamagents: How ai agents can simulate human-level scam calls. Preprint, arXiv:2508.06457. — [arXiv:2508.06457](https://arxiv.org/abs/2508.06457) find
  4. Sanghwan Bae, Donghyun Kwak, Sungdong Kim, Donghoon Ham, Soyoung Kang, Sang-Woo Lee, and Woomyoung Park. 2022. Building a role specified open-domain dialogue system leveraging large-scale language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Comp… find
  5. Mazal Bethany, Athanasios Galiopoulos, Emet Bethany, Mohammad Bahrami Karkevandi, Nicole Beebe, Nishant Vishwamitra, and Peyman Najafirad. 2025. Lateral phishing with large language models: A large organization comparative study. IEEE Access, 13:60684–60701. find
  6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Lepikhin, and Wesley Helmholz. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint, arXiv:2507.06261.
  7. Kirsi Halttu and Harri Oinas-Kukkonen. 2022. Susceptibility to social influence strategies and persuasive system design: exploring the relationship. Behaviour & Information Technology, 41(12):2705–2726. find
  8. Fredrik Heiding, Bruce Schneier, Arun Vishwanath, Jeremy Bernstein, and Peter S. Park. 2023. Devising and detecting phishing: Large language models vs. smaller human models. Preprint, arXiv:2308.12287. — [arXiv:2308.12287](https://arxiv.org/abs/2308.12287) find
  9. Liliana Ribeiro, Inês Sousa Guedes, and Carla Sofia Cardoso. 2024. Which factors predict susceptibility to phishing? an empirical study. Computers Security, 136:103558. find
  10. Keepnet Labs. 2025. The 2025 new hires phishing susceptibility report. Technical report, Keepnet Labs. Whitepaper. find
  11. Dawn M. Sarno, Maggie W. Harris, and Jeffrey Black. 2023. Which phish is captured in the net? understanding phishing susceptibility and individual differences. Applied Cognitive Psychology, 37(4):789– 803. find
  12. Tharindu Kumarage, Cameron Johnson, Jadie Adams, Lin Ai, Matthias Kirchner, Anthony Hoogs, Joshua Garland, Julia Hirschberg, Arslan Basharat, and Huan Liu. 2025. Personalized attacks of social engineering in multi-turn conversations: Llm agents for simulation and detection. Preprint, arXiv:2503.1555… — [arXiv:2503.15552](https://arxiv.org/abs/2503.15552) find
  13. Wenhao Li, Selvakumar Manickam, Yung wey Chong, and Shankar Karuppayah. 2025. Phishdebate: An llm-based multi-agent framework for phishing website detection. Preprint, arXiv:2506.15656. — [arXiv:2506.15656](https://arxiv.org/abs/2506.15656) find
  14. Pablo López-Aguilar, Carlota Urruela, Edgar Batista, Juvenal Machin, and Agusti Solanas. 2025. Phishing vulnerability and personality traits: Insights from a systematic review. Computers in Human Behavior Reports, 20:100784. find
  15. Ioana Andreea Marin, Pavlo Burda, Nicola Zannone, and Luca Allodi. 2023. The influence of human factors on the intention to report phishing emails. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Machinery. find
  16. Robert R. McCrae and Oliver P. John. 1992. An introduction to the five-factor model and its applications. Journal of Personality, 60(2):175–215. find
  17. Ufuk Oner, Orcun Cetin, and Erkay Savas. 2025. Human factors in phishing: Understanding susceptibility and resilience. Computer Standards Interfaces, 94:104014. find
  18. Pablo and Agusti Solanas. 2021. Human susceptibility to phishing attacks based on personality traits: The role of neuroticism. In 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC), pages 1363–1368. find
  19. Peng Peng, Chao Xu, Luke Quinn, Hang Hu, Bimal Viswanath, and Gang Wang. 2019. What happens after you leak your password: Understanding credential sharing on phishing sites. In Proceedings of the 2019 ACM Asia Conference on Computer and Communications Security, Asia CCS ’19, page 181–192, New York,… find
  20. Marc Schmitt and Ivan Flechais. 2024. Digital deception: generative artificial intelligence in social engineering and phishing. Artificial Intelligence Review, 57(12):324. find
  21. Steve Sheng, Mandy Holbrook, Ponnurangam Kumaraguru, Lorrie Faith Cranor, and Julie Downs. 2010. Who falls for phish? a demographic analysis of phishing susceptibility and effectiveness of interventions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10, page 373… find
  22. Biplav Srivastava, Kausik Lakkaraju, Nitin Gupta, Vansh Nagpal, Bharath C. Muppasani, and Sara E. Jones. 2025. Safechat: A framework for building trustworthy collaborative assistants and a case study of its usefulness. Preprint, arXiv:2504.07995. — [arXiv:2504.07995](https://arxiv.org/abs/2504.07995) find
  23. Nikolaos Tsinganos, Panagiotis Fouliras, and Ioannis Mavridis. 2022. Applying bert for early-stage recognition of persistence in chat-based social engineering attacks. Applied Sciences, 12(23). find
  24. Nikolaos Tsinganos, Panagiotis Fouliras, Ioannis Mavridis, and Dimitris Gritzalis. 2024. Cse-ars: Deep learning-based late fusion of multimodal information for chat-based social engineering attack recognition. IEEE Access, 12:16072–16088. find
  25. Yizhu Wang, Haoyu Zhai, Chenkai Wang, Qingying Hao, Nick A. Cohen, Roopa Foulger, Jonathan A. Handler, and Gang Wang. 2025. Can you walk me through it? explainable sms phishing detection using llm-based agents. In Proceedings of the Twenty-First USENIX Conference on Usable Privacy and Security, SOUP… find
  26. Zihao Yi, Jiarui Ouyang, Zhe Xu, Yuwen Liu, Tianhao Liao, Haohao Luo, and Ying Shen. 2025. A survey on recent advances in llm-based multi-turn dialogue systems. ACM Comput. Surv. Just Accepted.
  27. Jinsol Yoo and Youngho Cho. 2022. Icsa: Intelligent chatbot security assistant using text-cnn and multi-phase real-time defense against sns phishing attacks. Expert Systems with Applications, 207:117893. find
  28. Qinglin Qi, Yun Luo, Yijia Xu, Wenbo Guo, and Yong Fang. 2025. Spearbot: Leveraging large language models in a generative-critique framework for spear-phishing email generation. Information Fusion, 122:103176. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tewari-2026-trust-compromise-outcome-verified-llm/