Elamur Research⌕
☰
home / papers / 2026 / Utterance-level Detection Framework for LLM-Involved Content Detection in Conversational Setting

Utterance-level Detection Framework for LLM-Involved Content Detection in Conversational Setting

Paper · 2026 · Muyang Zhou and Huaxia Rui

Summary

Most detectors of machine-generated text are built for static, document-level inputs. They handle short, interactive exchanges poorly, especially when human and model turns are mixed in one dialogue. The authors reframe detection at the utterance level within conversations and supply both a dataset and a model. The dataset combines real human-human dialogues from Wizard of Wikipedia, Persona-Chat and MuTual with two kinds of synthetic dialogues. In LLM-LLM dialogues, two persona-prompted assistants converse turn by turn. In human-LLM dialogues, people chat with a model on summarized topics.

The detector is lightweight. It uses the LLaMA3 tokenizer with a small custom transformer, arranged recurrently over dialogue turns. It mixes self-attention and cross-attention between the two speakers. A detection focus augmentation strategy emphasizes the second speaker's replies while keeping the first speaker's turns as context.

On human-LLM dialogues it reaches 98.14% accuracy with fast inference and outperforms baseline detectors. It also generalizes reasonably in zero-shot tests on out-of-domain data. This makes it a practical candidate for real-time transparency checks in chat systems.

Key points

  • Dataset: over 80,000 training dialogues in three parts (human-human, LLM-LLM, human-LLM).
    • Synthetic turns come from GPT-4o, LLaMA-3.1-70B-Instruct and Qwen2.5-72B-Instruct.
    • Turn counts match those of the real human conversations.
  • Model: LLaMA3 tokenizer feeding a custom 4-layer transformer for feature extraction, followed by self-attention and cross-attention blocks in a recurrent arrangement with a decoder at each stage.
  • Results and ablations:
    • 98.14% accuracy on human-LLM dialogues.
    • The LLM tokenizer adds about 9% accuracy compared with a traditional tokenizer.
    • Accuracy exceeds 90% on some out-of-domain tasks.
  • Dialogue length: training uses 5-round dialogues.
    • Shorter inputs are padded with blank content.
    • Longer ones are split into 5-round sliding windows whose predictions are combined by voting.
    • Compared with models trained on 1–10 rounds, performance remains comparable.
  • Limitations:
    • Covers only two-party dialogues.
    • May be weaker in specialized domains such as medical or legal.
    • Has notable compute cost.
    • Cross-lingual and multi-party detection are left as future work, which matters for applying it to Ukrainian.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of RochesterRochester · United States43 works · 7,874 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 654 works

Cite

Cite as Muyang Zhou and Huaxia Rui. 2026. Utterance-level Detection Framework for LLM-Involved Content Detection in Conversational Setting. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.63

@inproceedings{zhou2026utterance,
  title = {{Utterance-level Detection Framework for LLM-Involved Content Detection in Conversational Setting}},
  author = {Muyang Zhou and Huaxia Rui},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.63},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.63}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2023. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130.
  3. Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Raj. 2023. Token prediction as implicit classification to identify llm-generated text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page 13112–13120. Association for Computational Li… find
  4. Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. MuTual: A dataset for multi-turn dialogue reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1406–1416, Online. Association for Computational Linguistics.
  5. Emily Dinan et al. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of ICLR.
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. 2021. Tweepfake: About detecting deepfake tweets. Plos one, 16(5):e0251415.
  8. Pengyu Wang, Linyang Li, Ke Ren, Botian Jiang, Dong Zhang, and Xipeng Qiu. 2023. Seqxgpt: Sentence-level ai-generated text detection. arXiv preprint arXiv:2310.08903. find
  9. Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation. Preprint, arXiv:1908.11540.
  10. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  11. Google. 2025. Gemini 2.0 flash. https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/ \#gemini-2-0-flash. ↗
  12. Xiaodong Gu, Kang Min Yoo, and Jung-Woo Ha. 2021. Dialogbert: Discourse-aware response generation via learning to recover and rank utterances. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12911–12919. find
  13. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. Preprint, arXiv:2301.07597.
  14. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  15. Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6818–6825. find
  16. Peipeng Yu, Jiahan Chen, Xuan Feng, and Zhihua Xia. 2024. Cheat: A large-scale dataset for detecting chatgpt-written abstracts. Preprint, arXiv:2304.12008. — [arXiv:2304.12008](https://arxiv.org/abs/2304.12008) find
  17. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. Advances in neural information processing systems, 32.
  18. Zijie Zeng, Lele Sha, Yuheng Li, Kaixun Yang, Dragan Gašević, and Guanliang Chen. 2023. Towards automatic boundary detection for human-ai collaborative hybrid essay in education. Preprint, arXiv:2307.12267. — [arXiv:2307.12267](https://arxiv.org/abs/2307.12267) find
  19. Bowen Zhang, Xiaofei Xu, Xutao Li, Yunming Ye, Xiaojun Chen, and Zhongjie Wang. 2020. A memory network based end-to-end personalized task-oriented dialogue generation. Knowledge-Based Systems, 207:106398. find
  20. Saizheng Zhang et al. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of ACL.
  21. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. Preprint, arXiv:2301.11305.
  22. Zhenpeng Su, Xing Wu, Wei Zhou, Guangyuan Ma, and Songlin Hu. 2024. Hc3 plus: A semantic-invariant human chatgpt comparison corpus. Preprint, arXiv:2309.02731. — [arXiv:2309.02731](https://arxiv.org/abs/2309.02731) find
  23. Jiamin Wang, Xiao Sun, Qian Chen, and Meng Wang. 2022. Information-enhanced hierarchical self-attention network for multiturn dialog generation. IEEE Transactions on Computational Social Systems, 10(5):2686–2697. find
  24. Jian Wang, Junhao Liu, Wei Bi, Xiaojiang Liu, Kejing He, Ruifeng Xu, and Min Yang. 2020. Dual dynamic memory network for end-to-end multi-turn task-oriented dialog systems. In Proceedings of the 28th international conference on computational linguistics, pages 4100–4110. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-utterance-level-detection-framework-llm/