Elamur Research⌕
☰
home / papers / 2026 / HumaniCA: A Benchmark Resource for the Detection of Users' Ascription of Humanness to Conversational Agents

HumaniCA: A Benchmark Resource for the Detection of Users' Ascription of Humanness to Conversational Agents

Summary

Users often treat text-based chatbots as if they were people, which can inflate their expectations beyond what the system can do. Currently, spotting this behaviour requires slow manual analysis or questionnaires, and no resource supports automatic detection. The authors address this with HumaniCA, a benchmark of 2,100 user turns taken from real conversations with three kinds of agents: task-oriented, Q&A and LLM-based. Each turn carries a binary label marking whether the user attributes human qualities to the agent. Drawing on prior work on communication styles, politeness and self-disclosure, they designed interpretable heuristic features that capture cues such as courtesy, social references and sharing personal information. They compared these features with TF-IDF weights and SentenceBERT embeddings, alone and in combination, using Naïve Bayes, Logistic Regression and SVM classifiers. The heuristic features, used alone or combined, gave the best accuracy for every agent type. The resource could let agents recognise anthropomorphizing users and adjust their replies to bring expectations back in line with reality.

Key points

  • HumaniCA contains three annotated datasets, one per agent type (task-oriented, Q&A, LLM-based), with 2,100 user turns in total and binary humanness labels.
  • The heuristic features build on earlier studies: the Push/Pull models from Cho (2018), humanness levels from Rapp et al. (2024), social-attitude signs from Novielli et al. (2010), the politeness framework from Danescu-Niculescu-Mizil et al. (2013) and self-disclosure cues from Wang et al. (2016).
  • The baselines compare three feature families: heuristic, frequency-based (TF-IDF) and semantic-based (SentenceBERT), plus their combinations.
  • Three classifiers were evaluated: an SVM, a Logistic Regression model and a Naïve Bayes model.
  • The linguistic heuristic features consistently outperform the alternatives across all three agent types.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3uujwof4yj3o

@inproceedings{villata2026humanica,
  title = {{HumaniCA: A Benchmark Resource for the Detection of Users' Ascription of Humanness to Conversational Agents}},
  author = {Villata, Sabrina and Rapp, Amon and Caro, Luigi Di and Cena, Federica},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3uujwof4yj3o},
  url = {https://doi.org/10.63317/3uujwof4yj3o}
}

References

44 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Issa Annamoradnejad and Gohar Zoghi. 2024. Colbert: Using bert sentence embedding in parallel neural networks for computational humor. Expert Systems with Applications, 249:123685. find
  2. Theo Araujo and Nadine Bol. 2024. From speaking like a person to being personal: The effects of personalized, regular interactions with conversational agents. Computers in Human Behavior: Artificial Humans, 2(1):100030. find
  3. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. Multi-WOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language… find
  4. Ana Paula Chaves and Marco Aurelio Gerosa. 2021. How should my chatbot interact? a survey on social characteristics in human–chatbot interaction design. International Journal of Human– Computer Interaction, 37(8):729–758. find
  5. Yi-Shyuan Chiang, Ruei-Che Chang, Yi-Lin Chuang, Shih-Ya Chou, Hao-Ping Lee, I-Ju Lin, Jian-Hua Jiang Chen, and Yung-Ju Chang. 2020. Exploring the design space of user-system communication for smart-home routine assistants. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Syst… find
  6. Janghee Cho. 2018. Mental models and home virtual assistants (hvas). In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–6. find
  7. Christopher Clarke, Joseph Joshua Peper, Karthik Krishnamurthy, Walter Talamonti, Kevin Leach, Walter Lasecki, Yiping Kang, Lingjia Tang, and Jason Mars. 2022. One agent to rule them all: Towards multi-agent conversational ai. arXiv preprint arXiv:2203.07665. find
  8. Cammy Crolic, Felipe Thomaz, Rhonda Hadi, and Andrew T Stephen. 2022. Blame the bot: Anthropomorphism and anger in customer–chatbot interactions. Journal of Marketing, 86(1):132– 148. find
  9. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. arXiv preprint arXiv:1306.6078. find our card
  10. Sidney K D’mello, Scotty D Craig, Amy Witherspoon, Bethany McDaniel, and Arthur Graesser. 2008. Automatic detection of learner’s affect from conversational cues. User modeling and user-adapted interaction, 18(1):45–80. find
  11. Nicholas Epley, Adam Waytz, and John T Cacioppo. 2007. On seeing human: a three-factor theory of anthropomorphism. Psychological review, 114(4):864. find
  12. Roger Ferrod, Federica Cena, Luigi Di Caro, Dario Mana, and Rossana Grazia Simeoni. 2021. Identifying users’ domain expertise from dialogues. In Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, pages 29–34. find
  13. Eun Go and S Shyam Sundar. 2019. Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions. Computers in human behavior, 97:304–316. find
  14. Margherita Grandini, Enrico Bagli, and Giorgio Visani. 2020. Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.05756. — [arXiv:2008.05756](https://arxiv.org/abs/2008.05756) find
  15. Yingjie Hu. 2018. Geo-text data and data-driven geospatial semantics. Geography Compass, 12(11):e12404. find
  16. Clayton Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media, volume 8, pages 216–225. find
  17. Mohit Jain, Pratyush Kumar, Ramachandra Kota, and Shwetak N Patel. 2018. Evaluating and informing the design of chatbots. In Proceedings of the 2018 designing interactive systems conference, pages 895–906. find
  18. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1–38. find our card
  19. Daniel Jurafsky and James H. Martin. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 1st edition. Prentice Hall PTR, USA. find
  20. Vishal Kuvar, Nathaniel Blanchard, Alexander Colby, Laura Allen, and Caitlin Mills. 2023. Automatically detecting task-unrelated thoughts during conversations using keystroke analysis. User Modeling and User-Adapted Interaction, 33(3):617–641. find
  21. Ewa Luger and Abigail Sellen. 2016. " like having a really bad pa" the gulf between user expectation and experience of conversational agents. In Proceedings of the 2016 CHI conference on human factors in computing systems, pages 5286–5297. find
  22. François Mairesse and Marilyn A Walker. 2009. Can conversational agents express big five personality traits through language?: Evaluating a psychologically-informed language generator. Research Gate, 8. find
  23. François Mairesse, Marilyn A Walker, Matthias R Mehl, and Roger K Moore. 2007. Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of artificial intelligence research, 30:457–500. find
  24. David Maulsby, Saul Greenberg, and Richard Mander. 1993. Prototyping an intelligent agent through wizard of oz. In Proceedings of the IN-TERACT’93 and CHI’93 conference on Human factors in computing systems, pages 277–284. find
  25. Sourabrata Mukherjee, Vojtěch Hudeček, and Ondřej Dušek. 2023. Polite chatbot: A text style transfer application. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 87–93. find
  26. Thai Ha Nguyen, Lena Waizenegger, and Angsana A Techatassanasoontorn. 2022. “don’t neglect the user!”–identifying types of human-chatbot interactions and their associated characteristics. Information Systems Frontiers, 24(3):797–838. find
  27. Andreea I Niculescu and Rafael E Banchs. 2019. Humor intelligence for virtual agents. In 9th international workshop on spoken dialogue system technology, pages 285–297. Springer. find
  28. Nicole Novielli, Fiorella de Rosis, and Irene Mazzotta. 2010. User attitude towards an embodied conversational agent: Effects of the interaction mode. Journal of Pragmatics, 42(9):2385–2397. find
  29. Mohammad Nuruzzaman and Omar Khadeer Hussain. 2018. A survey on chatbot implementation in customer service industry through deep neural networks. In 2018 IEEE 15th international conference on e-business engineering (ICEBE), pages 54–61. IEEE. find
  30. Amon Rapp, Arianna Boldi, Lorenzo Curti, Alessandro Perrucci, and Rossana Simeoni. 2023. Collaborating with a text-based chatbot: an exploration of real-world collaboration strategies enacted during human-chatbot interactions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing S… find
  31. Amon Rapp, Arianna Boldi, Lorenzo Curti, Alessandro Perrucci, and Rossana Simeoni. 2024. How do people ascribe humanness to chatbots? an analysis of real-world human-agent interactions and a theoretical model of humanness. International Journal of Human–Computer Interaction, 40(19):6027–6050. find
  32. Amon Rapp, Lorenzo Curti, and Arianna Boldi. 2021. The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. International Journal of Human-Computer Studies, 151:102630. find
  33. Amon Rapp, Maurizio Tirassa, and Lia Tirabeni. 2019. Rethinking technologies for behavior change: A view from the inside of human change. ACM Transactions on Computer-Human Interaction (TOCHI), 26(4):1–30. find
  34. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. find
  35. C Sekhar, MS Rao, ASK Nayani, and D Bhattacharyya. 2021. Emotion recognition through human conversation using machine learning techniques. Advances in Intelligent Systems and Computing. find
  36. William Seymour and Max Van Kleek. 2021. Exploring interactions between trust, anthropomorphism, and relationship development in voice assistants. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–16. find
  37. Marita Skjuve, Asbjørn Følstad, and Petter Bae Brandtzaeg. 2023. The user experience of chatgpt: findings from a questionnaire study of early users. In Proceedings of the 5th international conference on conversational user interfaces, pages 1–10. find
  38. Felipe Thomaz, Carolina Salge, Elena Karahanna, and John Hulland. 2020. Learning from the dark web: leveraging conversational agents in the era of hyper-privacy to enhance marketing. Journal of the Academy of Marketing Science, 48(1):43– 63. find
  39. Yi-Chia Wang, Moira Burke, and Robert Kraut. 2016. Modeling self-disclosure in social networking sites. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing, pages 74–85. find
  40. Haiqin Yang and Jianping Shen. 2021. Emotion dynamics modeling via bert. find
  41. Ran Zhao, Tanmay Sinha, Alan Black, and Justine Cassell. 2016. Automatic recognition of conversational strategies in the service of a socially-aware dialog system. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 381–392, Los Angeles. Associati… find
  42. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2025. A survey of l… find our card
  43. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. find our card
  44. Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020. The design and implementation of xiaoice, an empathetic social chatbot. Comput. Linguist., 46(1):53–93. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/villata-2026-humanica-benchmark-resource-detection-users/