Elamur Research⌕
☰
home / papers / 2026 / Dynamic Emotion and Personality Profiling for Multimodal Deception Detection

Dynamic Emotion and Personality Profiling for Multimodal Deception Detection

Paper · 2026 · Li Zheng, Yanyi Luo, Hao Fei et al.

Summary

Emotion and personality cues help in multimodal deception detection. However, the main existing resource, MDPE, labels these traits only once per participant. This ignores how one person's feelings and apparent traits change from one answer to the next. The authors re-label MDPE at the level of individual samples to create DDEP, a corpus for joint detection of deception, emotion (multi-label) and personality (single-label).

The labels come from a multi-model multi-prompt pipeline:

  • Several LLMs, each queried with varied prompts, propose labels.
  • The labels are aggregated by voting and scored for consistency and uncertainty.
  • Items that fail the quality check are re-labelled by multimodal LLMs, and any remaining failures go to human annotators.

The paper also presents Rel-DDEP, a fusion model with three main parts. It maps each modality into a Gaussian space to estimate uncertainty and weights modalities by reliability. An alignment module ties uncertainty to actual errors, and a sorting constraint module ensures uncertainty reflects modality importance. Rel-DDEP surpasses state-of-the-art baselines on all three tasks on both MDPE and DDEP. This suggests that per-sample affect and trait labels make deceptive behaviour much easier to separate.

Key points

  • Pilot study: 200 samples were drawn from 20 subjects, 10 per subject. The number correctly classified as deceptive or honest rose with richer labels:
    • fixed subject-level labels: 32
    • dynamic single emotion and personality labels: 85
    • multi-label emotion plus single personality label: 141
  • F1 gains over the best baselines: +2.53% for deception, +2.66% for emotion and +9.30% for personality.
  • Ablation: removing reliability-weighted fusion causes the largest performance drop. Equal-weight fusion performs worst, and among single-modality-dominant variants, text-dominant works best.
  • Joint vs. single-task: training the three tasks jointly outperforms separate single-task models on every task.
  • Limitation: a basic shared encoder restricts task-specific interactions.
  • Publication details: ACL 2026 (Long Papers), pp. 3930–3940; Wuhan University and NUS; funded by Kuaishou.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore207 works · 39,576 citations🎓Wuhan UniversityWuhan · China47 works · 1,246 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,954 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 worksSociology and Political Science · 626 worksSocial Psychology · 330 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksVision & language · 811 works

Cite

Cite as Li Zheng, Yanyi Luo, Hao Fei, Yuzhe Ding, Yujie Huang, Fei Li, Chong Teng and Donghong Ji. 2026. Dynamic Emotion and Personality Profiling for Multimodal Deception Detection. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.181

@inproceedings{zheng2026dynamic,
  title = {{Dynamic Emotion and Personality Profiling for Multimodal Deception Detection}},
  author = {Li Zheng, Yanyi Luo, Hao Fei, Yuzhe Ding, Yujie Huang, Fei Li, Chong Teng and Donghong Ji},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.181},
  url = {https://doi.org/10.18653/v1/2026.acl-long.181}
}

References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamed Abouelenien, Verónica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo. 2016. Detecting deceptive behavior via integration of discriminative features from multiple modalities. IEEE Transactions on Information Forensics and Security, 12(5):1042– 1055. find
  2. Jalal S Alowibdi, Ugo A Buy, S Yu Philip, and Leon Stenneth. 2014. Detecting deception in online social networks. In 2014 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2014), pages 383–390. IEEE. find
  3. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations.
  4. Advances in neural information processing systems, 33:12449–12460. find
  5. Cong Cai, Shan Liang, Xuefei Liu, Kang Zhu, Zhengqi Wen, Jianhua Tao, Heng Xie, Jizhou Cui, Yiming Ma, Zhenhua Cheng, et al. 2024. Mdpe: A multimodal deception dataset with personality and emotional characteristics. Advances in Neural Information Processing Systems. find
  6. Lile Cai, Xun Xu, Jun Hao Liew, and Chuan Sheng Foo. 2021. Revisiting superpixels for active learning in semantic segmentation with realistic annotation costs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10988–10997. find
  7. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518. find
  8. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  9. Alyt Damstra, Hajo G Boomgaarden, Elena Broda, Elina Lindgren, Jesper Strömbäck, Yariv Tsfati, and Rens Vliegenthart. 2021. What does fake look like? a review of the literature on intentional deception in the news and on social media. Journalism Studies, 22(14):1947–1963. find
  10. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Shafiq Joty, Boyang Li, and Lidong Bing. 2022. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212.10450.
  11. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
  12. Joseph P Gaspar and Maurice E Schweitzer. 2013. The emotion deception model: A review of deception in negotiation and the role of emotion in deception. Negotiation and Conflict Management Research, 6(3):160–179. find
  13. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120.
  14. Viresh Gupta, Mohit Agarwal, Manik Arora, Tanmoy Chakraborty, Richa Singh, and Mayank Vatsa. 2019. Bag-of-lies: A multimodal dataset for deception detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0. find
  15. Christopher N Gutierrez, Saurabh Bagchi, H Mohammed, and Jeff Avery. 2015. Modeling deception in information security as a hypergame–a primer. In Proceedings of the 16th Annual Information Security Symposium, page 41. CERIAS-Purdue University. find
  16. Xiao Han, Nizar Kheir, and Davide Balzarotti. 2018. Deception techniques in computer security: A research perspective. ACM Computing Surveys (CSUR), 51(4):1–36. find
  17. Jinglue Hang, Xiangbo Lin, Tianqiang Zhu, Xuan-heng Li, Rina Wu, Xiaohong Ma, and Yi Sun. 2024. Dexfuncgrasp: A robotic dexterous functional grasp dataset constructed from a cost-effective real-simulation annotation system. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38,… find
  18. Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854.
  19. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460.
  20. Ronald T Kneusel and Michael C Mozer. 2017. Improving human-machine cooperative visual search with soft highlighting. ACM Transactions on Applied Perception (TAP), 15(1):1–21. find
  21. Srijan Kumar, Chongyang Bai, VS Subrahmanian, and Jure Leskovec. 2021. Deception detection in group video conversations using dynamic interaction networks. In Proceedings of the international AAAI conference on web and social media, volume 15, pages 339–350. find
  22. Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency, pages 29–38. find
  23. Sarah I Levitan, Guzhen An, Mandi Wang, Gideon Mendels, Julia Hirschberg, Michelle Levine, and Andrew Rosenberg. 2015. Cross-cultural production and detection of deception from speech. In Proceedings of the 2015 ACM on workshop on multimodal deception detection, pages 1–8. find
  24. Timothy J Luke, Erik Mac Giolla, Amina Memon, Sara Landström, Pär Anders Granhag, and Saul Kassin. 2023. What have we learned about cues to deception? a survey of expert opinions. Psychology, Crime & Law, pages 1–20. find
  25. Leena Mathur and Maja J Matarić. 2020. Introducing representations of facial affect in automated multimodal deception detection. In Proceedings of the 2020 international conference on multimodal interaction, pages 305–314. find
  26. Verónica Pérez-Rosas, Mohamed Abouelenien, Rada Mihalcea, Yao Xiao, CJ Linton, and Mihai Burzo. 2015. Verbal and nonverbal clues for real-life deception detection. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2336–2346. find
  27. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. Pm…
  28. Nathan Vance, Jeremy Speth, Siamul Khan, Adam Czajka, Kevin W Bowyer, Diane Wright, and Patrick Flynn. 2022. Deception detection and remote physiological monitoring: A dataset and baseline experimental results. IEEE Transactions on Biometrics, Behavior, and Identity Science, 4(4):522–532. find
  29. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2108.13487.
  30. Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. Human-llm collaborative annotation through effective verification of llm labels. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–21. find
  31. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
  32. Li Zheng, Boyu Chen, Hao Fei, Fei Li, Shengqiong Wu, Lizi Liao, Donghong Ji, and Chong Teng. 2024a. Self-adaptive fine-grained multi-modal data augmentation for semi-supervised muti-modal coreference resolution. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8576–8585. find
  33. Li Zheng, Hao Fei, Ting Dai, Zuquan Peng, Fei Li, Huisheng Ma, Chong Teng, and Donghong Ji. 2025a. Multi-granular multimodal clue fusion for meme understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26057– 26065. find
  34. Li Zheng, Hao Fei, Fei Li, Bobo Li, Lizi Liao, Donghong Ji, and Chong Teng. 2024b. Reverse multi-choice dialogue commonsense inference with graph-of-thought. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19688–19696. find
  35. Li Zheng, Donghong Ji, Fei Li, Hao Fei, Shengqiong Wu, Jingye Li, Bobo Li, and Chong Teng. 2025b. Ec-qed: emotion-cause quadruple extraction in dialogs. IEEE Transactions on Audio, Speech and Language Processing. find
  36. Li Zheng, Tengyue Song, Yuzhe Ding, Xiaorui Wu, Fei Li, Dongdong Xie, Jinbo Li, Chong Teng, and Donghong Ji. 2025c. Improving emotion and intent understanding in multimodal conversations with progressive interaction. IEEE Transactions on Affective Computing. find
  37. Li Zheng, Sihang Wang, Hao Fei, Zuquan Peng, Fei Li, Jianming Fu, Chong Teng, and Donghong Ji. 2025d. Enhancing hyperbole and metaphor detection with their bidirectional dynamic interaction and emotion knowledge. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguist… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-dynamic-emotion-personality-profiling-multimodal/