Elamur Research⌕
☰
home / papers / 2026 / JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human risky health behavior Content in Jirai Community

JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human risky health behavior Content in Jirai Community

Paper · 2026 · Yunze Xiao, Tingyu He, Lionel Z. Wang et al.

Summary

Online content about risky health behaviors (RHB) is hard to moderate automatically. Prior work mostly targets one behavior at a time, relies heavily on English data, and pays little attention to community context and coded vocabulary. The authors address this through the Jirai ("landmine") subculture, a Japan-born social media phenomenon that spread to Chinese platforms and mixes drug misuse, eating disorders, and self-harm. They built JiraiBench, a cross-lingual benchmark of over 15,000 Chinese and Japanese posts. Trained annotators, guided by domain experts, labeled each post for whether it contains RHB, which behavior type it involves, and whether it is a first-person or third-party account. They tested four state-of-the-art LLMs with prompts in Chinese, Japanese, and English, and fine-tuned Qwen2.5 7B on Chinese data. Unexpectedly, Japanese prompts handled Chinese posts better than Chinese prompts did. The authors read this as evidence that cultural origin can outweigh linguistic closeness. The findings argue for culturally grounded, multilingual moderation resources that protect vulnerable users across language borders.

Key points

  • Data: JiraiBench contains more than 15,000 annotated posts, with Chinese content from Weibo and Japanese content from X (Twitter), collected via keyword search.
  • Annotation: Posts are classified by behavior type (drug overdose/misuse, self-harm/NSSI, eating disorders). A further label marks 0 for non-concerning, 1 for first-person, and 2 for third-party. Six trained annotators did the labeling, supported by domain experts.
  • Evaluation: Four LLMs were tested under two baseline configurations, using Chinese, Japanese, and English instruction prompts on both languages.
  • Prompt-language finding: For Chinese content, Japanese prompts consistently beat Chinese ones, which the authors attribute to the Japanese roots of the Jirai phenomenon.
  • Fine-tuning and ethics: Fine-tuning Qwen2.5 7B on Chinese data alone also improved detection on Japanese posts. Data are anonymized (PII removed) and released only under controlled academic access.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations🎓University of WashingtonSeattle · United States402 works · 212,762 citations🎓The University of TokyoTokyo · Japan96 works · 9,600 citations🎓Hong Kong Polytechnic UniversityHong Kong · China106 works · 5,190 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksSocial Psychology · 330 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Yunze Xiao, Tingyu He, Lionel Z. Wang, Yiming Ma, Xingyu Song, Xiaohang Xu, Mona T. Diab, Irene Li and Ka Chung Ng. 2026. JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human risky health behavior Content in Jirai Community. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.23

@inproceedings{xiao2026jiraibench,
  title = {{JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human risky health behavior Content in Jirai Community}},
  author = {Yunze Xiao, Tingyu He, Lionel Z. Wang, Yiming Ma, Xingyu Song, Xiaohang Xu, Mona T. Diab, Irene Li and Ka Chung Ng},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.23},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.23}
}

References

45 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roy F Baumeister and Steven J Scher. 1988. Self-defeating behavior patterns among normal individuals: review and analysis of common self-destructive tendencies. Psychological bulletin, 104(1):3. find
  2. Yuchen Cao, Jianglai Dai, Zhongyan Wang, Yeyubei Zhang, Xiaorui Shen, Yunchong Liu, and Yexin Tian. 2025. Machine learning approaches for depression detection on social media: A systematic review of biases and methodological challenges. Journal of Behavioral Data Science, 5(1):1–36. find
  3. William W Chan, Ellen E Fitzsimmons-Craft, Arielle C Smith, Marie-Laure Firebaugh, Lauren A Fowler, Bianca DePietro, Naira Topooco, Denise E Wilfley, C Barr Taylor, and Nicholas C Jacobson. 2022. The challenges in designing a prevention chatbot for eating disorders: observational study. JMIR Formati… find
  4. Stevie Chancellor and Munmun De Choudhury. 2020. Methods in predictive techniques for mental health status on social media: A critical review. npj Digital Medicine, 3:43. find
  5. Stevie Chancellor, Zhiyuan Lin, and Munmun De Choudhury. 2016a. " this post will just get taken down" characterizing removed pro-eating disorder social media content. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 1157–1162. find
  6. Stevie Chancellor, Zhiyuan Lin, Erica L Goodman, Stephanie Zerwas, and Munmun De Choudhury. 2016b. Quantifying and predicting mental illness severity in online pro-eating disorder communities. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing, pages… find
  7. Munmun De Choudhury and Emre Kıcıman. 2017. The language of social support in social media and its effect on suicidal ideation risk. Proceedings of the ... International AAAI Conference on Weblogs and Social Media. International AAAI Conference on Weblogs and Social Media, 2017:32–41. find
  8. Charlotte Cliffe, Azadeh Seyedsalehi, Katerina Var-davoulia, André Bittar, Sumithra Velupillai, Hitesh Shetty, Ulrike Schmidt, and Rina Dutta. 2021. Using natural language processing to extract self-harm and suicidality data from a clinical sample of patients with eating disorders: a retrospective c… find
  9. Glen Coppersmith, Mark Dredze, and Craig Harman. 2014. Quantifying mental health signals in twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 51–60.
  10. Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting depression via social media. Proceedings of the International AAAI Conference on Web and Social Media, 7:128–137. find
  11. DeepSeek-AI. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  12. Yujie Fan, Yiming Zhang, Yanfang Ye, Xin Li, and Wanhong Zheng. 2017. Social media for opioid addiction epidemiology: Automatic detection of opioid addicts from twitter and case studies. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 1259–1267. find
  13. Robert W Firestone and Richard H Seiden. 1990. Suicide and the continuum of self-destructive behavior. Journal of American College Health, 38(5):207–213. find
  14. Andrew Fisher, Matthew Maclaren Young, Doris Payer, Karen Pacheco, Chad Dubeau, and Vijay Mago. 2023. Automating detection of drug-related harms on social media: machine learning framework. Journal of medical internet research, 25:e43630. find
  15. Sharath Chandra Guntuku, Jami F Young, Daniel Romer, Lyle Ungar, and Russell Ramsay. 2023. Exploring the behavior of users with attention deficit hyperactivity disorder on twitter: Observational study. Journal of Medical Internet Research, 25:e43439. find
  16. Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2022. Mental-BERT: Publicly available pretrained language models for mental healthcare. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7184–7190, Marseille, France. European Langu… find
  17. Zhengping Jiang, Sarah Ita Levitan, Jonathan Zomick, and Julia Hirschberg. 2020. Detection of mental health from Reddit via deep contextualized representations. In Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, pages 147–156, Online. Association for Co… find
  18. Matthew Matero, Akash Idnani, Youngseo Son, Salvatore Giorgi, Huy Vu, Mohammad Zamani, Parth Limbachiya, Sharath Chandra Guntuku, and H. Andrew Schwartz. 2019. Suicide risk assessment with multi-level dual-context language and bert. In Proceedings of the Sixth Workshop on Computational Linguistics a… find
  19. Augusto R. Mendes and Helena Caseli. 2024. Identifying fine-grained depression signs in social media posts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 8594–8604, Torino, Italia. ELRA and ICCL. find
  20. Meta. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  21. Markus Moessner, Johannes Feldhege, Markus Wolf, and Stephanie Bauer. 2018. Analyzing big data in social media: Text and network analyses of an eating disorder forum. International Journal of Eating Disorders, 51(7):656–667. find
  22. Ankita Murarka, Aravind Sengadu Suresh, Aayushi Lalwani, Kajal Bajaj, Muskan Garg, and Sakshi Srivastava. 2021. Detection and classification of mental illnesses on social media using roberta. In Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis, pages 59–6… find
  23. Tareq Nasralah, Omar El-Gayar, and Yong Wang. 2020. Social media text mining framework for drug abuse: development and validation study with an opioid crisis case analysis. Journal of medical Internet research, 22(8):e18350. find
  24. Ka Chung Ng, Ping Fan Ke, Mike KP So, and Kar Yan Tam. 2023. Augmenting fake content detection in online platforms: A domain adaptive transfer learning via adversarial training approach. Production and Operations Management, 32(7):2101–2122. find
  25. Michael J. Paul and Mark Dredze. 2011. You are what you tweet: Analyzing twitter for public health. In Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media, pages 265–272. find
  26. Nhathai Phan, Soon Ae Chun, Manasi Bhole, and James Geller. 2017. Enabling real-time drug abuse detection in tweets. In 2017 IEEE 33rd international conference on data engineering (ICDE), pages 1510– 1514. IEEE. find
  27. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  28. Waleed Ragheb, Jerome Aze, Sandra Bringay, and Maximilien Servajean. 2021. Negatively correlated noisy learners for at-risk user detection on social networks: A study on depression, anorexia, self-harm, and suicide. IEEE Transactions on Knowledge and Data Engineering, 35(1):770–783. find
  29. Sebastian Scherr, Florian Arendt, Thomas Frissen, and José Oramas M. 2020. Detecting intentional self-harm on instagram: development, testing, and validation of an automatic image-recognition algorithm to discover cutting-related posts. Social science computer review, 38(6):673–685. find
  30. José Sixto-García and Ana Duarte-Melo. 2020. Self-destructive content in university teaching: new challenge in the digital competence of educators. Communication & Society, 33(3):187–199. find
  31. Elizabeth C. Stade, Shannon Wiltsey Stirman, Lyle H. Ungar, Cody L. Boland, H. Andrew Schwartz, David B. Yaden, João Sedoc, Robert J. DeRubeis, Robb Willer, and Johannes C. Eichstaedt. 2024. Large language models could change the future of behavioral healthcare: a proposal for responsible developmen… find
  32. Blanca Tébar and Anandha Gopalan. 2021. Early detection of eating disorders using´ social media. In 2021 IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), pages 193–198. IEEE. find
  33. Daniela Teodorescu, Tiffany Cheng, Alona Fyshe, and Saif Mohammad. 2023. Language and mental health: Measures of emotion dynamics from text as linguistic biosocial markers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3117–3133, Singapore. Associat… find
  34. Sho Tsugawa, Yusuke Kikuchi, Fumio Kishino, Kenji Nakajima, Yasuhide Itoh, and Hiroyuki Ohsaki. 2015. Recognizing depression from twitter activity. Proceedings of the ACM CHI Conference on Human Factors in Computing Systems, pages 3187–3196. find
  35. Qamar Un Nisa and Rafi Muhammad. 2021. Towards transfer learning using bert for early detection of self-harm of social media users. Proceedings of the Working Notes of CLEF, pages 21–24. find
  36. Bessel A Van der Kolk, J Christopher Perry, and Judith Lewis Herman. 1991. Childhood origins of self-destructive behavior. American journal of Psychiatry, 148(12):1665–1671. find
  37. Tao Wang, Markus Brede, Antonella Ianni, and Emmanouil Mentzakis. 2017a. Detecting and characterizing eating-disorder communities on social media. In Proceedings of the Tenth ACM International conference on web search and data mining, pages 91–100. find
  38. Yilin Wang, Jiliang Tang, Jundong Li, Baoxin Li, Yali Wan, Clayton Mellina, Neil O’Hare, and Yi Chang. 2017b. Understanding and discovering deliberate self-harm content in social media. In Proceedings of the 26th international conference on World Wide Web, pages 93–102. find
  39. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, T…
  40. Austin P. Wright, Christopher M. Jones, Duen Horng Chau, R. Matthew Gladden, and Steven A. Sumner. 2021. Detection of emerging drugs involved in overdose via diachronic word embeddings of substances discussed on social media. Journal of Biomedical Informatics, 119:103824. find
  41. Yunze Xiao, Yujia Hu, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. 2024. ToxiCloakCN: Evaluating robustness of offensive language detection in Chinese with cloaking perturbations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6012–6025, Miami, Florida, U… find
  42. Shweta Yadav, Jainish Chauhan, Joy Prakash Sain, Krishnaprasad Thirunarayan, Amit Sheth, and Jeremiah Schumm. 2020. Identifying depressive symptoms from tweets: Figurative language enabled multitask learning framework. In Proceedings of the 28th International Conference on Computational Linguistics,… find
  43. Chenghao Yang, Tuhin Chakrabarty, Karli Hochstat-ter, Melissa Slavin, Nabila El-Bassel, and Smaranda Muresan. 2024. Identifying self-disclosures of use, misuse and addiction in community-based social media posts. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2507–25… find
  44. Jingfeng Yang, Haoming Jiang, Qingyu Yin, Danqing Zhang, Bing Yin, and Diyi Yang. 2022. SEQZERO: Few-shot compositional semantic parsing with sequential prompts and zero-shot models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 49–60, Seattle, United States. Associ…
  45. Dongsong Zhang, Lina Zhou, Jie Tao, Tingshao Zhu, and Guodong Gao. 2024. Ketch: a knowledge-enhanced transformer-based approach to suicidal ideation detection from social media content. Information Systems Research. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2026-jiraibench-bilingual-benchmark-evaluating-large/