Elamur Research⌕
☰
home / papers / 2026 / Modeling Human Perspectives with Socio-Demographic Representations

Modeling Human Perspectives with Socio-Demographic Representations

Paper · 2026 · Leixin Zhang and Cagri Coltekin

Summary

Annotators of subjective content such as hate speech often disagree for legitimate reasons, and these differences may relate to their backgrounds. Earlier studies usually examined one demographic attribute at a time and sometimes found no benefit. Zhang and Çöltekin propose Socio-Contrastive Learning. Each annotator's multi-hot socio-demographic profile passes through a trainable projection layer and is refined with a contrastive objective. It is then joined with text embeddings to predict that annotator's own label under cross-entropy loss.

The contrastive component pulls together annotators who labeled a text identically and separates those who disagreed. Batches are grouped by text ID, and pairs from different texts are masked. This handles varying annotator counts and unanimous items.

Experiments use a hate speech corpus (Kennedy et al., 2020) and a toxicity corpus (Kumar et al., 2021). Baselines include models without demographics (majority-vote and multi-task) and simple fusion via multi-hot vectors or encoder-based demographic embeddings. The proposed method predicts labels best. Its learned space also shows which attributes align with perspectives, indicating that combined demographic profiles do help explain disagreement.

Key points

  • Each dataset records annotators' schooling level, political leaning, age, gender, racial background and sexual orientation, while income appears only in the toxicity set. Hate speech has 22,942 train and 10,359 test annotations (70/30 split). Toxic has 25,556 train and 16,151 test annotations (60/40 split). Splits are made by unique text ID.
  • The paper compares five setups: Simple Model, Multi-Task Model, Multi-Hot Encoding, Socio-Demographic Embedding, and Socio-Contrastive Learning. The new method beats all of them, including concatenation-based fusion.
  • A homophily ratio compares neighbor agreement among k = 50 nearest neighbors with chance agreement. Education ranks highest on both datasets, and ideology, income and age also exceed 2. The transgender attribute falls below chance, likely because such annotators make up under 5% of the pool.
  • Per-group F1 scores look similar across gender groups (hate speech: female 0.735, male 0.711, non-binary 0.711). The authors warn these are unreliable because non-binary annotators labeled only 64 hate speech and 86 toxicity test items.
  • The findings contradict Orlikowski et al. (2023), who reported no gain from demographics on the same toxicity data. Code is released on GitHub (Socio_Contrastive_Learning).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany78 works · 41,280 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksGeneral Social Sciences · 254 works

Cite

Cite as Leixin Zhang and Cagri Coltekin. 2026. Modeling Human Perspectives with Socio-Demographic Representations. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2026

@inproceedings{zhang2026modeling,
  title = {{Modeling Human Perspectives with Socio-Demographic Representations}},
  author = {Leixin Zhang and Cagri Coltekin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2026},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2026}
}

References

30 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6860–6868.
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  3. Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. 2024. Modular pluralism: Pluralistic alignment via multi-LLM collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4151–4171, Miami, F…
  4. Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–19. find
  5. Jing Huang and Diyi Yang. 2023. Culturally aware natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591–7609.
  6. Kamil Kanclerz, Alicja Figas, Marcin Gruza, Tomasz Kajdanowicz, Jan Kocoń, Daria Puchalska, and Przemyslaw Kazienko. 2021. Controversy and conformity: from generalized to personalized aggressiveness detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistic… find
  7. Chris J Kennedy, Geoff Bacon, Alexander Sahn, and Claudia von Vacano. 2020. Constructing interval variables via faceted rasch measurement and multi-task deep learning: a hate speech application. arXiv preprint arXiv:2009.10277. — [arXiv:2009.10277](https://arxiv.org/abs/2009.10277) find
  8. Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  9. Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), pages 299–318. find
  10. Savannah Larimore, Ian Kennedy, Breon Haskett, and Alina Arseniev-Koehler. 2021. Reconsidering annotator disagreement about racist language: Noise or signal? In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, pages 81–90. Association for Computational… find
  11. Elisa Leonardelli, Alexandra Uma, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, and Massimo Poesio. 2023. Semeval-2023 task 11: Learning with disagreements (LeWiDi). arXiv preprint arXiv:2304.14803.
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  13. Leland McInnes, John Healy, and James Melville. 2018. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426.
  14. Miller McPherson, Lynn Smith-Lovin, and James M Cook. 2001. Birds of a feather: Homophily in social networks. Annual review of sociology, 27(1):415– 444. find
  15. Negar Mokhberian, Myrl Marmarelis, Frederic Hopp, Valerio Basile, Fred Morstatter, and Kristina Lerman. 2024. Capturing perspectives of crowdsourced annotators in subjective learning tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingu… find
  16. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110.
  17. Matthias Orlikowski, Paul Röttger, Philipp Cimiano, and Dirk Hovy. 2023. The ecological fallacy in annotation: Modeling human label variation goes beyond sociodemographics. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1017… find
  18. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. PyTorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  19. Dong Pham, Xanh Ho, Quang Thuy Ha, and Akiko Aizawa. 2023. Solving label variation in scientific information extraction via multi-task learning. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pages 243– 256, Hong Kong, China. Association for Computationa… find
  20. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  21. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. find
  22. Yuxia Wang, Shimin Tao, Ning Xie, Hao Yang, Timothy Baldwin, and Karin Verspoor. 2023. Collective human opinions in semantic textual similarity. Transactions of the Association for Computational Linguistics, 11:997–1013. find
  23. Zeerak Waseem. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138– 142, Austin, Texas. Association for Computational Linguistics.
  24. Leixin Zhang. 2025. Proposal: From one-fit-all to perspective aware modeling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 1016–1025. find
  25. Leixin Zhang, David Burian, Vojtěch John, and Ondřej Bojar. 2024. Unveiling semantic information in sentence embeddings. In Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024, pages 39–47, Torino, Italia. ELRA and ICCL. find
  26. Leixin Zhang and Çağrı Çöltekin. 2024. Tübingen-CL at SemEval-2024 task 1: Ensemble learning for semantic relatedness estimation. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1019–1025, Mexico City, Mexico. Association for Computational Linguistics. find
  27. Leixin Zhang and Çağrı Çöltekin. 2026. Quantifying and predicting disagreement in graded human ratings. In Proceedings of the Fifth Workshop on Perspectivist Approaches to NLP (NLPerspectives) at Language Resources and Evaluation Conference (LREC) 2026. ELRA.
  28. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  29. Yisi Sang and Jeffrey Stanton. 2022. The origin and value of disagreement among data labelers: A case study of individual differences in hate speech annotation. In International Conference on Information, pages 425–444. Springer. find
  30. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-modeling-human-perspectives-socio-demographic/