Elamur Research⌕
☰
home / papers / 2024 / DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures

DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures

Paper · 2024 · Agrima Seth, Sanchit Ahuja, Kalika Bali et al.

Summary

LLMs learn mostly from web text, and the web underrepresents many communities. As a result, it is unclear how well these models know everyday cultural items from outside dominant cultures. The authors study this for state-level subcultures within India, arguing that culture varies inside a country and not only between countries. Instead of scraping or translating data, they used participatory methods. First, a survey asked people to self-identify their cultural identity and name artifacts that matter in it. Second, a game with a purpose modeled on Taboo had players describe those artifacts through free-form clues for others to guess. Volunteers added more items after playing. The resulting resource, DOSA, contains 615 artifact names with descriptions written by community members, gathered from 260 participants across 19 subcultures. The authors fed the clues to GPT-4, LLaMA2, PaLM 2 and Falcon, giving each model up to two guesses. Accuracy varied sharply across models and regions. The work offers a template for building community-sourced cultural benchmarks in non-Western settings.

Key points

  • DOSA is presented as the first community-generated dataset of social artifacts: 615 items from 260 participants representing 19 Indian state-based subcultures.
  • Data collection combined two participatory methods:
    • a free-text survey, manually cleaned to merge spelling variants (e.g., Bandhani/Bandhej);
    • a GWAP adapted from Taboo, where clues are written in natural speech rather than fixed templates.
  • The benchmarked models are GPT-4, LLaMA2, PaLM 2 and Falcon. Results are reported as overall accuracy, accuracy@GUESS1 and accuracy@GUESS2, separately for the original artifacts and the expanded (game-collected) artifacts.
  • The commercial models generally scored higher than the open ones. For example, on expanded artifacts GPT-4 reached 0.70 for Kerala, while Falcon was at or near 0 for most states.
  • No unique expanded artifacts were collected from Delhi participants, so Delhi has no expanded-set scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of MichiganAnn Arbor · United States70 works · 13,693 citations🏢Microsoft ResearchBengaluru · India13 works · 3,602 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSafety Research · 543 works

Cite

Cite as Agrima Seth, Sanchit Ahuja, Kalika Bali and Sunayana Sitaram. 2024. DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{seth2024dosa,
  title = {{DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures}},
  author = {Agrima Seth, Sanchit Ahuja, Kalika Bali and Sunayana Sitaram},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.2139/ssrn.4756716},
  url = {https://doi.org/10.2139/ssrn.4756716}
}

References

44 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. NITI Aayog. 2023. poverty index. find
  2. National multidimensional Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Large language models associate muslims with violence. Nature Machine Intelligence, 3(6):461–463. find
  3. Anurag Acharya, Kartik Talamadupula, and Mark A Finlayson. 2020. Towards an atlas of cultural commonsense for machine reasoning. arXiv preprint arXiv:2009.05664. — [arXiv:2009.05664](https://arxiv.org/abs/2009.05664) find
  4. Umair Z Ahmed, Kalika Bali, Monojit Choudhury, and VB Sowmya. 2011. Challenges in designing input method editors for indian lan-guages: The role of word-origin and context. In Proceedings of the Workshop on Advances in Text Input Methods (WTIM 2011), pages 1–9. find
  5. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-t… find
  6. Harshita Diddee, Kalika Bali, Monojit Choudhury, and Namrata Mukhija. 2022. The six conundrums of building and deploying language technologies for social good. In ACM SIG-CAS/SIGCHI Conference on Computing and Sustainable Societies (COMPASS), pages 12–19. find
  7. Xiao Fang, Shangkun Che, Minjia Mao, Hongzhe Zhang, Ming Zhao, and Xiaohang Zhao. 2023. Bias of ai-generated content: An examination of news produced by large language models. arXiv preprint arXiv:2309.09825. — [arXiv:2309.09825](https://arxiv.org/abs/2309.09825) find
  8. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403.
  9. Ronald Fischer, Markus Luczak-Roesch, and Johannes A Karl. 2023. What does chatgpt return about human values? exploring value bias in chatgpt using a descriptive value theory. arXiv preprint arXiv:2304.03612. — [arXiv:2304.03612](https://arxiv.org/abs/2304.03612) find
  10. Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augenstein. 2022. Probing pre-trained language models for cross-cultural differences in values. arXiv preprint arXiv:2203.13722.
  11. Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013. Moral foundations theory: The pragmatic validity of moral pluralism. In Advances in experimental social psychology, volume 47, pages 55–130. Elsevier. find
  12. Mohammad Atari, Mona J Xue, Peter S Park, Damián Blasi, and Joseph Henrich. 2023. Which humans? find
  13. Agathe Balayn, Gaole He, Andrea Hu, Jie Yang, and Ujwal Gadiraju. 2022. Ready player one! eliciting diverse knowledge using a configurable game. In Proceedings of the ACM Web Conference 2022, pages 1709–1719. find
  14. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  15. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  16. Johana Cabrera, M Soledad Loyola, Irene Magaña, and Rodrigo Rojas. 2023. Ethical dilemmas, mental health, artificial intelligence, and llm-based chatbots. In International Work-Conference on Bioinformatics and Biomedical Engineering, pages 313–326. Springer. find
  17. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. arXiv preprint arXiv:2303.17466.
  18. Harrison Chase. 2022. LangChain. find
  19. Kanika Gupta, Monojit Choudhury, and Kalika Bali. 2012. Mining hindi-english transliteration pairs from online hindi lyrics. In LREC, pages 2459–2465. find
  20. Katharina Hämmerl, Björn Deiseroth, Patrick Schramowski, Jindřich Libovickỳ, Constantin A Rothkopf, Alexander Fraser, and Kristian Kersting. 2022. Speaking multiple languages affects the moral bias of language models. arXiv preprint arXiv:2211.07733. find
  21. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, et al. 2022. Challenges and strategies in cross-cultural nlp. arXiv preprint arXiv:2203.10020.
  22. Geert Hofstede. 2011. Dimensionalizing cultures: The hofstede model in context. Online readings in psychology and culture, 2(1):8. find
  23. R. Inglehart, C. Haerpfer, A. Moreno, C. Welzel, K. Kizilova, J. Diez-Medrano, M. Lagos, E. Ponarin P. Norris, and B. Puranen et al. (eds.). 2018. World values survey: Round six - country-pooled datafile. madrid, spain & vienna, austria: Jd systems institute & wvsa secretariat. doi.org/10.14281/1824… — [doi:10.14281/18241.8](https://doi.org/10.14281/18241.8) find
  24. Eunkyung Jo, Daniel A Epstein, Hyunhoon Jung, and Young-Ho Kim. 2023. Understanding the benefits and challenges of deploying conversational ai leveraging large language models for public health intervention. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–16.… find
  25. David Jurgens, Agrima Seth, Jackson Sargent, Athena Aghighi, and Michael Geraci. 2023. Your spouse needs professional help: Determining the contextual appropriateness of messages through modeling social relationships. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin… find
  26. Agrima Seth, Jiyin Cao, Xiaolin Shi, Ron Dotsch, Yozen Liu, and Maarten W Bos. 2023. Cultural differences in friendship network behaviors: A snapchat case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–14. find
  27. Hadas Kotek, Rikker Dockum, and David Q Sun. 2023. Gender bias and stereotypes in large language models. arXiv preprint arXiv:2308.14921. — [arXiv:2308.14921](https://arxiv.org/abs/2308.14921) find
  28. Devinder Pal Singh and Manoj K Sharma. 2009. Unfolding the indian cultural mosaic: a cross-cultural study of four regional cultures. International Journal of Indian Culture and Business Management, 2(3):247–267. find
  29. Virginie Mamadouh. 2020. Writing the world in 301 languages: A political geography of the online encyclopedia wikipedia. Handbook of the Changing World Language Map, pages 3801–3824. find
  30. Janet Stephenson. 2023. Culture and Sustainability: Exploring Stability and Transformation with the Cultures Framework. Springer Nature. find
  31. Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2023. More human than human: Measuring chatgpt political bias. Public Choice, pages 1–21. find
  32. Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. 2021. Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503. find
  33. Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907–1917. find
  34. Kumar Tanmay, Aditi Khandelwal, Utkarsh Agarwal, and Monojit Choudhury. 2023. Exploring large language models’ cognitive moral development through defining issues test. arXiv preprint arXiv:2309.13356. — [arXiv:2309.13356](https://arxiv.org/abs/2309.13356) find
  35. OpenAI. 2023. Gpt-4 technical report.
  36. Vishesh Thakur. 2023. Unveiling gender bias in terms of profession across llms: Analyzing and addressing sociological implications. arXiv preprint arXiv:2307.09162. — [arXiv:2307.09162](https://arxiv.org/abs/2307.09162) find
  37. Shramay Palta and Rachel Rudinger. 2023. Fork: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9952–9962.
  38. Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. 2022. Cultural incongruencies in artificial intelligence. arXiv preprint arXiv:2211.13069. — [arXiv:2211.13069](https://arxiv.org/abs/2211.13069) find
  39. Rida Qadri, Renee Shelby, Cynthia L Bennett, and Emily Denton. 2023. Ai’s regimes of representation: A community-centered study of text-to-image models in south asia. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 506–517. find
  40. Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality traits in large language models. arXiv preprint arXiv:2307.00184.
  41. Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. 2022. Large pre-trained language models contain human-like biases of what is right Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra,…
  42. Luis Von Ahn, Mihir Kedia, and Manuel Blum. 2006. Verbosity: a game for collecting common-sense facts. In Proceedings of the SIGCHI conference on Human Factors in computing systems, pages 75–78. find
  43. Han Wang, Ming Shan Hee, Md Rabiul Awal, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. 2023. Evaluating gpt-3 generated explanations for hateful content moderation. arXiv preprint arXiv:2305.17680. — [arXiv:2305.17680](https://arxiv.org/abs/2305.17680) find
  44. Jing Yao, Xiaoyuan Yi, Xiting Wang, Jindong Wang, and Xing Xie. 2023. From instructions to intrinsic human values–a survey of alignment goals for big models. arXiv preprint arXiv:2308.12014. — [arXiv:2308.12014](https://arxiv.org/abs/2308.12014) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seth-2024-dosa-dataset-social-artifacts-different/