Elamur Research⌕
☰
home / papers / 2022 / EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English

EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English

Paper · 2022 · Weicheng Ma, Samiha Datta, Lili Wang et al.

Summary

Most NLP work on culture treats it at a coarse level, grouping people by shared official language or by broad "Western/Eastern" splits. This hides the variation that exists among speakers of a single language. The authors build EnCBP, an English dataset for predicting the cultural background of a text. It has two levels: a country subset covering five English-speaking nations and a district subset covering four US states. They gathered news paragraphs on five fixed controversial topics through Media Cloud and labeled each paragraph with the location of the outlet that published it. Local MTurk workers checked a sample of the labels. The dataset is benchmarked with BiLSTM, BERT and RoBERTa, and roberta-base scores best. Per-group language models show that each corpus fits its own cultural domain better than the others. Using EnCBP as auxiliary training data, through two-stage training or MTL, improves most of nine downstream tasks, but it slightly hurts Go-Emotions. The study shows that style varies across regions within one language and can be modeled. It also warns that cultural augmentation only pays off when the domains of the tasks are compatible.

Key points

  • Labels:
    • EnCBP-country has US, UK, AUS, CAN and IND, drawn from outlets such as BBC, Times of India and Sydney Morning Herald.
    • EnCBP-district has NY, CA, GA and TX, each represented by three local outlets.
    • Every label has 2,000 paragraphs, split 1,600/200/200 for train/dev/test.
  • Data source and filtering:
    • For each outlet, 1,000 articles from Jan 2020 to Jun 2021 were gathered, all dealing with one of five subjects: abortion, mandatory vaccination, global warming, immigration, or the social safety net.
    • Documents under 100 words were removed.
    • Place names were replaced with "[country]"/"[district]" tokens.
    • Paragraphs containing region-specific named entities, identified by IDF score, were dropped.
  • Validation:
    • Accuracy ranged from 58.67% to 81.33%, against roughly 33% chance.
    • Agreement (Fleiss' κ) ranged from 0.57 to 0.78.
  • Classification benchmark: roberta-base reaches 82.96 F1-macro on EnCBP-country and 73.96 on EnCBP-district, outperforming BERT and BiLSTM.
  • Downstream tasks:
    • Gains appear on CoNLL-2003, most semantic tasks (PAWS-Wiki, QNLI, STS-B, RTE), SST-2/SST-5 and Emotion.
    • Go-Emotions declines, which the authors attribute to its culturally imbalanced corpus.
    • Cross-group perplexity gaps range from 0.43 to 14.90.

Interesting ideas

  • I1. Masking place names and filtering out paragraphs with locally specific entities (by IDF) stops a classifier from relying on obvious lexical giveaways, so it has to learn real stylistic signals instead.
  • I2. Validators judged a pair of paragraphs, one of them a known local reference, instead of making an absolute call; with three options the chance level is known in advance.
  • I3. Training one language model per group and scoring it on every group's corpus yields a domain-compatibility matrix that quantifies how far apart text varieties are.
  • I4. Holding the topic set constant across all groups means that differences between groups reflect style rather than subject matter.
  • I5. Auxiliary "variety" signals helped culturally diverse targets but hurt a homogeneous one, which shows that augmentation must be justified per task and not applied everywhere.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use IDF-based removal of shortcut tokens when building negative controls, so an evaluation shows whether a component relies on structure or on memorized lexical cues.
  • I2 (learn): Disagreements between silver-data witness parsers or LLM teacher outputs could be checked by comparing them against a trusted gold analysis, a design that is easier to audit than an isolated yes/no.
  • I3 (tree): Run the same cross-perplexity or cross-LAS matrix over treebanks and UD dialects to decide which parser or converted treebank best matches an incoming text.
  • I4 (learn): Build evaluation slices for regional or genre variation with matched content, so changes in parser or frame accuracy cannot be blamed on topic shift.
  • I5 (frames): Before feeding style or register features into speech-act or irony detection, check per target corpus whether they help, with a control run that omits them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dartmouth CollegeHanover · United States37 works · 42,625 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Weicheng Ma, Samiha Datta, Lili Wang and Soroush Vosoughi. 2022. EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-acl.221

@inproceedings{ma2022encbp,
  title = {{EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English}},
  author = {Weicheng Ma, Samiha Datta, Lili Wang and Soroush Vosoughi},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.221},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.221}
}

References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Francisca Adoma Acheampong, Chen Wenyu, and Henry Nunoo-Mensah. 2020. Text-based emotion detection: Advances, challenges, and opportunities. Engineering Reports, 2(7):e12189. ↗
  2. Badar Almuhailib. 2019. Analyzing cross-cultural writing differences using contrastive rhetoric: A critical review. Advances in Language and Literary Studies, 10(2):102–106. ↗
  3. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 6, pages 6–4. Venice. ↗
  4. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In TAC. find
  5. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055. ↗
  6. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. find
  7. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics. find
  8. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. ↗
  9. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9. Association for Computational Linguistics. ↗
  10. Maarten Grootendorst. 2020. Bertopic: Leveraging bert and c-tf-idf to create easily interpretable topics. find
  11. Shlomo Hareli, Konstantinos Kafetsios, and Ursula Hess. 2015. A cross-cultural study on emotion expression and the learning of social norms. Frontiers in psychology, 6:1501. ↗
  12. Ali Shariq Imran, Sher Mohammad Doudpota, Zenun Kastrati, and Rakhi Bhatra. 2020. Cross-cultural polarity and emotion detection using sentiment analysis and deep learning–a case study on covid-19. arXiv preprint arXiv:2008.10031. ↗
  13. Heejung S Kim. 2002. We talk, therefore we think? a cultural analysis of the effect of talking on thinking. Journal of personality and social psychology, 83(4):828. ↗
  14. Hiroko Kitano. 1990. Cross-cultural differences in written discourse patterns: a study of acceptability of japanese expository compositions in american universities. find
  15. Erin McClure Krampetz. 2005. International educational administration and policy analysis. find
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  17. Richard E Nisbett, Kaiping Peng, Incheol Choi, and Ara Norenzayan. 2001. Culture and systems of thought: holistic versus analytic cognition. Psychological review, 108(2):291. ↗
  18. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. ↗
  19. Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687–3697, Brussels, Belgium. Association for Computational Linguistics. ↗
  20. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics. you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.
  21. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR. ↗
  22. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  23. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase Adversaries from Word Scrambling. In Proc. of NAACL. ↗
  24. Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R. Mortensen. 2021. Cross-cultural similarity features for cross-lingual transfer learning of pragmatically motivated tasks. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2403–2414, Online. Association for Computational Linguistics.
  25. Timothy Tambassi. 2018. From geographical lines to cultural boundaries: Mapping the ontological debate. Rivista di estetica, 67:150–164. ↗
  26. Yufei Tian, Tuhin Chakrabarty, Fred Morstatter, and Nanyun Peng. 2021. Identifying distributional perspectives from colingual groups. In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, pages 178–190, Online. Association for Computational Linguistics. find
  27. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147. ↗
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2022-encbp-new-benchmark-dataset-finer/