Elamur Research
☰
home / papers / 2023 / From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models

From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.acl-long.656.pdf

Summary

Large language models absorb text from news, forums, books and encyclopedias, and much of that text carries political opinions. The authors ask whether such ideological slant passes from pretraining data into LMs and then into classifiers used for sensitive tasks. To measure an LM's leaning, they adapt the political compass test, which places responses on a social axis and an economic axis. Encoder models answer the test statements through mask filling. For generative models, the authors prompt for free-text answers and label each answer's stance with a classifier. They then continue pretraining existing LMs on left, center and right corpora from news and social media and track how each model's position moves. Finally, they fine-tune these partisan variants for hate speech and misinformation detection. The models do have measurable political leanings, and these reflect the polarization of their training data. Overall accuracy stays similar across variants, but behavior toward specific identity groups and media sources differs noticeably. This shows that fairness audits must look beyond aggregate scores.

Key points

  • The probing framework uses the 62 statements of the political compass test. Each model gets a social and an economic score, both ranging from −10 to 10. The authors report similar findings with the 8 values test.
  • They probe 14 LMs, from BERT, RoBERTa and ALBERT to GPT-2, GPT-3, LLaMA, Alpaca, Codex, ChatGPT and GPT-4.
  • Generated answers are scored by a BART-based stance detector trained on MultiNLI. In a human check of 110 responses, it reached 0.97 accuracy, with annotator agreement of 0.85 Fleiss' Kappa.
  • Partisan news text for continued pretraining comes from the POLITICS dataset. Downstream evaluation uses three datasets:
    • HATE-IDENTITY (159,872 examples, Yoder et al., 2022)
    • HATE-DEMOGRAPHIC (276,872 examples, Yoder et al., 2022)
    • MISINFORMATION (29,556 examples, Wang, 2017)
  • Code and data are released as PoliLean on GitHub. The trained hyperpartisan checkpoints are withheld, and access to the corpora is limited to research use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

162 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 158 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.656

@inproceedings{feng2023pretraining,
  title = {{From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models}},
  author = {Feng, Shangbin and Park, Chan Young and Liu, Yuhan and Tsvetkov, Yulia},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.656},
  url = {https://doi.org/10.18653/v1/2023.acl-long.656}
}

References

124 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Abramowitz and Jennifer McCoy. 2019. United states: Racial resentment, negative partisanship, and polarization in trump’s america. The ANNALS of the American Academy of Political and Social Science, 681(1):137–156. find
  2. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2020. Modeling annotator perspective and polarized opinions to improve hate speech detection. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 151–154. find
  3. Jacob Amedie. 2015. The impact of social media on society. find
  4. Aishwarya Anegundi, Konstantin Schulz, Christian Rauh, and Georg Rehm. 2022. Modelling cultural and socio-economic dimensions of political bias in German tweets. In Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022), pages 29–40, Potsdam, Germany. find
  5. KONVENS 2022 Organizers. find
  6. Lisa P. Argyle, E. Busby, Nancy Fulda, Joshua Ronald Gubler, Christopher Michael Rytting, and David Wingate. 2022. Out of one, many: Using language models to simulate human samples. ArXiv, abs/2209.06899. find our card
  7. Eugene Bagdasaryan and Vitaly Shmatikov. 2022. Spinning language models: Risks of propaganda-as-a-service and countermeasures. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1532–1532. IEEE Computer Society. find
  8. Pieter Ballon. 2014. Old and new issues in media economics. In The Palgrave handbook of European media policy, pages 70–95. Springer. find
  9. Yejin Bang, Nayeon Lee, Etsuko Ishii, Andrea Madotto, and Pascale Fung. 2021. Assessing political prudence of open-domain chatbots. In Proceedings Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too… find
  10. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc. find
  11. Irene V Blair. 2002. The malleability of automatic stereotypes and prejudice. Personality and social psychology review, 6(3):242–261. find
  12. Charles Blattberg. 2001. Political philosophies and political ideologies. Public Affairs Quarterly, 15(3):193–217. find
  13. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020a. Language (technology) is power: A critical survey of “bias” in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476. find our card
  14. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020b. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics. find our card
  15. Norberto Bobbio. 1996. Left and right: The significance of a political distinction. University of Chicago Press. find
  16. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems, pages 4349– 4357. most divisive. Journal of personality and social psycho… find our card
  17. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pages 491–500. find
  18. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110. find our card
  19. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  20. Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, and Dan Jurafsky. 2019. Analyzing polarization in social media: Method and application to tweets on 21 mass shootings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Comp… find
  21. Philip Bump. 2016. How likely are bernie sanders supporters to actually vote for donald trump? here are some clues. Washingtonpost. com. find
  22. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. PMLR. find
  23. Michael A Cacciatore, Ashley A Anderson, Doo-Hun Choi, Dominique Brossard, Dietram A Scheufele, Xuan Liang, Peter J Ladwig, Michael Xenos, and Anthony Dudo. 2012. Coverage of emerging technologies: A comparison between print and online media. New media & society, 14(6):1039–1059. find
  24. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  25. Yang Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Galstyan. 2022. On the intrinsic and extrinsic fairness evaluation metrics for contextualized language representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Lingui… find
  26. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. find our card
  27. Elanor Colleoni, Alessandro Rozza, and Adam Arvidsson. 2014. Echo chamber or public sphere? predicting political orientation and measuring political homophily in twitter using big data. Journal of communication, 64(2):317–332. find
  28. Michael C Corballis and Ivan L Beale. 2020. The psychology of left and right. Routledge. find
  29. Jarret T Crawford, Mark J Brandt, Yoel Inbar, John R Chambers, and Matt Motyl. 2017. Social and economic ideologies differentially predict prejudice across the political spectrum, but social issues are Patricia G Devine. 1989. Stereotypes and prejudice: Their automatic and controlled components. Jou… find
  30. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  31. Stanley Diamond and Eric Wolf. 2017. In search of the primitive: A critique of civilization. Routledge. find
  32. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. find
  33. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… find
  34. Maeve Duggan. 2017. Online harassment 2017. find
  35. Denis Emelin, Ronan Le Bras, Jena D Hwang, Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 698–718. find
  36. R. S. Enikolopov, Maria Petrova, and Ekaterina Zhu-ravskaya. 2019. Political effects of the internet and social media. Political Behavior: Cognition. find
  37. Hans Jurgen Eysenck. 1957. Sense and nonsense in psychology. find
  38. William Falcon and The PyTorch Lightning team. 2019. PyTorch Lightning. find
  39. Shangbin Feng, Zilong Chen, Wenqian Zhang, Qingyao Li, Qinghua Zheng, Xiaojun Chang, and Minnan Luo. 2021. Kgap: Knowledge graph augmented political perspective detection in news media. arXiv preprint arXiv:2108.03861. — [arXiv:2108.03861](https://arxiv.org/abs/2108.03861) find
  40. Shangbin Feng, Zhaoxuan Tan, Zilong Chen, Ningnan Wang, Peisheng Yu, Qinghua Zheng, Xiaojun Chang, and Minnan Luo. 2022. PAR: Political actor representation learning with social context and expert knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. find
  41. Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A survey of race, racism, and anti-racism in nlp. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:… find
  42. Claudia Flores-Saviaga, Shangbin Feng, and Saiph Savage. 2022. Datavoidant: An ai system for addressing political data voids on social media. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):1–29. find
  43. Daniel J Galvin. 2020. Party domination and base mobilization: Donald trump and republican party building in a polarized era. In The Forum, volume 18, pages 135–168. De Gruyter. find
  44. Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis. 2018. Political discourse on social media: Echo chambers, gatekeepers, and the price of bipartisanship. In Proceedings of the 2018 world wide web conference, pages 913–922. find
  45. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf… find
  46. Sayan Ghosh, Dylan Baker, David Jurgens, and Vinodkumar Prabhakaran. 2021. Detecting cross-geographic biases in toxicity modeling on social media. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 313– 328. find
  47. Allen Gindler. 2021. The theory of the political spectrum. Journal of Libertarian Studies, 24(2):24375. find
  48. Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confere… find
  49. Hila Gonen and Kellie Webster. 2020. Automatically identifying gender issues in machine translation using perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1991–1995, Online. Association for Computational Linguistics. find
  50. Lauren Guggenheim, S Mo Jang, Soo Young Bae, and W Russell Neuman. 2015. The dynamics of issue frame competition in traditional and social media. The ANNALS of the American Academy of Political and Social Science, 659(1):207–224. find
  51. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360. find
  52. Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29. find our card
  53. Camille Harris, Matan Halevy, Ayanna Howard, Amy Bruckman, and Diyi Yang. 2022. Exploring the role of grammar and word choice in bias toward african american english (aae) in hate speech classification. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 789–798. find
  54. Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. 2020. Array programming with numpy. Nature, 585(7825):357–362. find our card
  55. Alfred Hermida. 2016. Social media and the news. The SAGE handbook of digital journalism, pages 81–94. find
  56. Alfred Hermida, Fred Fletcher, Darryl Korell, and Donna Logan. 2012. Share, like, recommend: Decoding the social media news consumer. Journalism studies, 13(5-6):815–824. find
  57. David G Horrell. 2005. Paul among liberals and com-munitarians: models for christian ethics. Pacifica, 18(1):33–52. find
  58. Michael Hout and Christopher Maggio. 2021. Immigration, race & political polarization. Daedalus, 150(2):40–55. find
  59. Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 483–488, Beijing, C… find
  60. Kenneth Hudson. 1978. The language of modern politics. Springer. find
  61. Ben Hutchinson and Margaret Mitchell. 2019. 50 years of test (un) fairness: Lessons for machine learning. In Proceedings of the conference on fairness, accountability, and transparency, pages 49–58. find
  62. Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022a. CommunityLM: Probing partisan worldviews from language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6818–6826, Gyeongju, Republic of Korea. International Committee on Computational Linguis… find
  63. Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022b. CommunityLM: Probing partisan worldviews from language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6818–6826, Gyeongju, Republic of Korea. International Committee on Computational Linguis… find
  64. Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. 2021. On transferability of bias mitigation effects in language model fine-tuning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis… find
  65. Christopher D Johnston and Julie Wronski. 2015. Personality dispositions and political preferences across hard and easy issues. Political Psychology, 36(1):35– 53. find
  66. Kenneth Joseph and Jonathan M. Morgan. 2020. When do word embeddings accurately reflect surveys on our beliefs about people? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find
  67. Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations. find
  68. Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. 2022. Language generation models can cause harm: So what can we do about it? an actionable survey. arXiv preprint arXiv:2210.07700. find
  69. Anna Sophie Kümpel, Veronika Karnowski, and Till Keyling. 2015. News sharing in social media: A review of current research on news sharing users, content, and networks. Social media+ society, 1(2):2056305115610141. find
  70. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing. find our card
  71. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. find
  72. Anti-Defamation League. 2019. Online hate and harassment: The American experience. find
  73. Anti-Defamation League. 2021. The dangers of disinformation. find
  74. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computa… find our card
  75. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  76. Chang Li and Dan Goldwasser. 2019. Encoding social information with graph convolutional networks forPolitical perspective detection in news media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2594– 2604, Florence, Italy. Association for Computatio… find
  77. Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2021. Contextualized perturbation for textual adversarial attack. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find
  78. Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020. UNQOVERing stereotyping biases via underspecified questions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3475–3489, Online. Association for Computational Linguistics. find
  79. Yizhi Li, Ge Zhang, Bohao Yang, Chenghua Lin, Anton Ragni, Shi Wang, and Jie Fu. 2022. Herb: Measuring hierarchical regional bias in pre-trained language models. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 334–346. find
  80. Inna Lin, Lucille Njoo, Anjalie Field, Ashish Sharma, Katharina Reinecke, Tim Althoff, and Yulia Tsvetkov. 2022. Gendered mental health stigma in masked language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. find
  81. Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2020. Does gender matter? towards fairness in dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4403–4416, Barcelona, Spain (Online). International Committee on Computa… find
  82. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021. Mitigating political bias in language models through reinforced calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14857– 14866. find
  83. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. find
  84. Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022a. POLI-TICS: Pretraining with same-story article comparison for ideology prediction and stance detection. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1354–1374, Seattle, Uni… find
  85. Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022b. POLI-TICS: Pretraining with same-story article comparison for ideology prediction and stance detection. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1354–1374, Seattle, Uni… find
  86. Peter Mair. 2007. Left–right orientations. find
  87. Aibek Makazhanov and Davood Rafiei. 2013. Predicting political preference of twitter users. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pages 298–305. on Natural Language Processing (Volume 1: Long Papers), pages 5356–5371, Online.… find
  88. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967. find our card
  89. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774. find our card
  90. ArXiv, Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799–2804, Brussels, Belgium. Association for Computational Linguistics. find our card
  91. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32. find our card
  92. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  93. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find our card
  94. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14867–14875. find
  95. Daniel Preoţiuc-Pietro, Ye Liu, Daniel Hopkins, and Lyle Ungar. 2017. Beyond binary labels: Political ideology prediction of Twitter users. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 729–740, Vancouver, Canada. Associati… find
  96. Brian Patrick Mitchell. 2007. Eight ways to run the country: A new and revealing look at left and right. Greenwood Publishing Group. find
  97. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  98. Eni Mustafaraj and Panagiotis Takis Metaxas. 2011. What edited retweets reveal about online political discourse. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. find
  99. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67. find
  100. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Lee Rainie, Aaron Smith, Kay Lehman Schlozman,… find our card
  101. Cameron Raymond, Isaac Waller, and Ashton Anderson. 2022. Measuring alignment of online grassroots political communities with political campaigns. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 806–816. find
  102. Megan Trudell. 2016. Sanders, trump and the us working class. International Socialism. find
  103. Milton Rokeach. 1973. The nature of human values. Free press. find
  104. Sebastián Valenzuela, Yonghwan Kim, and Homero Gil de Zúñiga. 2012. Social networks that matter: Exploring the role of political discussion for online political participation. International Journal of Public Opinion Research, 24:163–184. find
  105. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  106. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find our card
  107. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling… find our card
  108. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2022. On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061. find our card
  109. Qinlan Shen and Carolyn Rose. 2021. What sounds “right” to me? experiential factors in the perception of political ideology. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. find
  110. Tom Utley. 2001. I’m v. right-wing, says the bbc, but it’s not that simple. find
  111. Alcides Velasquez. 2012. Social media and online political discussion: The effect of cues and informational cascades on participation in online political communities. New Media & Society, 14(8):1286–1303. find
  112. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax. ↗
  113. Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2… find
  114. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). find our card
  115. Ryan Steed, Swetasudha Panda, Ari Kobren, and Michael Wick. 2022. Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find
  116. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  117. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… find our card
  118. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language pr… find our card
  119. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https: //github.com/tatsu-lab/stanford\_alpaca. find
  120. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  121. Michael Yoder, Lynnette Ng, David West Brown, and Kathleen Carley. 2022. How hate speech varies by target identity: A computational analysis. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL). find
  122. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR. find our card
  123. Wenqian Zhang, Shangbin Feng, Zilong Chen, Zhenyu Lei, Jundong Li, and Minnan Luo. 2022. KCD: Knowledge walks and textual cues enhanced political perspective detection in news media. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics:… find
  124. Category Tokens positive agree, agrees, agreeing, agreed, support, supports, supported, supporting, believe, believes, believed, believing, accept, accepts, accepted, accepting, approve, approves, approved, approving, endorse, endorses, endorsed, endorsing Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vice… find our card