Elamur Research⌕
☰
home / papers / 2023 / Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media

Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media

Paper · 2023 · Mark Mets, Andres Karjus, Indrek Ibrus et al.

Summary

The paper asks whether automatic stance detection works in a hard setting: Estonian, a morphologically rich language with relatively few NLP resources, and immigration, a politically charged topic. The authors selected about 8000 topic-relevant sentences from two news sources, mainstream Ekspress Grupp and the right-wing populist portal Uued Uudised, published 2015–2022, using a keyword and stem lexicon. Human annotators labeled the sentences as pro- or anti-immigration. Several pretrained multilingual and Estonian-specific LLMs were fine-tuned on these labels. The authors compared them with ChatGPT 3.5 used as a zero-shot classifier and with a classifier tuned on general sentiment data.

Fine-tuned models reached acceptable accuracy, and ChatGPT came close to the best of them. The sentiment-based shortcut did poorly, because sentences with a negative tone can still support immigration. Applied to the full corpus, the best model traced stance shifts over seven years that line up with real-world events.

This matters because it suggests instructable LLMs could make media monitoring cheaper in small languages, provided they are validated against annotated data.

Key points

  • The released dataset contains 7345 stance-annotated Estonian sentences. Data and code are at github.com/markmets/immigration-prediction-EST.
  • ChatGPT 3.5 as a zero-shot classifier performed nearly as well as the top fine-tuned LLM. Bootstrapping stance from a generic sentiment classifier did not work well.
  • The diachronic analysis contrasts Uued Uudised with Ekspress Grupp outlets. Stance changes are linked to:
    • the 2015–2016 migrant crisis,
    • the 2018–2019 UN migration pact debate and the 2019 Estonian elections,
    • the 2022 Russian invasion of Ukraine.
  • In a qualitative error analysis, both annotators and models struggled with sarcasm, dog-whistles and context-dependent opinions. Choosing sentences versus paragraphs as the unit of analysis is a trade-off between too little context and mixed stances.
  • Reproducibility is a stated concern for prompt-based methods: results can shift with wording, with model randomness, and with updates to cloud-hosted models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Estonian Business SchoolTallinn · Estonia2 works · 59 citations🎓Tallinn UniversityTallinn · Estonia2 works · 59 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 972 works

Cite

Cite as Mark Mets, Andres Karjus, Indrek Ibrus and Maximilian Schich. 2023. Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media. arXiv:2305.13047 · Plos one, 19(4), e0302380 (2024).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1371/journal.pone.0302380 · arXiv 2305.13047

@misc{mets2023automated,
  title = {{Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media}},
  author = {Mark Mets, Andres Karjus, Indrek Ibrus and Maximilian Schich},
  year = {2023},
  doi = {10.1371/journal.pone.0302380},
  eprint = {2305.13047},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1371/journal.pone.0302380}
}

References

46 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abts, K., & Rummens, S. (2007). Populism versus Democracy. Political Studies, 55(2), 405–424. https://doi.org/10.1111/j.1467-9248.2007.00657.x — [doi:10.1111/j.1467-9248.2007.00657.x](https://doi.org/10.1111/j.1467-9248.2007.00657.x) ↗
  2. Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023). Can we trust the evaluation on ChatGPT? ArXiv:2303.12767 [Cs]. Accessed 10 May 2023.
  3. ALDayel, A., & Magdy, W. (2021). Stance detection on social media: State of the art and trends. Information Processing & Management, 58(4), 102597. https://doi.org/10.1016/j.ipm.2021.102597 ↗
  4. Allaway, E., & McKeown, K. (2020). Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 8913–8931.
  5. Auers, D. (2018). Populism and Political Party Institutionalisation in the Three Baltic States of Estonia, Latvia and Lithuania. Fudan Journal of the Humanities and Social Sciences, 11(3), 341–355. https://doi.org/10.1007/s40647-018-0231-1 — [doi:10.1007/s40647-018-0231-1](https://doi.org/10.1007/s40647-018-0231-1) ↗
  6. Batanović, V., Cvetanović, M., & Nikolić, B. (2020). A versatile framework for resource-limited sentiment articulation, annotation, and analysis of short texts. PLOS ONE, 15(11), e0242050. https://doi.org/10.1371/journal.pone.0242050 — [doi:10.1371/journal.pone.0242050](https://doi.org/10.1371/journal.pone.0242050) ↗
  7. 🦜 Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? . Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922 — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) ↗
  8. Braghiroli, S., & Makarychev, A. (2023). Сonservative populism in Italy and Estonia: Playing the multicultural card and engaging “domestic others.” East European Politics, 39(1), 128–149. https://doi.org/10.1080/21599165.2022.2077725 — [doi:10.1080/21599165.2022.2077725](https://doi.org/10.1080/21599165.2022.2077725) ↗
  9. Burscher, B., van Spanje, J., & de Vreese, C. H. (2015). Owning the issues of crime and immigration: The relation between immigration and crime news and anti-immigrant voting in 11 countries. Electoral Studies, 38, 59–69. https://doi.org/10.1016/j.electstud.2015.03.001 — [doi:10.1016/j.electstud.2015.03.001](https://doi.org/10.1016/j.electstud.2015.03.001) ↗
  10. Card, D., Chang, S., Becker, C., Mendelsohn, J., Voigt, R., Boustan, L., Abramitzky, R., & Jurafsky, D. (2022). Computational analysis of 140 years of US political speeches reveals more positive but increasingly polarized framing of immigration. Proceedings of the National Academy of Sciences, 119(3… find
  11. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8440–8451.… find
  12. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, 4171–…
  13. Do, H. H., Prasad, P., Maag, A., & Alsadoon, A. (2019). Deep Learning for Aspect-Based Sentiment Analysis: A Comparative Review. Expert Systems with Applications, 118, 272–299. https://doi.org/10.1016/j.eswa.2018.10.003 — [doi:10.1016/j.eswa.2018.10.003](https://doi.org/10.1016/j.eswa.2018.10.003) ↗
  14. Du, J., Xu, R., He, Y., & Gui, L. (2017). Stance Classification with Target-specific Neural Attention. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 3988–3994. find
  15. Engesser, S., Ernst, N., Esser, F., & Büchel, F. (2017). Populism and social media: How politicians spread a fragmented ideology. Information, Communication & Society, 20(8), 1109–1126. https://doi.org/10.1080/1369118X.2016.1207697 — [doi:10.1080/1369118x.2016.1207697](https://doi.org/10.1080/1369118x.2016.1207697) ↗
  16. Ghosh, S., Singhania, P., Singh, S., Rudra, K., & Ghosh, S. (2019). Stance Detection in Web and Social Media: A Comparative Study. In F. Crestani, M. Braschler, J. Savoy, A. Rauber, H. Müller, D. E. Losada, G. Heinatz Bürki, L. Cappellato, & N. Ferro (Eds.), Experimental IR Meets Multilinguality, Mu… — [doi:10.1007/978-3-030-28577-7_4](https://doi.org/10.1007/978-3-030-28577-7_4) find
  17. Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks. ArXiv:2303.15056 [Cs]. Accessed 10 Mai 2023.
  18. Hedderich, M. A., Lange, L., Adel, H., Strötgen, J., & Klakow, D. (2021). A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
  19. Kaal, E., & Renser, B. (2019). Rändetemaatika kajastamine Eesti meedias. MTÜ Mondo. https://mondo.org.ee/wp-content/uploads/2019/09/R%C3%A4ndetemaatika-kajas tamine-Eesti-meedias.pdf Kasekamp, A., Madisson, M.-L., & Wierenga, L. (2019). Discursive Opportunities for the Estonian Populist Radical Righ… — [doi:10.1080/10758216.2018.1445973](https://doi.org/10.1080/10758216.2018.1445973) ↗
  20. Khatua, A., & Nejdl, W. (2022). Unraveling Social Perceptions & Behaviors towards Migrants on Twitter. Proceedings of the Sixteenth International AAAI Conference on Web and Social Media (ICWSM 2022), 16, 512–523. find
  21. Koppel, K., & Jakobson, M.-L. (2023). Who Is the Worst Migrant? Migrant Hierarchies in Populist Radical-Right Rhetoric in Estonia. In M.-L. Jakobson, R. King, L. Moroşanu, & R. Vetik (Eds.), Anxieties of Migration and Integration in Turbulent Times (pp. 225–241). Springer International Publishing. h… — [doi:10.1007/978-3-031-23996-0_13](https://doi.org/10.1007/978-3-031-23996-0_13) find
  22. Küçük, D., & Can, F. (2020). Stance Detection: A Survey. ACM Computing Surveys, 53(1), 1–37. https://doi.org/10.1145/3369026 ↗
  23. Li, W., Gao, S., Zhou, H., Huang, Z., Zhang, K., & Li, W. (2019). The Automatic Text Classification Method Based on BERT and Feature Union. 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS), 774–777. https://doi.org/10.1109/ICPADS47876.2019.00114 — [doi:10.1109/icpads47876.2019.00114](https://doi.org/10.1109/icpads47876.2019.00114) ↗
  24. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 55(9), 1–35. https://doi.org/10.1145/3560815 ↗
  25. Lotman, Yu. M., Uspensky, B. A., & Mihaychuk, G. (1978). On the Semiotic Mechanism of Culture. New Literary History, 9(2), 211. https://doi.org/10.2307/468571 — [doi:10.2307/468571](https://doi.org/10.2307/468571) ↗
  26. Madisson, M.-L., & Ventsel, A. (2016). Autocommunicative meaning-making in online communication of the Estonian extreme right. Sign Systems Studies, 44(3), 326–354. https://doi.org/10.12697/SSS.2016.44.3.02 — [doi:10.12697/sss.2016.44.3.02](https://doi.org/10.12697/sss.2016.44.3.02) ↗
  27. Madisson, M.-L., & Ventsel, A. (2018). Groupuscular identity-creation in online-communication of the Estonian extreme right. Semiotica, 2018(222), 25–46. https://doi.org/10.1515/sem-2016-0077 — [doi:10.1515/sem-2016-0077](https://doi.org/10.1515/sem-2016-0077) ↗
  28. Magueresse, A., Carles, V., & Heetderks, E. (2020). Low-resource Languages: A Review of Past Work and Future Challenges. ArXiv:2006.07264 [Cs]. Accessed 10 May 2023. — [arXiv:2006.07264](https://arxiv.org/abs/2006.07264) find
  29. Meltzer, C. E., & Schemer, C. (2021). Miscounting the others: Media effects on perceptions of the immigrant population size. In Media and Public Attitudes Toward Migration in Europe (pp. 174–189). Routledge. find
  30. Mohammad, S. M., Sobhani, P., & Kiritchenko, S. (2017). Stance and Sentiment in Tweets. ACM Transactions on Internet Technology, 17(3), 1–23. https://doi.org/10.1145/3003433 ↗
  31. Mudde, C. (2007). Populist radical right parties in Europe. Cambridge University Press. find
  32. Pajupuu, H., Altrov, R., & Pajupuu, J. (2016). Identifying Polarity in Different Text Types. Folklore: Electronic Journal of Folklore, 64, 125–142. https://doi.org/10.7592/FEJF2016.64.polarity — [doi:10.7592/fejf2016.64.polarity](https://doi.org/10.7592/fejf2016.64.polarity) ↗
  33. Pang, B., & Lee, L. (2008). Opinion Mining and Sentiment Analysis. Foundations and Trends® in Information Retrieval, 2(1–2), 1–135. https://doi.org/10.1561/1500000011 ↗
  34. Petsinis, V. (2019). Identity Politics and Right-Wing Populism in Estonia: The Case of EKRE. Nationalism and Ethnic Politics, 25(2), 211–230. https://doi.org/10.1080/13537113.2019.1602374 — [doi:10.1080/13537113.2019.1602374](https://doi.org/10.1080/13537113.2019.1602374) ↗
  35. Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023). Is ChatGPT a General-Purpose Natural Language Processing Task Solver? ArXiv:2302.06476 [Cs]. Accessed 10 May 2023.
  36. Reimers, N., & Gurevych, I. (2020). Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 4512–4525.
  37. Reiss, M. V. (2023). Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark. ArXiv:2304.11085 [Cs]. Accessed 10 May 2023. — [arXiv:2304.11085](https://arxiv.org/abs/2304.11085) find
  38. Rooduijn, M., de Lange, S. L., & van der Brug, W. (2014). A populist Zeitgeist ? Programmatic contagion by populist parties in Western Europe. Party Politics, 20(4), 563–575. https://doi.org/10.1177/1354068811436065 — [doi:10.1177/1354068811436065](https://doi.org/10.1177/1354068811436065) ↗
  39. Saarts, T., Jakobson, M.-L., & Kalev, L. (2021). When a Right-Wing Populist Party Inherits a Mass Party Organisation: The Case of EKRE. Politics and Governance, 9(4), Article 4. https://doi.org/10.17645/pag.v9i4.4566 — [doi:10.17645/pag.v9i4.4566](https://doi.org/10.17645/pag.v9i4.4566) ↗
  40. Sahlgren, M. (2008). The distributional hypothesis. Italian journal of linguistics, 20(1), 33-54. find
  41. Sobhani, P., Inkpen, D., & Zhu, X. (2017). A Dataset for Multi-Target Stance Detection. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 551–557. https://aclanthology.org/E17-2088 ↗
  42. Tanvir, H., Kittask, C., Eiche, S., & Sirts, K. (2021). EstBERT: A Pretrained Language-Specific BERT for Estonian. Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). 11–19 find
  43. Ulčar, M., Žagar, A., Armendariz, C. S., Repar, A., Pollak, S., Purver, M., & Robnik-Šikonja, M. (2021). Evaluation of contextual embeddings on less-resourced languages. ArXiv:2107.10614 [Cs]. Accessed 10 Mai 2023. — [arXiv:2107.10614](https://arxiv.org/abs/2107.10614) find
  44. Vamvas, J., & Sennrich, R. (2020). X-Stance: A Multilingual Multi-Target Dataset for Stance Detection. 5th Swiss Text Analytics Conference (SwissText) & 16th Conference on Natural Language Processing (KONVENS). find
  45. Yantseva, V., & Kucher, K. (2021). Machine Learning for Social Sciences: Stance Classification of User Messages on a Migrant-Critical Discussion Forum. 2021 Swedish Workshop on Data Science (SweDS), 1–8. https://doi.org/10.1109/SweDS53855.2021.9637718 — [doi:10.1109/sweds53855.2021.9637718](https://doi.org/10.1109/sweds53855.2021.9637718) ↗
  46. Zhang, B., Ding, D., & Jing, L. (2023). How would Stance Detection Techniques Evolve after the Launch of ChatGPT? ArXiv:2212.14548 [Cs]. Accessed 10 Mai 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mets-2023-automated-stance-detection-complex-topics/