Elamur Research⌕
☰
home / papers / 2022 / Mawqif: A Multi-label Arabic Dataset for Target-specific Stance Detection

Mawqif: A Multi-label Arabic Dataset for Target-specific Stance Detection

Paper · 2022 · Nora Saleh Alturayeif, Hamzah Abdullah Luqman and Moataz Aly Kamaleldin Ahmed

Summary

Most stance detection research has dealt with English, and earlier Arabic stance resources pair claims with reference texts in Modern Standard Arabic. None of them supports deciding a social-media author's position toward a named target. Mawqif fills this gap. It contains 4,121 multi-dialectal Arabic tweets, each tied to one of three topics that are contested in the Middle East: COVID-19 vaccine, digital transformation and women empowerment. Each tweet carries three labels: stance (favor, against, none), sentiment and sarcasm. Sarcasm was added because earlier work found sarcastic posts to be a major cause of stance errors. The authors describe the collection, filtering and crowdsourced annotation and analyse the resulting labels. They then fine-tune four BERT-based models as baselines. AraBERT-twitter performs best, reaching a macro-F1 of 78.89% (the average of the favor and against F1 scores) on the test set. All models are weak on the none class. The resource matters because it gives Arabic a public, multi-dimensional benchmark for stance, sentiment and sarcasm, released with guidelines and code.

Key points

  • Mawqif has 4,121 tweets and three targets, with labels for stance, sentiment and sarcasm. The data, annotation guidelines and code are on GitHub (NoraAlt/Mawqif-Arabic-Stance).
  • The baselines are CAMeLBERT-da, MARBERT, AraBERT and AraBERT-twitter. On the test set, AraBERT-twitter scores F1 86.54 on favor, 71.25 on against and only 27.91 on none, with 79.78% accuracy.
  • The authors attribute MARBERT's weaker results to its pretraining, which used MLM only, while AraBERT also used NSP, which may help with text–target pairs.
  • They attribute CAMeLBERT-da's lower scores to less pretraining data: 5.8B words and a 30K vocabulary, against 8.6B words, a 60K vocabulary and 60M dialect tweets for AraBERT-twitter.
  • On the two-class average, the women empowerment target is the easiest. Proposed future work includes handling class imbalance (resampling or weighted loss) and multi-task models that learn stance together with sentiment and sarcasm.

Interesting ideas

  • I1. Each text gets several parallel opinion layers (stance, sentiment, sarcasm), so researchers can study how the layers interact rather than treating stance alone
  • I2. Stance is always judged relative to an explicitly given target, so one text can be positive in tone yet against the target
  • I3. Sarcasm is annotated because an earlier error analysis named it a leading cause of stance mistakes, so the dataset design follows the observed failures
  • I4. The headline metric averages only the favor and against F1 scores and sets aside the uninformative none class, while still reporting the three-class average
  • I5. Results are broken down per target, which shows that difficulty depends strongly on the topic

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could tag each speech act separately for attitude toward a target, emotional tone and an irony flag, so that an ironic positive surface can explicitly flip the inferred stance.
  • I2 (world): the world of the text could store holder→target attitude facts keyed to entities, which keeps polarity separate from tone and makes it explainable.
  • I3 (learn): self-induced repair rules could be seeded from error audits by adding targeted labels for the phenomena that most often break a stage.
  • I4 (answer): evaluation of school tests and fable QA could report a primary score over the classes that matter plus a full-class score, so that frequent "no answer" cases do not hide weaknesses.
  • I5 (learn): evaluation could report results per genre or text source, such as fables versus maths problems, to find which domains the silver data underserves.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Fahd University of Petroleum and MineralsDhahran · Saudi Arabia52 works · 4,666 citations🎓Imam Abdulrahman Bin Faisal UniversityDammam · Saudi Arabia7 works · 95 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Nora Saleh Alturayeif, Hamzah Abdullah Luqman and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A Multi-label Arabic Dataset for Target-specific Stance Detection. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.16

@inproceedings{alturayeif2022mawqif,
  title = {{Mawqif: A Multi-label Arabic Dataset for Target-specific Stance Detection}},
  author = {Nora Saleh Alturayeif, Hamzah Abdullah Luqman and Moataz Aly Kamaleldin Ahmed},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.16},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.16}
}

References

43 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: Deep bidirectional transformers for arabic. arXiv. ↗
  2. Yamen Ajjour, Henning Wachsmuth, Johannes Kiesel, Martin Potthast, Matthias Hagen, and Benno Stein. 2019. Data acquisition for argument search: The args.me corpus. volume 11793 LNAI of Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz), pages 48–59. find
  3. Abdulrahman I. Al-Ghadir, Aqil M. Azmi, and Amir Hussain. 2021. A novel approach to stance detection in social media tweets by fusing ranked lists and sentiments. Information Fusion, 67:29–40. ↗
  4. Abeer AlDayel and Walid Magdy. 2021. Stance detection on social media: state of the art and trends. Information Processing and Management, 58. ↗
  5. Tariq Alhindi, Amal Alabdulkarim, Ali Alshehri, Muhammad Abdul-Mageed, and Preslav Nakov. 2021. Arastance: A multi-country and multi-domain dataset of arabic stance detection for fact checking. pages 57–65.
  6. Emily Allaway and Kathleen Mckeown. 2020. Zero-shot stance detection: A dataset and model using generalized topic representations. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8913–8931.
  7. Emily Allaway, Malavika Srikanth, and Kathleen Mckeown. 2021. Adversarial learning for zero-shot stance detection on social media. pages 4756–4767.
  8. Nora Alturayeif and Hamzah Luqman. 2021. Fine-grained sentiment analysis of arabic covid-19 tweets using bert-based transformers and dynamically weighted loss function. Applied Sciences, 11(22):10694. ↗
  9. Roy Bar-Haim, Indrajit Bhattacharya, Francesco Din-uzzo, Amrita Saha, and Noam Slonim. 2017. Stance classification of context-dependent claims. volume 1 of Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251–261.
  10. Pengyuan Chen, Kai Ye, and Xiaohui Cui. 2021. Integrating n-gram features into pre-trained model: A novel ensemble model for multi-target stance detection. volume 12893 LNCS, pages 269–279. Springer Science and Business Media Deutschland GmbH. find
  11. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing things from a different angle: Discovering diverse perspectives about claims. volume 1 of NAACL HLT 2019 -2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  12. Alessandra Teresa Cignarella, Mirko Lai, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2020. Sardistance @ evalita2020: Overview of the task on stance detection in italian tweets. volume 2765 of EVALITA 2020 Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, pages 1–10. find
  13. Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-they-won’t-they: A very large dataset for stance detection on twitter. arXiv.
  14. Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. Semeval-2017 task 8: Rumoureval: Determining rumour veracity and support for rumours. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 69–76.
  15. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies. ACL, pages 1163– 1168.
  16. Shalmoli Ghosh, Prajwal Singhania, Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019. Stance detection in web and social media: A comparative study. International Conference of the Cross-Language Evaluation Forum for European Languages, pages 75–87. find
  17. Mirko Lai, Viviana Patti, Giancarlo Ruffo, and Paolo Rosso. 2018. Stance evolution and twitter interactions in an italian political debate. volume 10859 LNCS of International Conference on Applications of Natural Language to Information Systems, pages 15–27. find
  18. Santiago González-Carvajal and Eduardo C. Garrido-Merchán. 2020. Comparing bert against traditional machine learning text classification. arXiv e-prints. find
  19. Mirko Lai, Viviana Patti, Giancarlo Ruffo, and Paolo Rosso. 2020. #brexit: Leave or remain? the role of user’s community and diachronic evolution on stance detection. Journal of Intelligent and Fuzzy Systems, 39:2341–2352. ↗
  20. Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. 2019. Rumoureval 2019: Determining rumour veracity and support for rumours. Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 845– 854. ↗
  21. Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Ja-yaraman Nair, Diana Inkpen, and Cornelia Caragea. 2021a. P-stance: A large dataset for stance detection in political domain. pages 2355–2365.
  22. Lara Grimminger and Roman Klinger. 2021. Hate towards the political opponent: A twitter corpus study of the 2020 us elections on the basis of offensive speech and stance detection. arXiv. ↗
  23. Yingjie Li, Chenye Zhao, and Cornelia Caragea. 2021b. Improving stance detection with multi-dataset learning and knowledge distillation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6332–6345.
  24. Andreas Hanselowski, Avinesh P.V.S., Benjamin Schiller, Felix Caspelherr, Debanjan * Chaudhuri, Christian M Meyer, and Iryna Gurevych. 2018. A retrospective analysis of the fake news challenge stance-detection task. Proceedings of the 27th International Conference on Computational Linguistics (COLING 2018).
  25. Bin Liang, Yonghao Fu, Lin Gui, Min Yang, Jiachen Du, Yulan He, and Ruifeng Xu. 2021. Target-adaptive graph for cross-target stance detection. Proceedings of the World Wide Web Conference, WWW 2021, pages 3453–3464. Association for Computing Machinery, Inc. find
  26. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. ↗
  27. Tomáš Hercig, Peter Krejzl, Barbora Hourová, Josef Steinberger, and Ladislav Lenc. 2017. Detecting stance in czech news commentaries. ITAT, pages 176–180. find
  28. Marjan Hosseinia, Eduard Dragut, and Arjun Mukherjee. 2020. Stance prediction for contemporary issues: Data and experiments. Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics (ACL). find
  29. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. pages 92–104.
  30. Jude Khouja. 2020. Stance prediction and claim verification: An arabic perspective. Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER), pages 8–17.
  31. Elena Kochkina, Maria Liakata, and Isabelle Augenstein. 2017. Turing at semeval-2017 task 8: Sequential approach to rumour stance classification with branch-lstm. Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 475–480. find
  32. Dilek Küçük and Fazli Can. 2018. Stance detection on tweets: An svm-based approach. arXiv, pages 1–13. find
  33. Dilek Küçük and C. A.N. Fazli. 2020. Stance detection: A survey. ACM Computing Surveys, 53. find
  34. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41.
  35. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017. Stance and sentiment in tweets. ACM Transactions on Internet Technology (TOIT), 17:1–23. ↗
  36. Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017. A dataset for multi-target stance detection. volume 2 of 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, pages 551–557.
  37. Parinaz Sobhani, Saif M Mohammad, and Svetlana Kiritchenko. 2016. Detecting stance in tweets and analyzing its interaction with sentiment. Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (SEM 2016), pages 159–169. find
  38. Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. 2018. Cross-topic argument mining from heterogeneous sources using attention-based neural networks. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. ↗
  39. Mariona Taulé, M. Antónia Martín, Francisco Rangel, Paolo Rosso, Cristina Bosco, and Viviana Patti. 2017. Overview of the task on stance and gender detection in tweets on catalan independence at ibereval 2017. volume 1881 of 2nd Workshop on Evaluation of Human Language Technologies for Iberian Languages, IberEval 2017, pages 157–177. find
  40. Jannis Vamvas and Rico Sennrich. 2020. X-stance: A multilingual multi-target dataset for stance detection. 5th SwissText & 16th KONVENS Joint Conference 2020. ↗
  41. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  42. Ruifeng Xu, Yu Zhou, Dongyin Wu, Lin Gui, Jiachen Du, and Yun Xue. 2016. Overview of nlpcc shared task 4: Stance detection in chinese microblogs. Natural language understanding and intelligent applications, pages 907–916. find
  43. Elena Zotova, Rodrigo Agerri, and German Rigau. 2021. Semi-automatic generation of multilingual datasets for stance detection in twitter. Expert Systems with Applications, 170:1–29. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alturayeif-2022-mawqif-multi-label-arabic-dataset/