Elamur Research⌕
☰
home / papers / 2024 / MARASTA: A Multi-dialectal Arabic Cross-domain Stance Corpus

MARASTA: A Multi-dialectal Arabic Cross-domain Stance Corpus

Paper · 2024 · Anis Charfi, Mabrouka Bessghaier, Andria Atalla et al.

Summary

MARASTA is a new Arabic stance-detection dataset. It addresses a gap in earlier Arabic stance resources, which tend to cover one topic or one country, or have badly skewed label distributions. The authors collected 4,657 sentences from four dialect regions: Maghreb (Tunisia), Egypt, the Levant (Palestine and Jordan) and the Gulf. For each region they chose two contested topics, giving eight in total. Within each topic, about half the sentences are written in the dialect of the region and the other half in Modern Standard Arabic (MSA). Each sentence was labelled as favouring the topic, opposing it, or neutral. At least two annotators labelled every item, and a third resolved disagreements. The labels were deliberately kept balanced.

Classical and neural classifiers were then trained on the data. A Multi-Layer Perceptron (MLP) using Unigram or TF-IDF features performed best, at about 0.66 for both accuracy and F1. The same model reached higher overall scores on the comparable MAWQIF dataset. However, its neutral class collapsed there, while on MARASTA performance stayed even across all three labels. The authors take this as evidence that label balance matters when building stance resources.

Key points

  • The corpus has 4,657 sentences across four regions, with at least 1,000 per region and about 500 per topic. There are eight topics, two per region.
  • Topics were proposed by native-speaker team members and then voted on. The criteria were:
    • the topic was clearly controversial;
    • it was relevant within the last 15 years;
    • its scope could be one country or several.
  • Sentences were gathered with seed keywords and Python scripts that called social-media APIs.
  • The best model was an MLP with Unigram or TF-IDF features, scoring accuracy 0.66 and macro F1 0.66. Stacking several feature types did worse than the best single feature set.
  • The model was compared with MAWQIF:
    • On MAWQIF it scored accuracy 0.73 and F1 0.71, but neutral-class F1 was only 0.26.
    • On MARASTA, per-class F1 was 0.66 for pro, 0.65 for against and 0.66 for neutral.
  • The stated limitations are:
    • the corpus is small;
    • Tunisian is the only Maghreb dialect included;
    • some dialects, such as Iraqi and Yemeni, are not covered.

Interesting ideas

  • I1. Looking at per-class precision, recall and F1, rather than one headline number, showed that the higher overall score on MAWQIF hid a failing minority class.
  • I2. Every topic was split evenly between the standard variety and a local variety, so the effect of register can be measured directly rather than confounded with topic.
  • I3. Annotation relied on two independent labellers, and a third was brought in only when they disagreed. This keeps cost low while still resolving hard cases.
  • I4. Stance was annotated with balanced pro, against and neutral labels toward an explicitly named topic, across many different subjects.
  • I5. Adding more feature types together performed worse than picking the single best feature set.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation reports should break scores down by label, such as each UD relation, frame role or speech-act type. This keeps rare categories from being masked by strong frequent ones.
  • I2 (tree): Parser test sets could pair formal English with informal or social-media English on the same subjects. This would isolate how much gparse or the perceptron parser loses on non-standard text.
  • I3 (learn): Silver data could apply the same tie-breaker pattern. Two witness parsers vote, and a third parser or an LLM teacher tier is called only on arcs where they conflict.
  • I4 (frames): A small balanced stance set, built on varied topics, could test whether prag correctly reads a speaker's attitude toward a named target. It would also help separate genuine neutrality from implied positions.
  • I5 (learn): Each new feature or signal should go through an ablation against the current best single configuration before being combined with others.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Anis Charfi, Mabrouka Bessghaier, Andria Atalla, Raghda Akasheh, Sara Al-Emadi and Wajdi Zaghouani. 2024. MARASTA: A Multi-dialectal Arabic Cross-domain Stance Corpus. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{charfi2024marasta,
  title = {{MARASTA: A Multi-dialectal Arabic Cross-domain Stance Corpus}},
  author = {Anis Charfi, Mabrouka Bessghaier, Andria Atalla, Raghda Akasheh, Sara Al-Emadi and Wajdi Zaghouani},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/39rrei99qj8g},
  url = {https://doi.org/10.63317/39rrei99qj8g}
}

References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. Daict: A dialectal arabic irony corpus extracted from twitter. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6265–6271.
  2. Arfan Ahmed, Nashva Ali, Mahmood Alzubaidi, Wajdi Zaghouani, Alaa A Abd-alrazaq, and Mowafa Househ. 2022. Freely available arabic corpora: A scoping review. Computer Methods and Programs in Biomedicine Update, 2:100049. ↗
  3. Abeer AlDayel and Walid Magdy. 2021. Stance detection on social media: State of the art and trends. Inf. Process. Manag., 58(4):102597. ↗
  4. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184.
  5. Christopher A Bail, Lisa P Argyle, Taylor W Brown, John P Bumpus, Haohan Chen, MB Fallin Hunzaker, Jaemin Lee, Marcus Mann, Friedolin Merhout, and Alexander Volfovsky. 2018. Exposure to opposing views on social media can increase political polarization. Proceedings of the National Academy of Sciences, 115(37):9216– 9221. ↗
  6. Ramy Baly, Mitra Mohtarami, James Glass, Lluís Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact checking in a unified corpus. arXiv preprint arXiv:1804.08012. ↗
  7. Douglas Biber and Edward Finegan. 1988. Adverbial stance types in english. Discourse Processes, 11(1):1–34. ↗
  8. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  9. Rong Cao, Xiangyang Luo, Yaoyi Xi, and Yaqiong Qiao. 2022. Stance detection for online public opinion awareness: An overview. International Journal of Intelligent Systems, 37(12):11944– 11965. ↗
  10. Anis Charfi, Wajdi Zaghouani, Syed Hassan Mehdi, and Esraa Mohamed. 2019. A fine-grained annotated multi-dialectal arabic corpus. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 198–204.
  11. Kareem Darwish, Walid Magdy, and Tahar Zanouda. 2017. Improved stance prediction in a user similarity feature space. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, ASONAM ’17, page 145– 148, New York, NY, USA. Association for Computing Machinery. find
  12. Tim Draws, Karthikeyan Natesan Ramamurthy, Ioana Baldini, Amit Dhurandhar, Inkit Padhi, Benjamin Timmermans, and Nava Tintarev. 2023. Explainable cross-topic stance detection for search results. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval, pages 221–235. find
  13. Seth Flaxman, Sharad Goel, and Justin M Rao. 2016. Filter bubbles, echo chambers, and online news consumption. Public opinion quarterly, 80(S1):298–320. ↗
  14. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, et al. 2018. Unified guidelines and resources for arabic dialect orthography. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  15. Mohammad Mehdi Jaziriyan, Ahmad Akbari, and Hamed Karbasi. 2021. ExaASC: A general target-based stance detection corpus in arabic language. In 2021 11th International Conference on Computer Engineering and Knowledge (ICCKE). IEEE.
  16. Dilek Küçük and Fazli Can. 2021. Stance detection: A survey. ACM Comput. Surv., 53(1):12:1– 12:37. find
  17. Yingjie Li and Cornelia Caragea. 2019. Multi-task stance detection with sentiment and stance lexicons. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6299–6305, Hong Kong, China. Association for Computational Linguistics.
  18. Nolan McCarty. 2019. Polarization: What everyone needs to know®. Oxford University Press. find
  19. Dana Mckay, Stephann Makri, Marisela Gutierrez-Lopez, Andrew MacFarlane, Sondess Missaoui, Colin Porlezza, and Glenda Cooper. 2020. We are the change that we seek: Information interactions during a change of viewpoint. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, CHIIR ’20, page 173–182, New York, NY, USA. Association for Computing Machinery. find
  20. Francisco Rangel, Paolo Rosso, Wajdi Zaghouani, and Anis Charfi. 2020. Fine-grained analysis of language varieties and demographics. Natural Language Engineering, 26(6):641–661. ↗
  21. Paolo Rosso, Francisco Rangel, Irazu Hernández Farías, Leticia Cagnina, Wajdi Zaghouani, and Anis Charfi. 2018. A survey on author profiling, deception, and irony detection for the arabic language. Language and Linguistics Compass, 12(4):e12275. ↗
  22. Apoorva Upadhyaya, Marco Fisichella, and Wolfgang Nejdl. 2023. A multi-task model for sentiment aided stance detection of climate change tweets. In Proceedings of the International AAAI Conference on Web and Social Media, volume 17, pages 854–865. find
  23. Ingmar Weber, Venkata R Kiran Garimella, and Alaa Batayneh. 2013. Secular vs. islamist polarization in egypt on twitter. In Proceedings of the 2013 IEEE/ACM international conference on advances in social networks analysis and mining, pages 290–297. find
  24. Wajdi Zaghouani. 2014. Critical survey of the freely available arabic corpora. In International Conference on Language Resources and Evaluation (LREC’2014), OSACT Workshop. Reykjavik, Iceland, 26-31 May 2014. ↗
  25. Wajdi Zaghouani, Houda Bouamor, Abdelati Hawwari, Mona Diab, Ossama Obeid, Mahmoud Ghoneim, Sawsan Alqahtani, and Kemal Oflazer. 2016a. Guidelines and framework for a large scale arabic diacritized corpus. In The Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3637–3643. European Language Resources Association (ELRA).
  26. Wajdi Zaghouani and Anis Charfi. 2018. Guidelines and annotation framework for arabic author profiling. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). ↗
  27. Wajdi Zaghouani, Nizar Habash, Ossama Obeid, Behrang Mohit, Houda Bouamor, and Kemal Oflazer. 2016b. Building an arabic machine translation post-edited corpus: Guidelines and annotation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1869–1876.
  28. Wajdi Zaghouani, Behrang Mohit, Nizar Habash, Ossama Obeid, Nadi Tomeh, Alla Rozovskaya, Noura Farra, Sarah Alkuhlani, and Kemal Oflazer. 2014. Large scale arabic error annotation: Guidelines and framework. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/charfi-2024-marasta-multi-dialectal-arabic-cross/