Elamur Research⌕
☰
home / papers / 2024 / ThatiAR: Subjectivity Detection in Arabic News Sentences

ThatiAR: Subjectivity Detection in Arabic News Sentences

Summary

Most subjectivity detection research has targeted English and other well-resourced languages. Arabic news has had no large dedicated dataset. The authors built ThatiAR, about 3.6K Arabic news sentences labeled by crowd workers as subjective or objective. Source articles came from fact-checked claims in AraFacts, plus a hand-picked set of opinion pieces added to counter the dominance of factual sentences.

Candidate sentences were filtered by length. The pool was then enriched with likely-subjective cases by keeping any sentence that at least one model in a classifier ensemble flagged as subjective. Annotation ran on Amazon Mechanical Turk, with gold questions used both to admit workers and to check them during the task. The analysis shows that annotators' political, cultural and religious backgrounds shaped their labels, most strongly early in the process.

The dataset also includes LLM-written explanations for each label and an instruction-tuning version. Benchmarks cover PLMs and GPT-4. Few-shot GPT-4 clearly outperformed fine-tuned encoders. The resource supports media-bias analysis and fact-checking in Arabic.

Key points

  • Collection drew on 1,159 articles from AraFacts and 221 manually found opinion articles. Pages were scraped with Goose3, Newspaper3k and Trafilatura, keeping the longest extraction, which yielded 15,947 sentences.
  • Selection kept sentences of 10–45 words that received at least one "subjective" vote from ARABERTv2, ARBERTv2, MARBERTv2, GIGABERTv4 and GPT-4. These models were trained on CheckThat! 2023 Arabic subjectivity data, and the step produced a pool of 4,524 sentences.
  • Quality control used guidelines adapted from Antici et al. (2021), pilot rounds consuming roughly 850 sentences, and 115 gold sentences. Of these, 10 formed a qualification test and 105 served as ongoing checks, with workers required to stay above 60% accuracy. The job ran as 245 HITs of at most 15 sentences each.
  • GPT-4 accuracy rose from 0.768 at 0-shot to 0.795 at 3-shot. The best fine-tuned PLM, AraBERT, reached only F1 0.566. Arabic-only training worked best for mBERT and RoBERTa.
  • The release adds Arabic and English rationales for each label. It also adds GPT-4o-generated instructions in three styles: simple, straightforward and detailed.

Interesting ideas

  • I1. When several extraction tools disagree, the authors keep the longest extracted text, a cheap heuristic against truncated scraping. — For Mova (tokens): Running more than one cleaner or segmenter on raw web input and keeping the most complete result could reduce broken sentences before tagging.
  • I2. They enrich a rare class by accepting any sentence that even one weak model flags, so human effort goes to cases most likely to matter. — For Mova (learn): Sentences where even one witness parser deviates could be routed to the LLM teacher tier, focusing costly labeling on contested structures.
  • I3. Known-answer items are hidden inside every work batch, and annotators are dropped if their running accuracy falls below a threshold. — For Mova (learn): Seeding LLM teacher batches with gold trees or frames and tracking accuracy on them would give a built-in negative control on teacher drift.
  • I4. Prompt instructions generated for the dev set are randomly reused for training, while test instructions stay independently generated, which cuts cost without leaking prompts into evaluation. — For Mova (answer): Test questions for the readers could use phrasings disjoint from those seen in tuning, checking that answers do not depend on memorized wording.
  • I5. The guidelines label sayings quoted from someone else as objective, so subjectivity is tied to the writer's own voice rather than the words alone. — For Mova (frames): The prag crate could attach stance and evaluation to the speaker of a reported-speech frame, separating the narrator's opinion from quoted opinions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.05559 · arXiv 2406.05559

@misc{suwaileh2024thatiar,
  title = {{ThatiAR: Subjectivity Detection in Arabic News Sentences}},
  author = {Suwaileh, Reem and Hasanain, Maram and Hubail, Fatema and Zaghouani, Wajdi and Alam, Firoj},
  year = {2024},
  doi = {10.48550/arxiv.2406.05559},
  eprint = {2406.05559},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.05559}
}

References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684. ↗ our card
  2. Muhammad Abdul-Mageed. 2015. Subjectivity and sentiment analysis of Arabic as a morophologically-rich language. Ph.D. thesis, Indiana University. find
  3. Muhammad Abdul-Mageed and Mona Diab. 2011. Subjectivity and sentiment annotation of modern standard arabic newswire. In Proceedings of the 5th linguistic annotation workshop, pages 110–118. find
  4. Muhammad Abdul-Mageed, Mona Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of modern standard arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 587–591. find
  5. Muhammad Abdul-Mageed, Mona Diab, and Sandra Kübler. 2014. SAMAR: subjectivity and sentiment analysis for arabic social media. Computer Speech & Language, 28(1):20–37. find
  6. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗ our card
  7. Mohamed Elhag M Abo, Nordiana Ahmad Khar-man Shah, Vimala Balakrishnan, Mohamed Kamal, Ahmed Abdelaziz, and Khalid Haruna. 2019. Ssa-sda: subjectivity and sentiment analysis of sudanese dialect arabic. In 2019 International Conference on Computer and Information Sciences (ICCIS), pages 1–5. IEEE. find
  8. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗ our card
  9. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics. find our card
  10. Firoj Alam, Shaden Shaar, Fahim Dalvi, Hassan Sajjad, Alex Nikolov, Hamdy Mubarak, Giovanni Da San Martino, Ahmed Abdelali, Nadir Durrani, Kareem Darwish, Abdulaziz Al-Homaid, Wajdi Zaghouani, Tommaso Caselli, Gijs Danoe, Friso Stolk, Britt Bruntink, and Preslav Nakov. 2021. Fighting the COVID-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 611–649, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  11. Francesco Antici, Luca Bolognini, Matteo Antonio In-ajetovic, Bogdan Ivasiuk, Andrea Galassi, and Federico Ruggeri. 2021. Subjectivita: An italian corpus for subjectivity detection in newspapers. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 40–52, Cham. Springer International Publishing. find
  12. Francesco Antici, Federico Ruggeri, Andrea Galassi, Katerina Korre, Arianna Muti, Alessandra Bardi, Alice Fedotova, and Alberto Barrón-Cedeño. 2024. A corpus for sentence-level subjectivity detection on English news articles. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 273–285, Torino, Italia. ELRA and ICCL. find our card
  13. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. find our card
  14. Aqil M Azmi and Samah M Alzanin. 2014. Aara’–a system for mining the polarity of saudi public opinion through e-newspaper comments. Journal of Information Science, 40(3):398–410. find
  15. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675—-718, Indonesia. Association for Computational Linguistics. find our card
  16. Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 335–336. find
  17. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  18. Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, and Ahmed Ali. 2024. LLMeBench: A flexible framework for accelerating LLMs benchmarking. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 214–222, St. Julians, Malta. Association for Computational Linguistics. find
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  20. Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018. Addressing age-related bias in sentiment analysis. In Proceedings of the 2018 chi conference on human factors in computing systems, pages 1–14. find
  21. Andrea Galassi, Federico Ruggeri, Alberto Barrón-Cede no, Firoj Alam, Tommaso Caselli, Mucahid Kutlu, Julia Maria Struss, Francesco Antici, Maram Hasanain, Juliane Köhler, Katerina Korre, Folkert Leistra, Arianna Muti, Melanie Siegel, Mehmet Deniz Turkmen, Michael Wiegand, and Wajdi Zaghouani. 2023. Overview of the CLEF-2023 CheckThat! lab task 2 on subjectivity in news articles. In Working Notes of CLEF 2023– Conference and Labs of the Evaluation Forum, CLEF ’2023, Thessaloniki, Greece. find
  22. Nizar Habash, Behrang Mohit, Ossama Obeid, Kemal Oflazer, Nadi Tomeh, and Wajdi Zaghouani. 2013. Qalb: Qatar arabic language bank. In Proceedings of Qatar Annual Research Conference (ARC-2013), pages ICTP–032, Doha, Qatar. find
  23. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are GPT models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. ↗
  24. Hairong Huo and Mizuho Iwaihara. 2020. Utilizing bert pretrained models with various fine-tune methods for subjectivity detection. In Web and Big Data: 4th International Joint Conference, APWeb-WAIM 2020, Tianjin, China, September 18-20, 2020, Proceedings, Part II 4, pages 270–284. Springer. find
  25. Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. GPTAraEval: A comprehensive evaluation of ChatGPT on Arabic NLP. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 220–247, Singapore. Association for Computational Linguistics. find
  26. Mohammed Korayem, David Crandall, and Muhammad Abdul-Mageed. 2012. Subjectivity and sentiment analysis of arabic: A survey. In Advanced Machine Learning Technologies and Applications: First International Conference, AMLTA 2012, Cairo, Egypt, December 8-10, 2012. Proceedings 1, pages 128–139. Springer. find
  27. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. Gigabert: Zero-shot transfer learning from english to arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP). find
  28. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  29. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. ↗
  30. Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020. Detecting stance in media on global warming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3296–3315, Online. Association for Computational Linguistics. find
  31. Ahmed Mourad and Kareem Darwish. 2013. Subjectivity and sentiment analysis of modern standard arabic and arabic microblogs. In Proceedings of the 4th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 55–64. find
  32. OpenAI. 2023. GPT-4 technical report. Technical report, OpenAI. find our card
  33. Eshrag Refaee and Verena Rieser. 2014a. An arabic twitter corpus for subjectivity and sentiment analysis. In LREC, pages 2268–2273. find
  34. Eshrag Refaee and Verena Rieser. 2014b. Subjectivity and sentiment analysis of arabic twitter feeds with limited resources. In Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools Workshop Programme, volume 16. find
  35. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. ↗
  36. Elena Savinova and Fermin Moscoso Del Prado. 2023. Analyzing subjectivity using a transformer-based regressor trained on naïve speakers’ judgements. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 305–314, Toronto, Canada. Association for Computational Linguistics. find
  37. Zien Sheikh Ali, Watheq Mansour, Tamer Elsayed, and Abdulaziz Al-Ali. 2021. AraFacts: The first large Arabic dataset of naturally occurring claims. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 231–236, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find
  38. TBA. 2024. Overview of the CLEF-2024 Check-That! lab task 2 on subjectivity in news articles. In Working Notes of CLEF 2024–Conference and Labs of the Evaluation Forum, CLEF ’2024, Grenoble - France. find
  39. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/suwaileh-2024-thatiar-subjectivity-detection-arabic-news/