Elamur Research⌕
☰
home / papers / 2024 / Comparison of Multilingual and Bilingual Models for Satirical News Detection of Arabic and English

Comparison of Multilingual and Bilingual Models for Satirical News Detection of Arabic and English

Paper · 2024 · Omar W. Abdalla, Aditya Joshi, Rahat Masood et al.

Summary

The paper asks whether a model trained mainly on two languages handles satire better than a broadly multilingual one, and whether step-by-step reasoning prompts help. Satire matters here because it imitates real news and is easily mistaken for misinformation. The authors compare Jais-chat (13B, bilingual Arabic–English) with LLaMA-2-chat (7B, multilingual) on four article-level corpora: two Arabic (Assiri; Saadany, balanced with BBC Arabic news) and two English (Phosseini, SatiricLR). They use zero-shot prompting and a two-stage chain-of-thought (CoT) setup, which first asks for an analysis and then for a 0/1 label. Every corpus is tested with both English and Arabic instructions.

CoT helped Jais-chat in every configuration, with a top F1 of 80%. LLaMA-2-chat scored about the same under both prompting styles, with near-total recall. The authors note that the two models differ in more than language coverage, so the comparison cannot isolate the effect of bilingual training. The work offers early evidence on cross-lingual satire detection with prompted LLMs.

Key points

  • Datasets:
    • Assiri: 760 satirical and 765 real articles, Arabic.
    • Saadany: 3185 satirical articles, merged with 4763 non-satirical articles from "bbc-arabic-utf8" on SourceForge.
    • Phosseini: 3956 satirical and 2987 real articles, English.
    • SatiricLR: 1706 satirical and 1705 real articles, English.
  • Prompt grid: instruction language (English or Arabic) was crossed with article language, giving four configurations. Zero-shot prompts asked for only "1" or "0".
  • Jais-chat with English instructions on Assiri: F1 rose from 59.2 (zero-shot) to 80.0 (CoT).
  • Jais-chat on Saadany: zero-shot F1 was only 12.9 and rose to 66.7 with CoT. With Arabic instructions, Jais-chat's best F1 was about 70.
  • LLaMA-2-chat: F1 stayed between about 56 and 73 with recall of at least 97% everywhere. Its accuracy is close to the satire share of each dataset (≈50/40/57/50%), which suggests it labels nearly everything as satire.

Interesting ideas

  • I1. Crossing the instruction language with the input language in a full 2×2 design separates the effect of how the task is posed from the effect of what text is processed.
  • I2. Splitting CoT into two calls, a free-form analysis followed by a separate constrained labelling prompt, keeps the reasoning apart from the final decision, which is easier to inspect.
  • I3. Near-perfect recall combined with accuracy near the class prior is a warning sign of a degenerate always-positive classifier, so metrics should always be read against dataset class balance.
  • I4. The paper frames both error directions as harmful: calling real news satire dismisses true information, while calling satire real spreads falsehoods.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): When evaluating translation through the meta-world, cross source-text language against generation-target settings, so failures can be traced to parsing or to realisation.
  • I2 (frames): Irony and satire detection in prag could first emit an explicit analysis (expectation vs. stated frame, exaggeration cues) and then decide, keeping the verdict traceable.
  • I3 (learn): Add a majority-class / always-yes negative control to every classifier evaluation, so inflated F1 from one-sided prediction is flagged automatically.
  • I4 (answer): Before answering factual questions about a text, readers should mark whether the text is meant literally, and should report uncertainty rather than silently picking one reading.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓UNSW SydneySydney · Australia36 works · 1,509 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksPsychology · 1,649 worksSociology and Political Science · 626 worksSocial Psychology · 330 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,379 worksSemitic languages · 3,066 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Omar W. Abdalla, Aditya Joshi, Rahat Masood and Salil S. Kanhere. 2024. Comparison of Multilingual and Bilingual Models for Satirical News Detection of Arabic and English. Proceedings of the 22nd Annual Workshop of the Australasian Language Technology Association.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{abdalla2024comparison,
  title = {{Comparison of Multilingual and Bilingual Models for Satirical News Detection of Arabic and English}},
  author = {Omar W. Abdalla, Aditya Joshi, Rahat Masood and Salil S. Kanhere},
  booktitle = {Proceedings of the 22nd Annual Workshop of the Australasian Language Technology Association},
  year = {2024},
  url = {https://aclanthology.org/2024.alta-1.14/}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fatmah Assiri and Hanen Himdi. 2023. Comprehensive study of arabic satirical article classification. Applied Sciences, 13(19). ↗
  2. Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2015. Is this tweet satirical? a computational approach for satire detection in spanish. Procesamiento del Lenguaje Natural, (55):135–142. find
  3. Clint Burfoot and Timothy Baldwin. 2009. Automatic satire detection: Are you having a laugh? In Proceedings of the ACL-IJCNLP 2009 conference short papers, pages 161–164. find
  4. Alice Frain and Sander Wubben. 2016. SatiricLR: a language resource of satirical news articles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4137–4140, Portorož, Slovenia. European Language Resources Association (ELRA). find
  5. Or Levi, Pedram Hosseini, Mona Diab, and David Broniatowski. 2019. Identifying nuances in fake news vs. satire: Using semantic and linguistic cues. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 31–35, Hong Kong, China. Association for Computational Linguistics. find
  6. Lily Li, Or Levi, Pedram Hosseini, and David Broniatowski. 2020. A multi-modal method for satire detection using textual and visual cues. In Proceedings of the 3rd NLP4IF Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 33–38, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL). ↗
  7. Robert McHardy, Heike Adel, and Roman Klinger. 2019. Adversarial training for satire detection: Controlling for confounding variables. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 660–665, Minneapolis, Minnesota. Association for Computational Linguistics. find
  8. Ana-Cristina Rogoz, Gaman Mihaela, and Radu Tudor Ionescu. 2021. SaRoCo: Detecting satire in a novel Romanian corpus of news articles. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1073–1079, Online. Association for Computational Linguistics. find
  9. Hadeel Saadany, Constantin Orasan, and Emad Mohamed. 2020. Fake or real? a study of arabic satirical fake news. In Proceedings of the 3rd International Workshop on Rumours and Deception in Social Media (RDSM), pages 70–80. ↗
  10. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. ↗
  11. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdalla-2024-comparison-multilingual-bilingual-models-satirical/