Elamur Research⌕
☰
home / papers / 2023 / Raphael at ArAIEval Shared Task: Understanding Persuasive Language and Tone, an LLM Approach

Raphael at ArAIEval Shared Task: Understanding Persuasive Language and Tone, an LLM Approach

Paper · 2023 · Utsav Shukla, Manan Vyas and Shailendra Tiwari

Summary

The paper describes a team's entry to ArAIEval 2023 Shared Task 1, which targets persuasion and propaganda detection in Arabic, a language with little prior work on this problem. Subtask 1A asks whether a tweet or news paragraph contains any persuasion technique. Subtask 1B asks which of 24 techniques appear, as a multi-label problem. Using only the organizers' data, the authors removed emojis and "LINK" placeholders, then built on MARBERT's [CLS] embedding. Their main addition was prompting GPT-3.5 to write an English description of each Arabic text, including its tone and emotion. That English output was encoded with BERT or RoBERTa and concatenated with the MARBERT vector before classification. They also tried gating features by text source and a multi-task setup with learned loss weights. The LLM-derived features consistently beat the MARBERT baseline. The final systems ranked 5th on 1A and 6th on 1B (micro F1) on test. The work suggests LLM-generated descriptions can be useful supervised features for under-resourced languages.

Key points

  • Dev results for 1A: MARBERT alone reached micro/macro F1 0.8145/0.7192. MARBERT+GPT 3.5(RoBERTa) with the source feature gate reached 0.8610/0.7922.
  • Dev results for 1B: the baseline scored 0.6088/0.1996, and MARBERT+GPT 3.5(RoBERTa) scored 0.6399/0.2365. Adding the source gate did not help 1B, and multi-task training hurt it (0.5694/0.1602).
  • RoBERTa encoded the GPT-3.5 English outputs better than BERT did.
  • Test scores were 0.7475/0.7221 on 1A (best team: 0.7634/0.7321) and 0.5347/0.1772 on 1B (best team: 0.5666/0.2156). On 1B macro F1 the system ranked 4th.
  • The 1B labels are highly skewed: Loaded Language has 1574 training examples, while the rarest technique has 2.
  • Training is cheap: one 12 GB GPU, 5–15 minutes per model, up to 7 epochs, AdamW with lr 2e-5. Code, prompts and GPT-3.5 outputs are released.

Interesting ideas

  • I1. An LLM writes a natural-language note about each input's tone and likely persuasive intent, and a separate encoder turns that note into extra features alongside the native-text encoder.
  • I2. The authors use text provenance (tweet vs. news paragraph) as a multiplicative gate on features instead of simply appending it, and the gate worked better.
  • I3. The authors run an ablation ladder over two related tasks and find that one component (the gate, multi-task training) can help the coarse task while hurting the fine-grained one.
  • I4. One head detects whether something is present and another identifies which kind it is, with the two losses balanced by a learned weight.
  • I5. The authors publish the prompts and cached LLM outputs, so the expensive closed-model step does not need to be rerun.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): An LLM teacher tier could write short tone and intent glosses for silver data. These glosses could then be checked against prag's own irony and speech-act decisions, so disagreements flag cases for review rather than becoming opaque features.
  • I2 (tree): Treebank or genre identity could gate parser features in gparse, letting social-media and edited-prose inputs share one model while still behaving differently.
  • I3 (learn): Evaluations should report each added component separately per subtask, such as speech-act presence versus speech-act type, so that a gain on the easy level does not hide damage on the detailed level.
  • I4 (frames): Irony or implicature detection could be split into a "present?" decision and a "which type?" decision. Given the paper's results, the joint objective should be validated per head before it is adopted.
  • I5 (learn): Freezing and versioning all LLM teacher responses would make silver-data training reproducible and auditable, and would avoid repeated API cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Thapar Institute of Engineering & TechnologyPatiāla · India6 works · 11 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Utsav Shukla, Manan Vyas and Shailendra Tiwari. 2023. Raphael at ArAIEval Shared Task: Understanding Persuasive Language and Tone, an LLM Approach. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.60

@inproceedings{shukla2023raphael,
  title = {{Raphael at ArAIEval Shared Task: Understanding Persuasive Language and Tone, an LLM Approach}},
  author = {Utsav Shukla, Manan Vyas and Shailendra Tiwari},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.60},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.60}
}

References

17 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022. Overview of the WANLP 2022 shared task on propaganda detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 108–118, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  3. Galen Andrew and Jianfeng Gao. 2007. Scalable training of L 1 -regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33–40. ↗
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ↗
  5. Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414, Barcelona (online). International Committee for Computational Linguistics. ↗
  6. Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeño, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2021. A survey on computational propaganda detection. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI’20. find
  7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ↗
  9. Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. SemEval-2021 task 6: Detection of persuasion techniques in texts and images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 70–98, Online. Association for Computational Linguistics. ↗
  10. Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Alhakim Freihat. 2023. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023), Singapore. Association for Computational Linguistics.
  11. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
  13. Yi-Ju Lu and Cheng-Te Li. 2020. Gcan: Graph-aware co-attention networks for explainable fake news detection on social media.
  14. Panayot Panayotov, Utsav Shukla, Husrev Taha Sencar, Mohamed Nabeel, and Preslav Nakov. 2022. GREENER: Graph neural networks for news media profiling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7470–7480, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  15. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426, Vancouver, Canada. Association for Computational Linguistics.
  16. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. ↗
  17. Dimitrina Zlatkova, Preslav Nakov, and Ivan Koychev. 2019. Fact-checking meets fauxtography: Verifying claims about images. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2099–2108, Hong Kong, China. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shukla-2023-raphael-araieval-shared-task-understanding/