Elamur Research⌕
☰
home / papers / 2024 / StanceCrafters at StanceEval2024: Multi-task Stance Detection using BERT Ensemble with Attention Based Aggregation

StanceCrafters at StanceEval2024: Multi-task Stance Detection using BERT Ensemble with Attention Based Aggregation

Summary

The paper describes a system for the StanceEval2024 shared task. The task is to label the attitude of Arabic tweets toward three targets: COVID-19 vaccination, digital transformation and women's empowerment. Labeled Arabic stance data is scarce. The authors therefore train stance jointly with two related labels from the MAWQIF corpus, sentiment and sarcasm, using hard parameter sharing. The shared encoder is not one model but a pair of fine-tuned BERT variants, AraBERT-twitter (trained on tweets) and CAMeLBERT (trained on MSA, dialectal and classical Arabic). Their outputs are merged by either a simple mean or a learned attention module. Each task then gets its own output head. They also compare equal, static and learnable ("relative") loss weighting. The best configuration uses attention fusion with static weights. It reaches 91.37 validation F1 and 81.68 macro F1 on the blind test set. The work shows that combining encoders trained on different kinds of text beats any single encoder, even when one member is individually weak.

Key points

  • Data: MAWQIF, 4,121 Arabic tweets annotated for stance, sentiment and sarcasm, split 85% train and 15% test. Only Favour/Against stance labels were used in training. Preprocessing used ArabertPreprocessor plus removal of hashtags, URLs, mentions, punctuation and repeated characters.
  • Single encoders on validation: AraBERT-twitter scored best (89.89 F1) and CAMeLBERT scored worst (84.92). Averaging exactly these two gave 90.23, the top two-model pair.
  • With attention fusion on that pair, the loss-weighting schemes scored as follows: equal 90.85, static 91.37 and relative 90.71. The static weights were 0.6 for stance, 0.3 for sentiment and 0.1 for sarcasm.
  • Blind test macro F1: Women Empowerment 85.06, Digital Transformation 80.14, COVID Vaccine 79.84, overall 81.68.
  • Training setup: 20 epochs, learning rate 2e-5, weight decay 1e-5, dropout 0.1, batch size 4, max length 128. Code is open on GitHub (StanceDetection-MultiTaskLearning).

Interesting ideas

  • I1. The strongest ensemble paired the best single model with the weakest one, so diversity of training domain mattered more than individual strength. — For Mova (learn): when choosing witness parsers for silver data, favour witnesses trained on different genres or treebanks over several near-identical strong ones.
  • I2. Instead of fixed averaging, a small attention module decides per input how much each encoder contributes, which slightly beat the mean. — For Mova (tree): a per-sentence learned weighting between the perceptron parser and gparse could replace fixed voting, and logging the weights would keep each choice explainable.
  • I3. Auxiliary tasks were trained alongside the main one with fixed, main-task-heavy loss weights, and these outperformed learnable weights; the authors attribute this to noisy, low-confidence labels. — For Mova (tree): gparse could add tagging or morphology heads as auxiliaries with hand-set small weights, avoiding unstable dynamic weighting when silver labels are noisy.
  • I4. Stance, sentiment and sarcasm were learned together on the assumption that they inform one another. — For Mova (frames): the irony detector in prag could draw on evaluative polarity cues from the same frame, since a mismatch between surface sentiment and stance is a natural irony signal.
  • I5. The authors linked the weakest target to a training/evaluation label mismatch: None was dropped in training, and the COVID subset contains the most None tweets. — For Mova (answer): evaluation sets should be broken down by how often "no answer / cannot tell" occurs, so that a reader never trained to abstain is not misjudged on abstention-heavy subsets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.94

@inproceedings{hasanaath2024stancecrafters,
  title = {{StanceCrafters at StanceEval2024: Multi-task Stance Detection using BERT Ensemble with Attention Based Aggregation}},
  author = {Hasanaath, Ahmed and Alansari, Aisha},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.94},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.94}
}

References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shams Forruque Ahmed, Md Sakib Bin Alam, Maruf Hassan, Mahtabin Rodela Rozbu, Taoseef Ishtiak, Nazifa Rafa, M Mofijur, ABM Shawkat Ali, and Amir H Gandomi. 2023. Deep learning modelling techniques: current progress, applications, advantages, and challenges. Artificial Intelligence Review, 56(11):13521–13617. find
  2. Abeer AlDayel and Walid Magdy. 2021. Stance detection on social media: State of the art and trends. Information Processing & Management, 58(4):102597. find
  3. Nora Alturayeif, Hamzah Luqman, and Moataz Ahmed. 2023. Enhancing stance detection through sequential weighted multi-task learning. Social Network Analysis and Mining, 14(1):7. find
  4. Nora Alturayeif, Hamzah Luqman, Zaid Alyafeai, and Asma Yamani. 2024. Stanceeval 2024: The first arabic stance detection shared task. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024). find our card
  5. Ting Gong, Tyler Lee, Cory Stephenson, Venkata Renduchintala, Suchismita Padhy, Anthony Ndirango, Gokce Keskin, and Oguz H Elibol. 2019. A comparison of loss weighting strategies for multi task learning in deep neural networks. IEEE Access, 7:141627– 141632. find
  6. Dilek Küçük and Fazli Can. 2021. Stance detection: Concepts, approaches, resources, and outstanding issues. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  7. Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018. Jointly multiple events extraction via attention-based graph information aggregation. arXiv preprint arXiv:1809.09078. ↗ our card
  8. Khouloud Mnassri, Praboda Rajapaksha, Reza Farah-bakhsh, and Noel Crespi. 2022. Bert-based ensemble approaches for hate speech detection. In GLOBE-COM 2022-2022 IEEE Global Communications Conference, pages 4649–4654. IEEE. find
  9. Jonathan Pilault, Amine Elhattami, and Christopher Pal. 2020. Conditionally adaptive multi-task learning: Improving transfer learning in nlp using fewer parameters & less data. arXiv preprint arXiv:2009.09139. ↗
  10. Kim-Han Thung and Chong-Yaw Wee. 2018. A brief review on multi-task learning. Multimedia Tools and Applications, 77(22):29705–29725. find
  11. Yu Zhang and Qiang Yang. 2021. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586–5609. find
  12. Nora Saleh Alturayeif, Hamzah Abdullah Luqman, and Moataz Aly Kamaleldin Ahmed. 2022. Mawqif: A multi-label arabic dataset for target-specific stance detection. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 174–184. find
  13. J Briskilal and CN Subalalitha. 2022. An ensemble model for classifying idioms and literal texts using bert and roberta. Information Processing & Management, 59(1):102756. find
  14. Huong Dang, Kahyun Lee, Sam Henry, and Ozlem Uzuner. 2020. Ensemble bert for classifying medication-mentioning tweets. In Proceedings of the Fifth Social Media Mining for Health Applications Workshop & Shared Task, pages 37–41. find
  15. Hang-Yuan Du and Wen-Jian Wang. 2022. A clustering ensemble framework with integration of data characteristics and structure information: a graph neural networks approach. Mathematics, 10(11):1834. find
  16. Ronen Feldman. 2013. Techniques and applications for sentiment analysis. Communications of the ACM, 56(4):82–89. find
  17. Mudasir A Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai N Sug-anthan. 2022. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hasanaath-2024-stancecrafters-stanceeval2024-multi-task-stance/