Elamur Research⌕
☰
home / papers / 2023 / Towards Arabic Multimodal Dataset for Sentiment Analysis

Towards Arabic Multimodal Dataset for Sentiment Analysis

Summary

Arabic multimodal sentiment analysis (MSA) lags well behind work on English and other Western languages. The main obstacle is the lack of standard datasets that combine text, audio and video. The authors address this in two steps. First, they build a construction pipeline for an Arabic multimodal sentiment dataset. It chains state-of-the-art transformers (AraBERT for the text side) with feature-extraction tools and uses word alignment so that each modality's features are synchronized at the word level. Second, they check that the resulting resource is usable by training established multimodal models on it: the Multimodal Transformer (MulT) and an LS-LSTM-based model, both with early fusion of modalities. The dataset is small. Even so, the experiments indicate that combining modalities pays off for Arabic sentiment classification, and the authors argue multimodality is essential for accurate Arabic SA. The work matters because it provides a reproducible recipe for creating multimodal resources in an under-resourced language. It also points to the two most promising directions: more data and better alignment.

Key points

  • The paper offers two contributions: a dataset-building pipeline (transformers, feature extractors and word alignment) and a validation of the resulting Arabic multimodal dataset using transformer-based multimodal models.
  • Text is represented with AraBERT, a BERT-base-sized model (12 encoder blocks, 768 hidden dimensions, 12 heads, about 110M parameters) pretrained on roughly 70M sentences (24GB of Arabic text).
  • Modalities are combined by early fusion, meaning they are merged before the decision. In late fusion, by contrast, per-modality predictions are combined at the end.
  • Evaluation models are MulT (Tsai et al.) and LS-LSTM. Earlier Arabic MSA work was tiny and relied on classical classifiers: Najadat & Abushaqra used Weka and SVM/KNN/NB/NN, and Alqarafi et al. built AMMD from 40 YouTube videos and used SVM.
  • Planned future work is to enlarge the dataset to a size that suits deep learning and to improve the alignment step, which the authors expect to raise accuracy noticeably.

Interesting ideas

  • I1. Synchronizing all non-text signals to word boundaries gives each token one shared anchor that every feature stream attaches to, which makes heterogeneous annotations directly comparable. — For Mova (tokens): Any external layer, such as prosody marks, OCR confidence or LLM-teacher labels, could be stored as token-anchored attributes in the same way, so every downstream decision can point back to the exact token span it used.
  • I2. The paper explicitly contrasts early fusion (merge evidence, then decide) with late fusion (decide per source, then combine), which is a useful lens for any multi-source system. — For Mova (learn): The witness-parser agreement could be run in both modes, either voting on finished trees or merging arc scores before decoding, and each mode reported against negative controls to see which yields cleaner silver data.
  • I3. A newly built resource is validated by training established off-the-shelf models on it instead of only reporting corpus statistics, so the resource's usefulness is shown in practice. — For Mova (learn): Every new silver treebank or induced rule set could be accepted only after a standard baseline (the perceptron parser or gparse) trained on it improves on a held-out gold set.
  • I4. Emotion (a psychological state such as anger or surprise) is kept apart from sentiment (a polarity attitude built on emotion), instead of treating the two as one label. — For Mova (frames): The prag frames could carry separate slots for emotion and polarity, which would help irony detection, where positive surface polarity co-occurs with a negative emotion.
  • I5. The pipeline assembles a new resource mostly from existing pretrained tools chained together, keeping manual effort for labeling and quality checks. — For Mova (world): Compiled knowledge layers (verb classes, ontology links from WordNet/Wikidata) could be bootstrapped the same way, chaining existing tools and adding a reviewed explanation step for each automatic link.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

5 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.06322 · arXiv 2306.06322

@misc{haouhat2023arabic,
  title = {{Towards Arabic Multimodal Dataset for Sentiment Analysis}},
  author = {Haouhat, Abdelhamid and Bellaouar, Slimane and Nehar, Attia and Cherroun, Hadda},
  year = {2023},
  doi = {10.48550/arxiv.2306.06322},
  eprint = {2306.06322},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.06322}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Soleymani, D. Garcia, B. Jou, B. Schuller, S.-F. Chang, and M. Pantic, “A survey of multimodal sentiment analysis,” Image and Vision Computing, vol. 65, pp. 3–14, 2017. find
  2. A. Zadeh, “Micro-opinion sentiment intensity analysis and summarization in online videos,” in Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ser. ICMI ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 587–591. [Online]. Available: https://doi-org.sndl1.arn.dz/10.1145/2818346.2823317 ↗
  3. S. Poria, E. Cambria, N. Howard, G.-B. Huang, and A. Hussain, “Fusing audio, visual and textual clues for sentiment analysis from multimodal content,” Neurocomputing, vol. 174, pp. 50–59, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0925231215011297 ↗
  4. T. Diwan and J. V. Tembhurne, “Sentiment analysis: a convolutional neural networks perspective,” Multimedia Tools and Applications, vol. 81, no. 30, pp. 44 405–44 429, Dec 2022. [Online]. Available: https://doi.org/10.1007/s11042-021-11759-2 ↗
  5. J. V. Tembhurne and T. Diwan, “Sentiment analysis in textual, visual and multimodal inputs using recurrent neural networks,” Multimedia Tools and Applications, vol. 80, no. 5, pp. 6871–6910, Feb 2021. [Online]. Available: https://doi.org/10.1007/s11042-020-10037-x ↗
  6. S. A. Abdu, A. H. Yousef, and A. Salem, “Multimodal video sentiment analysis using deep learning approaches, a survey,” Information Fusion, vol. 76, pp. 204–226, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1566253521001299 ↗
  7. A. Ghallab, A. Mohsen, and Y. Ali, “Arabic sentiment analysis: A systematic literature review,” Applied Computational Intelligence and Soft Computing, vol. 2020, pp. 1–21, 2020. find
  8. H. Najadat and F. Abushaqra, “Multimodal sentiment analysis of arabic videos,” Journal of Image and Graphics, vol. 6, no. 1, 2018. find
  9. A. S. Alqarafi, A. Adeel, M. Gogate, K. Dashitpour, A. Hussain, and T. Durrani, “Toward’s arabic multi-modal sentiment analysis,” in Communications, Signal Processing, and Systems: Proceedings of the 2017 International Conference on Communications, Signal Processing, and Systems. Springer, 2019, pp. 2378–2386. find
  10. G. Vinodhini and R. Chandrasekaran, “Sentiment analysis and opinion mining: a survey,” International Journal, vol. 2, no. 6, pp. 282–292, 2012. find
  11. L. Yue, W. Chen, X. Li, W. Zuo, and M. Yin, “A survey of sentiment analysis in social media,” Knowledge and Information Systems, vol. 60, pp. 617–663, 2019. find
  12. L.-P. Morency, P. P. Liang, and A. Zadeh, “Tutorial on multimodal machine learning,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts. Seattle, United States: Association for Computational Linguistics, Jul. 2022, pp. 33–38. [Online]. Available: https://aclanthology.org/2022.naacl-tutorials.5 ↗
  13. P. P. Liang, A. Zadeh, and L.-P. Morency, “Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions,” 2022. [Online]. Available: https://arxiv.org/abs/2209.03430 ↗
  14. P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: a survey,” arXiv preprint arXiv:2206.06488, 2022. ↗
  15. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 8748–8763. [Online]. Available: https://proceedings.mlr.press/v139/radford21a.html ↗ our card
  16. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018. ↗
  17. J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532–1543. find
  18. W. Antoun, F. Baly, and H. Hajj, “Arabert: Transformer-based model for arabic language understanding,” arXiv preprint arXiv:2003.00104, 2020. ↗ our card
  19. Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” in Proceedings of the conference. Association for Computational Linguistics. Meeting, vol. 2019. NIH Public Access, 2019, p. 6558. find our card
  20. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. find
  21. A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2236–2246. find our card
  22. A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,” arXiv preprint arXiv:1606.06259, 2016. ↗
  23. T. Baltrušaitis, P. Robinson, and L.-P. Morency, “Openface: An open source facial behavior analysis toolkit,” in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), 2016, pp. 1–10. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haouhat-2023-towards-arabic-multimodal-dataset-sentiment/