Elamur Research⌕
☰
home / papers / 2015 / Benchmarking Machine Translated Sentiment Analysis for Arabic Tweets

Benchmarking Machine Translated Sentiment Analysis for Arabic Tweets

Paper · 2015 · Eshrag Refaee and Verena Rieser

Summary

The paper asks whether Arabic tweets can be classified for sentiment without building a dedicated Arabic system. Arabic social media mixes Modern Standard Arabic with dialects, and few labelled resources exist for it. The authors translate tweets into English with two free services, Google Translate and Microsoft Translator. The resulting English text is then labelled by Socher et al.'s (2013) Stanford classifier for sentiment, a recursive deep model trained on English phrases.

They built a new held-out gold set of 937 tweets, annotated by two native speakers. On it they compare the pipeline with three native-Arabic baselines: a lexicon-based system, a fully supervised SVM, and a distant-supervision SVM. The Microsoft-based pipeline reaches about 76% accuracy. That is roughly equal to the costly hand-built lexicon approach and well above the supervised and distant-supervision models.

An error analysis traces most failures to untranslated dialect words, mistranslated idioms, misspellings, cultural connotation and reordered sentence structure. Dialectal tweets score significantly worse than MSA ones. The study suggests that off-the-shelf MT is a cheap, credible route to sentiment analysis for under-resourced languages.

Key points

  • Evaluation set: 937 Arabic tweets (470 positive, 467 negative, 10,310 tokens), with inter-annotator Kappa of 0.81. Classification is binary.
  • Accuracy by system:
    • Microsoft Translator + SSC: 76.34%
    • Google Translate + SSC: 71.28%
    • Lexicon-based: 76.72%
    • Distant supervision: 57.06%
    • Fully supervised SVM on n-grams: 49.65%
  • The five-way SSC output was collapsed to two classes: very-positive was folded into positive and very-negative into negative.
  • Lexicon baseline resources:
    • A merged lexicon of 4,422 words, combining an Arabic subjectivity lexicon, a machine-translated and hand-cleaned MPQA, and 484 hand-annotated dialect words.
    • MADAMIRA lemmatisation, ±1 word scores, flipping of polarity under negation, and summing of scores.
    • Tweet coverage of 76.62%.
  • Microsoft beat Google on 108 tweets (11.5%), mainly by keeping sentence structure intact. AIDA tagging of MSA vs. dialect showed a significant link between language class and accuracy.

Interesting ideas

  • I1. Pivot the task into a richer language and reuse its strongest classifier, but score that pipeline on the same native gold set as the home-grown baselines, so its cost-to-accuracy trade-off is measured honestly.
  • I2. For error analysis, sample only items where both MT systems produced the same label, so translator quirks are separated from classifier faults, and study the disagreement items on their own to compare translators.
  • I3. To test a suspected cause, the authors hand-fixed the word order of a mistranslated tweet and reran the classifier; the label then flipped to the correct one.
  • I4. When building distant-supervision data from a lexicon, the cue words were swapped for placeholders, so the learner cannot just memorise the labelling rule.
  • I5. Break accuracy down by language variety (MSA vs. dialect) to locate where a pipeline breaks, instead of reporting one global number.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Mova's translation through the meta-world could be judged the same way: route foreign input into English, run the normal pipeline, and compare against a native baseline on one shared held-out set.
  • I2 (learn): when several witness parsers are compared, agreed-but-wrong cases can be analysed apart from contested ones, which helps tell model bias from input noise.
  • I3 (learn): self-induced repair rules could be checked with similar small interventions: apply the suspected fix and confirm that the downstream frame or answer changes as predicted.
  • I4 (learn): silver training data labelled by heuristics could mask the triggering cues, acting as a built-in control against shortcut learning.
  • I5 (learn): evaluation could be split by register or genre (fables, school texts, informal prose) to show which stage degrades on which kind of English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heriot-Watt UniversityEdinburgh · United Kingdom15 works · 5,365 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Eshrag Refaee and Verena Rieser. 2015. Benchmarking Machine Translated Sentiment Analysis for Arabic Tweets. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/n15-2010

@inproceedings{refaee2015benchmarking,
  title = {{Benchmarking Machine Translated Sentiment Analysis for Arabic Tweets}},
  author = {Eshrag Refaee and Verena Rieser},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop},
  year = {2015},
  doi = {10.3115/v1/n15-2010},
  url = {https://doi.org/10.3115/v1/n15-2010}
}

References

23 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abbasi, Ammar Hassan, and Milan Dhar. 2014. Benchmarking twitter sentiment analysis tools. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, may. European Language Resources Association (ELRA). find
  2. Muhammad Abdul-Mageed and Mona Diab. 2012. AWATIF: A multi-genre corpus for modern standard Arabic subjectivity and sentiment analysis. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA).
  3. Muhammad Abdul-Mageed, Mona T. Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of modern standard Arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2, HLT ’11, pages 587– 591, Stroudsburg, PA, USA. Association for Computational Linguistics.
  4. Muhammad Abdul-Mageed, Sandra Kuebler, and Mona Diab. 2012. SAMAR: A system for subjectivity and sentiment analysis of Arabic social media. In Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis, pages 19–28. Association for Computational Linguistics.
  5. Alexandra Balahur and Marco Turchi. 2013. Improving sentiment analysis in twitter using multilingual machine translated data. In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, pages 49–55, Hissar, Bulgaria, September. INCOMA Ltd. Shoumen, BUL-GARIA. find
  6. Mikhail Bautin, Lohit Vijayarenu, and Steven Skiena. 2008. International sentiment analysis for news and blogs. In ICWSM. find
  7. Ryan Cotterell and Chris Callison-Burch. 2014. A multi-dialect, multi-genre corpus of informal written Arabic. In The 9th edition of the Language Resources and Evaluation Conference, Reykjavik, Iceland, May. European Language Resources Association.
  8. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of NAACL-HLT, pages 359–369.
  9. Heba Elfardy, Mohamed Al-Badrashiny, and Mona Diab. 2014. AIDA: Identifying code switching in informal Arabic text. EMNLP 2014, page 94.
  10. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, pages 1–12. find
  11. Ahmed Mourad and Kareem Darwish. 2013. Subjectivity and sentiment analysis of modern standard Arabic and Arabic microblogs. WASSA 2013, page 55.
  12. A. Pak and P. Paroubek. 2010. Twitter as a corpus for sentiment analysis and opinion mining. In Proceedings of LREC.
  13. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, may. European Language Resources Association (ELRA).
  14. Eshrag Refaee and Verena Rieser. 2014a. An Arabic twitter corpus for subjectivity and sentiment analysis. In 9th International Conference on Language Resources and Evaluation (LREC’14).
  15. Eshrag Refaee and Verena Rieser. 2014b. Evaluating distant supervision for subjectivity and sentiment analysis on Arabic twitter feeds. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP).
  16. Eshrag Refaee and Verena Rieser. 2014c. Subjectivity and sentiment analysis of Arabic twitter feeds with limited resources. In Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools (OS-ACT). find
  17. Wael Salloum and Nizar Habash. 2012. Elissa: A dialectal to standard Arabic machine translation system. In COLING (Demos), pages 385–392.
  18. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631, page 1642. Citeseer.
  19. Carlo Strapparava, Oliviero Stock, and Ilai Alon. 2012. Corpus-based explorations of affective load differences in arabic-hebrew-english. In COLING (Posters), pages 1201–1208.
  20. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational linguistics, 37(2):267–307. ↗
  21. Xiaojun Wan. 2009. Co-training for cross-lingual sentiment classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 235–243. Association for Computational Linguistics.
  22. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2009. Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis. Computational Linguistics, 35(3):399–433. ↗
  23. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 49– 59. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/refaee-2015-benchmarking-machine-translated-sentiment-analysis/