Elamur Research⌕
☰
home / papers / 2015 / Sentiment after Translation: A Case-Study on Arabic Social Media Posts

Sentiment after Translation: A Case-Study on Arabic Social Media Posts

Paper · 2015 · Mohammad Salameh, Saif Mohammad and Svetlana Kiritchenko

Summary

Most sentiment tools and lexicons exist only for English, so one shortcut for other languages is to translate the text into English and then analyse it. The authors measure how much sentiment this shortcut loses, using dialectal Arabic social media posts as the source language. They build a grid of conditions. Arabic posts are translated into English both by hand and by an SMT system. Every version, original or translated, is labelled for sentiment both by human annotators and by automatic systems. One of these systems is an Arabic port of the NRC-Canada system, which had won SemEval competitions. Comparing each condition against the manual Arabic labels shows that sentiment analysis of English translations, even machine translations, performs about as well as a state-of-the-art Arabic system. Translation pushes many polar posts toward neutral. Machine-translation errors also confuse human annotators more than they confuse the automatic classifier. The labelled parallel data and the Arabic lexicons are released, making this the first resource of its kind.

Key points

  • The design crosses translation type (manual vs. automatic) with labelling type (manual vs. automatic). All conditions are scored against Ar(Manl.Sent.), the human-labelled original Arabic.
  • Experiments cover two Arabic datasets, including the BBN dialectal Arabic–English parallel collection. The texts are treated as Levantine dialect.
  • The English system scores 66.51% on manual translations, close to the 68% human agreement there. On machine translations it scores 69.58%, above the 65.7% human agreement.
  • Human sentiment labels on manual and automatic translations of the same posts agree only about 60% of the time.
  • Typical failures are ambiguous words translated with the wrong sense (scorpions rendered as clock arms), sarcasm, and Arabic typos. Some errors are systematic: a prayer asking God to grant victory is regularly rendered as God forsake.

Interesting ideas

  • I1. Lay out every combination of how the input is produced (human or machine) and how it is judged (human or machine), so the loss at each step can be measured separately.
  • I2. A trained model can exploit consistent machine errors, here a stable mistranslation, which a human reader sees as plain nonsense.
  • I3. Translation tends to flatten polarity toward neutral, so a label-shift matrix (positive→neutral, etc.) says more than accuracy alone.
  • I4. Comparing system accuracy with human inter-annotator agreement on the same data gives a ceiling. Here the system beating human agreement on degraded input was itself a finding.
  • I5. Many errors come from word-sense ambiguity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation could cross gold vs. predicted tokens/trees with gold vs. predicted frames, isolating how much error each pipeline stage contributes.
  • I2 (learn): When silver data comes from witness parsers, systematic witness errors may be learnable artefacts rather than random noise. They should be flagged so repair rules fix them instead of baking them in.
  • I3 (back): The text → graph → text compressor could be checked with a polarity/stance shift matrix between original and regenerated text, catching lost evaluative meaning.
  • I4 (answer): Report reader accuracy next to human agreement on paraphrased or noisy versions of the test texts, to show where degraded input rather than reasoning causes failures.
  • I5 (world): Linking lemmas to WordNet senses with explicit, explainable choices would make such sense errors visible in the world model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️National Research Council CanadaOttawa · Canada60 works · 21,790 citations🎓University of AlbertaEdmonton · Canada51 works · 5,017 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksSemitic languages · 3,066 works

Cite

Cite as Mohammad Salameh, Saif Mohammad and Svetlana Kiritchenko. 2015. Sentiment after Translation: A Case-Study on Arabic Social Media Posts. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/n15-1078

@inproceedings{salameh2015sentiment,
  title = {{Sentiment after Translation: A Case-Study on Arabic Social Media Posts}},
  author = {Mohammad Salameh, Saif Mohammad and Svetlana Kiritchenko},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1078},
  url = {https://doi.org/10.3115/v1/n15-1078}
}

References

41 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Mona Diab, and Sandra Kübler. 2014. SAMAR: Subjectivity and sentiment analysis for Arabic social media. Computer Speech & Language, 28(1):20 – 37. find
  2. Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 2011. Sentiment analysis of Twitter data. In Proceedings of the Workshop on Languages in Social Media, LSM ’11, pages 30–38, Portland, Oregon.
  3. Khurshid Ahmad, David Cheng, and Yousif Almas. 2006. Multi-lingual sentiment analysis of financial news streams. In Proceedings of the 1st International Conference on Grid in Finance. find
  4. Mohammed Al-Kabi, Amal Gigieh, Izzat Alsmadi, Heider Wahsheh, and Mohamad Haidar. 2013. An opinion analysis tool for colloquial and standard Arabic. In Proceedings of the 4th International Conference on Information and Communication Systems, ICICS ’13. find
  5. Massih-Reza Amini and Cyril Goutte. 2010. A co-classification approach to learning from multilingual corpora. Machine learning, 79(1-2):105–121. find
  6. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale Arabic sentiment lexicon for Arabic opinion mining. In Proceedings of the EMNLP Workshop on Arabic Natural Language Processing (ANLP), pages 165–173. Association for Computational Linguistics.
  7. Alexandra Balahur and Marco Turchi. 2014. Comparative experiments using supervised learning and machine translation for multilingual sentiment analysis. Computer Speech & Language, 28(1):56–75. find
  8. Jerome Bellegarda. 2010. Emotion analysis using latent affective folding and embedding. In Proceedings of the NAACL-HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 1–9, Los Angeles, California. find
  9. Anthony C. Boucouvalas. 2002. Real time text-to-emotion engine for expressive Internet communication. Emerging Communication: Studies on New Technologies and Practices in Communication, 5:305– 318. find
  10. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1–27:27. find
  11. Boxing Chen and Xiaodan Zhu. 2014. Bilingual sentiment consistency for statistical machine translation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 607–615, Gothenburg, Sweden, April. Association for Computational Linguistics. find
  12. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 427–436. Association for Computational Linguistics.
  13. David Chiang, Yuval Marton, and Philip Resnik. 2008. Online large-margin training of syntactic and structural translation features. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 224–233. Association for Computational Linguistics.
  14. Samhaa R El-Beltagy and Ahmed Ali. 2013. Open issues in the sentiment analysis of Arabic social media: A case study. In Proceedings of the 9th International Conference on Innovations in Information Technology, pages 215–220. IEEE. find
  15. Ahmed El Kholy and Nizar Habash. 2012. Orthographic and morphological processing for English— Arabic statistical machine translation. Machine Translation, 26(1-2):25–45, March. find
  16. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL ’11, pages 42–47.
  17. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. Technical report, Stanford University. find
  18. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. MADA+TOKAN: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, POS tagging, stemming and lemmatization. In Proceedings of the 2nd International Conference on Arabic Language Resources and Tools, pages 102–109, Cairo, Egypt, April. The MEDAR Consortium. find
  19. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, pages 168–177, New York, NY, USA. ACM. find
  20. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, 50:723–762. find
  21. Hugo Liu, Henry Lieberman, and Ted Selker. 2003. A model of textual affect sensing using real-world knowledge. In Proceedings of the 8th International Conference on Intelligent User Interfaces, IUI ’03, pages 125–132, New York, NY. ACM. find
  22. Eugenio Martı́nez-Cámara, M Teresa Martı́n-Valdivia, L Alfonso Ureñalópez, and A Rturo Montejoráez. 2012. Sentiment analysis in Twitter. Natural Language Engineering, pages 1–28. find
  23. Rada Mihalcea, Carmen Banea, and Janyce Wiebe. 2007. Learning multilingual subjective language via cross-lingual projections. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, page 976.
  24. Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL-HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 26–34, LA, California.
  25. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436–465. ↗
  26. Saif M. Mohammad and Tony (Wenda) Yang. 2011. Tracking sentiment in mail: How genders differ on emotional axes. In Proceedings of the ACL Workshop on Computational Approaches to Subjectivity and Sentiment Analysis, WASSA ’11, pages 70–79, Portland, OR, USA. find
  27. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the 7th International Workshop on Semantic Evaluation Exercises, SemEval ’13, Atlanta, Georgia, USA, June.
  28. Saif M. Mohammad. 2012. #Emotional tweets. In Proceedings of the First Joint Conference on Lexical and Computational Semantics, *SEM ’12, pages 246–255, Montréal, Canada.
  29. Ahmed Mourad and Kareem Darwish. 2013. Subjectivity and sentiment analysis of modern standard Arabic and Arabic microblogs. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, WASSA ’13, pages 55–64.
  30. Alena Neviarouskaya, Helmut Prendinger, and Mitsuru Ishizuka. 2011. Affect analysis model: novel rule-based approach to affect sensing from text. Natural Language Engineering, 17:95–135, 1. find
  31. Jian-Yun Nie, Michel Simard, Pierre Isabelle, and Richard Durand. 1999. Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the Web. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 74–81. ACM. find
  32. Alexander Pak and Patrick Paroubek. 2010. Twitter as a corpus for sentiment analysis and opinion mining. In Proceedings of the 7th Conference on International Language Resources and Evaluation, LREC ’10, pages 1320–1326, Valletta, Malta, May.
  33. Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 Task 4: Aspect based sentiment analysis. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’14, pages 27–35, Dublin, Ireland, August.
  34. Eshrag Refaee and Verena Rieser. 2014a. An Arabic Twitter corpus for subjectivity and sentiment analysis. In Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC ’14, Reykjavik, Iceland, May. European Language Resources Association.
  35. Eshrag Refaee and Verena Rieser. 2014b. Subjectivity and sentiment analysis of Arabic Twitter feeds with limited resources. In Proceedings of the Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools, page 16. find
  36. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 Task 9: Sentiment analysis in Twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval ’14, pages 73–80, Dublin, Ireland, August.
  37. Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. 2011. Sentiment in Twitter events. Journal of the American Society for Information Science and Technology, 62(2):406–418. find
  38. Xiaojun Wan. 2008. Using bilingual knowledge and ensemble techniques for unsupervised Chinese sentiment analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 553–561, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  39. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 347–354, Stroudsburg, PA, USA.
  40. Theresa Wilson, Zornitsa Kozareva, Preslav Nakov, Sara Rosenthal, Veselin Stoyanov, and Alan Ritter. 2013. SemEval-2013 Task 2: Sentiment analysis in Twitter. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’13, Atlanta, Georgia, USA, June.
  41. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 49–59. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salameh-2015-sentiment-after-translation-case-study/