Elamur Research⌕
☰
home / papers / 2020 / Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation of Arabic Reviews

Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation of Arabic Reviews

Paper · 2020 · Hadeel Saadany and Constantin Orasan

Summary

Online NMT systems often mishandle sentiment in Arabic user-generated reviews. These reviews mix Modern Standard Arabic (MSA) with dialect, mostly Egyptian. Some words carry opposite polarity in the two varieties, so a translation can lose the evaluation entirely or reverse it.

The authors translated Goodreads book reviews (Aly and Atiya, 2013) with the Google Translate API. They then compared Microsoft Azure sentiment scores on the English output with each review's star rating to flag suspicious translations, and built an error typology from a manually analysed sample.

Next they trained NMT models aimed at these polarity-ambiguous words, which they call contronyms:

  • a baseline trained on post-edited data;
  • transformer models, the strongest of which combined sentiment-sensitive pretrained vectors with polarity tags on the source side.

On a hand-crafted test set, they scored models with word-level precision/recall/F1 and a sentence-level sentiment-distance cost. This model clearly beat Google Translate, which mostly failed on the dialectal positive senses. The study shows that BLEU can hide sentiment flips. It also shows that small but linguistically informed training data can help low-resource domains.

Key points

  • Data: the Goodreads Arabic review dataset, with long reviews cut into chunks of at most 20 tokens (about 230,000 sentences). Azure Text Analytics scored roughly 13,000 of the Google Translate outputs.

  • Errors fall into two kinds: the output turns neutral, or the polarity is fully inverted. Contronyms tend to be negative in MSA and positive in dialect, and the positive uses made up about 40% of test instances.

  • Word-level contronym results:

    SystemF1Precision
    Google Translate0.12–
    Second transformer (tagged source + sentiment-aware vectors)0.810.85

    Post-editing the baseline's training data alone did not help.

  • Sentence-level cost is the mean squared difference between Azure sentiment scores of the output and of the reference (lower is better):

    SystemPositiveNegative
    Second transformer0.060.14
    Google Translate0.710.15

  • A mistranslation that flipped the sentiment still got a BLEU of about 76. Short sentences suffered the largest sentiment cost; one "awesome by all means" sentence was rendered as "Terrible by all accounts".

Interesting ideas

  • I1. Compare an independent external signal (the author's star rating) with an automatic sentiment score of the output, and use the mismatch to choose which items humans should inspect; this focuses manual error analysis cheaply.
  • I2. Treat words whose polarity depends on register or variety as a dedicated list, and tag their polarity explicitly on the input rather than hoping context resolves it.
  • I3. Measure meaning preservation by the distance between sentiment scores of the output and of the reference, alongside surface-overlap metrics, because high n-gram overlap can coexist with an inverted message.
  • I4. Report results separately for each polarity of an ambiguous item; aggregate scores hid that Google handled the negative sense well but failed the positive one.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): disagreement between a known label (e.g., a test answer key or fable moral) and Mova's derived stance could select sentences for review and for inducing repair rules.
  • I2 (tokens): a small lexicon of register-sensitive evaluative words (slang like "sick", "wicked") could attach a polarity/register feature at the lemma level, which frames can then consume transparently.
  • I3 (back): the text → graph → text compressor could be checked by comparing the evaluative stance of the input and the regenerated text, catching polarity loss that string metrics miss.
  • I4 (learn): splitting evaluation by sense or class acts as a built-in control that exposes systems that only learned the majority reading.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SurreyGuildford · United Kingdom21 works · 677 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Machine translation · 1,087 worksSentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Hadeel Saadany and Constantin Orasan. 2020. Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation of Arabic Reviews. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{saadany2020is,
  title = {{Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation of Arabic Reviews}},
  author = {Hadeel Saadany and Constantin Orasan},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.3/}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamed Abdalla and Graeme Hirst. 2017. Cross-lingual sentiment analysis without (good) translation. arXiv preprint arXiv:1707.01626. ↗
  2. Haithem Afli, Sorcha Maguire, and Andy Way. 2017. Sentiment translation for low resourced languages: Experiments on Irish general election tweets. In 18th International Conference on Computational Linguistics and Intelligent Text Processing, Budapest, Hungry, pages 17–21. find
  3. Hind M Alotaibi. 2017. Arabic-English parallel corpus: a new resource for translation training and language teaching. Arab World English Journal (AWEJ) Volume, 8. find
  4. Mohamed Aly and Amir Atiya. 2013. Labr: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498. ↗
  5. Matheus Araujo, Julio Reis, Adriano Pereira, and Fabricio Benevenuto. 2016. An evaluation of machine translation for multilingual sentence-level sentiment analysis. In Proceedings of the 31st Annual ACM Symposium on Applied Computing, pages 1140–1145. find
  6. Amira Barhoumi, Chafik Aloulou, Nathalie Camelin, Yannick Estève, and Lamia Belguith. 2018. Arabic sentiment analysis: an empirical study of machine translation’s impact. In Proceedings of the second Conference on Language Processing and Knowledge Management Kerkennah (Sfax), Tunisia. find
  7. Alexandre Bérard, Ioan Calapodescu, Marc Dymetman, Claude Roux, Jean-Luc Meunier, and Vassilina Nikoulina. 2019. Machine translation of restaurant reviews: New corpus for domain adaptation and robustness. arXiv preprint arXiv:1910.14589. ↗
  8. Shanbo Cheng, Shaohui Kuang, Rongxiang Weng, Heng Yu, Changfeng Zhu, and Weihua Luo. 2020. Auto-repair the synthetic data for neural machine translation. arXiv preprint arXiv:2004.02196. ↗
  9. Mara Chinea-Rios, Alvaro Peris, and Francisco Casacuberta. 2017. Adapting neural machine translation with parallel synthetic data. In Proceedings of the Second Conference on Machine Translation, pages 138–147. find
  10. Erkin Demirtas and Mykola Pechenizkiy. 2013. Cross-lingual polarity detection with machine translation. In Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining, pages 1–8. find
  11. Trang Ho and Allan Simon. 2016. manythings.org/anki/. Tatoeba: Collection of sentences and translations. http://www. ↗
  12. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339– 351. ↗
  13. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT: Open-Source Toolkit for Neural Machine Translation. In Proc. ACL. ↗
  14. Pintu Lohar, Haithem Afli, and Andy Way. 2017. Maintaining sentiment polarity in translation of user-generated content. The Prague Bulletin of Mathematical Linguistics, 108(1):73–84. ↗
  15. Pintu Lohar, Haithem Afli, and Andy Way. 2018. Balancing translation quality and sentiment preservation (non-archival extended abstract). In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 81–88. find
  16. Emad Mohamed, Behrang Mohit, and Kemal Oflazer. 2012. Transforming standard Arabic to colloquial Arabic. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 176–180.
  17. Saif M Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. How translation alters sentiment. Journal of Artificial Intelligence Research, 55:95–130. ↗
  18. Santanu Pal, Braja Gopal Patra, Dipankar Das, Sudip Kumar Naskar, Sivaji Bandyopadhyay, and Josef van Genabith. 2014. How sentiment analysis can help machine translation. In Proceedings of the 11th International Conference on Natural Language Processing, pages 89–94. find
  19. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. arXiv preprint cs/0205070.
  20. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta, May. ELRA. http://is.muni.cz/publication/884893/en. ↗
  21. Mohammad Salameh, Saif Mohammad, and Svetlana Kiritchenko. 2015. Sentiment after translation: A case-study on Arabic social media posts. In Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 767–777.
  22. Wael Salloum and Nizar Habash. 2013. Dialectal Arabic to English machine translation: Pivoting through modern standard Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 348–358.
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. ↗
  24. Gayane Shalunts, Gerhard Backfried, and Nicolas Commeignes. 2016. The impact of machine translation on sentiment analysis. Data Analytics, 63:51–56. find
  25. Chenglei Si, Kui Wu, Aiti Aw, and Min-Yen Kan. 2019. Sentiment aware neural machine translation. In Proceedings of the 6th Workshop on Asian Translation, pages 200–206. find
  26. Usama Soltan. 2017. The fine structure of the Neg-domain: evidence from Cairene Egyptian Arabic sentential negation. Florida Linguistics Papers, 4(3). find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  28. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 49–59.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saadany-2020-is-it-great-or-terrible/