Elamur Research⌕
☰
home / papers / 2022 / A Semi-supervised Approach for a Better Translation of Sentiment in Dialectical Arabic UGT

A Semi-supervised Approach for a Better Translation of Sentiment in Dialectical Arabic UGT

Paper · 2022 · Hadeel Saadany, Constantin Orăsan, Emad Mohamed et al.

Summary

Machine translation of Arabic user-generated text (UGT), such as tweets and reviews, often flips sentiment. Online posts mix Dialectical Arabic (DA) with Modern Standard Arabic (MSA). The two varieties share homographs and polysemous words whose polarity can be opposite, UGT usually omits diacritics, and existing DA-EN parallel corpora are clean, hand-built data unlike noisy, profane online language.

The authors treat DA and MSA as separate languages and build a multi-directional DA-MSA-EN NMT system. Their pipeline has three steps:

  • A custom classifier pulls DA sentences out of mixed UGT.
  • A cross-lingual language model, following Lample and Conneau (2019), is pretrained with MLM on monolingual text and TLM on gold MSA-EN and MSA-DA pairs.
  • This model initialises the encoder and decoder, which are trained on parallel data (including silver DA-EN with a sentiment lexicon) plus back-translation over DA monolingual UGT.

They evaluate with a lexicon-based sentiment-distance score (SAM) and human ratings. The system outperforms a parallel-only baseline and an MSA-pivot model and roughly matches Google Translate. This shows that in-domain monolingual data can fix polarity errors in low-resource dialect MT.

Key points

  • Four data types are required: clean MSA-EN parallel data, clean MSA-DA parallel data, silver DA-EN UGT data with a sentiment lexicon infused, and DA monolingual UGT. MLM uses text streams truncated at 200 tokens.
  • Prior resources are small or off-domain. Zbib et al. (2012) offer about 250k DA-EN pairs, none of them UGT. MDC, MADAR, QCA and PADIC reach at most 14.7k sentences.
  • SAM works as a translation cost, so lower is better. Lexicon lookup uses spaCy V3.1 lemma-POS tags. The semi-supervised model scored lowest, with a test-set total of 5.26 and an average of 0.10.
  • Human evaluation:
    • Three native Arabic speakers rated sentiment preservation on a 1–5 scale for four systems on a hand-crafted set of hard examples.
    • The semi-supervised model was slightly above Google Translate for two annotators and below it for the third.
    • The baseline and pivot model scored about 2 points lower.
  • Error analysis found that the model handles aggressive curses and clitic negation better than the alternatives, sometimes at the cost of fluency. The authors release the dataset, the cross-lingual LMs and the NMT system.

Interesting ideas

  • I1. Treating a dialect as its own language, with a classifier routing sentences to it, avoids letting the dominant standard variety's word senses override dialect meanings.
  • I2. In TLM, a masked word can be recovered from either its own sentence or the aligned translation, which pulls the embedding spaces of the languages together.
  • I3. SAM compares lexicon-based polarity of lemma-POS items between output and reference, giving a sentiment-specific cost alongside BLEU-style scores.
  • I4. A small hand-crafted challenge set of examples known to break existing systems gives a sharper test than a generic test set.
  • I5. Many sentiment flips came from single tokens, such as clitic negation or a word whose polarity depends on register.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Detect informal or social-media English as a distinct variety and send it through a dialect-aware conversion or adapted parser path instead of forcing standard-treebank analyses.
  • I2 (back): When aligning meta-world graphs with surface text in several languages, train the alignment so that a hidden node can be predicted from any linked realisation.
  • I3 (learn): Add a polarity-drift check to evaluation of the text → graph → text compressor, flagging reconstructions whose sentiment moved even when wording looks fine.
  • I4 (answer): Keep a curated set of fable and test questions with negation, curses and register-dependent words as a standing stress test for the readers.
  • I5 (frames): Make negation scope explicit in frames and mark lemmas with register-dependent polarity (e.g. "sick", "wicked") so pragmatics can resolve them from context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WolverhamptonWolverhampton · United Kingdom35 works · 6,331 citations🎓University of SurreyGuildford · United Kingdom21 works · 677 citations🎓De Montfort UniversityLeicester · United Kingdom3 works · 31 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Machine translation · 1,087 worksSentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Hadeel Saadany, Constantin Orăsan, Emad Mohamed and Ashraf Tantawy. 2022. A Semi-supervised Approach for a Better Translation of Sentiment in Dialectical Arabic UGT. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.20

@inproceedings{saadany2022semi,
  title = {{A Semi-supervised Approach for a Better Translation of Sentiment in Dialectical Arabic UGT}},
  author = {Hadeel Saadany, Constantin Orăsan, Emad Mohamed and Ashraf Tantawy},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.20},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.20}
}

References

33 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wael Abid. 2020. The SADID Evaluation Datasets for Low-Resource Spoken Language Machine Translation of Arabic Dialects. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6030–6043.
  2. Mohamed Aly and Amir Atiya. 2013. Labr: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451. ↗
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  4. Samhaa R. El-Beltagy. 2016. NileULex: A phrase and word level sentiment lexicon for Egyptian and Modern Standard Arabic. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2900–2905, Portorož, Slovenia. European Language Resources Association (ELRA).
  5. Mohamed Elmahdy, Mark Hasegawa-Johnson, and Eiman Mustafawi. 2014. Development of a tv broadcasts speech recognition system for Qatari Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3057–3061.
  6. Donia Gamal, Marco Alfonse, El-Sayed M El-Horbaty, and Abdel-Badeeh M Salem. 2019. Twitter benchmark dataset for Arabic sentiment analysis. Int J Mod Educ Comput Sci, 11(1):33. ↗
  7. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. arXiv preprint arXiv:1902.01313. ↗
  8. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041. ↗
  9. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. Sentiwordnet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In LREC, volume 10, pages 2200–2204.
  10. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A Multidialectal Parallel Corpus of Arabic. In LREC, pages 1240–1245.
  11. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  13. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Lorenzo Gatti, Marco Guerini, and Marco Turchi. 2015. Sentiwords: Deriving a high precision and high coverage lexicon for sentiment analysis. IEEE Transactions on Affective Computing, 7:409–421. find
  14. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  15. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, pages 177–180. find
  16. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. ↗
  17. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043. ↗
  18. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. arXiv preprint arXiv:1804.07755. ↗
  19. Karima Meftouh, Salima Harrat, Salma Jamoussi, Mourad Abbas, and Kamel Smaili. 2015. Machine translation experiments on PADIC: A parallel Arabic dialect corpus. In The 29th Pacific Asia conference on language, information and computation.
  20. Matt Post. 2018. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771. ↗
  21. Hadeel Saadany and Constantin Orasan. 2020. Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation of Arabic Reviews. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 24–37.
  22. Hadeel Saadany and Constantin Orasan. 2021. BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text. TRITON 2021, page 48. find
  23. Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020. AraBench: Benchmarking Dialectal Arabic-English Machine Translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107. find
  24. Omar Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41.
  25. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 49–59.
  26. Mohammad Salameh, Saif Mohammad, and Svetlana Kiritchenko. 2015. Sentiment after translation: A case-study on Arabic social media posts. In Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 767–777.
  27. Wael Salloum and Nizar Habash. 2013. Dialectal Arabic to English machine translation: Pivoting through modern standard Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 348–358.
  28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. ↗
  29. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. ↗
  30. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In LREC, volume 2012, pages 2214– 2218. Citeseer. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  32. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096–1103. find
  33. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior research methods, 45(4):1191–1207. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saadany-2022-semi-supervised-approach-better-translation/