Elamur Research⌕
☰
home / papers / 2025 / ADOR: Dataset for Arabic Dialects in Hotel Reviews: A Human Benchmark for Sentiment Analysis

ADOR: Dataset for Arabic Dialects in Hotel Reviews: A Human Benchmark for Sentiment Analysis

Paper · 2025 · Maram I. Alharbi, Saad Ezzini, Hansi Hettiarachchi et al.

Summary

NLP tools for Arabic are trained mostly on Modern Standard Arabic (MSA), so regional dialects remain poorly served. Machine translation into dialects is also seldom checked by people. The authors start from the Arabic hotel reviews of SemEval 2016 Task 5 (ABSA) and clean them in four steps. They remove duplicate sentences and merge their annotations, normalise the text with the Ruqya library, drop sentences shorter than six words, and collapse aspect-level polarities into one sentence label using fixed rules. This leaves 538 sentences.

Meta's NLLB-200 then translates each sentence from MSA into Saudi, Maghribi and Egyptian Arabic. One native speaker per dialect gives yes/no verdicts on four questions: is the meaning right, is the dialect right, is the sentiment kept, and is the aspect target kept. Where needed, the annotator also writes a corrected sentence and target.

The resulting resource, ADOR, shows that sentiment and aspects mostly survive translation, but dialect authenticity varies widely. Saudi output was rarely judged genuinely Saudi and almost always needed editing, while Egyptian came out best. The work offers a human-verified dialect benchmark and shows that MT output can look fluent yet fail on dialect.

Key points

  • The source is the SemEval 2016 Task 5 Arabic hotel reviews (over 10,000 MSA sentences). After cleaning, the set has 538 sentences: 200 positive, 200 negative and 138 neutral.
  • Polarity consolidation follows three rules:
    • a sentence with both positive and negative aspects becomes neutral;
    • when neutral appears alongside one polar label, the polar label wins;
    • a single polarity is used as it is.
  • Translation correctness / dialect accuracy were 68.4% / 5.2% for Saudi, 77.6% / 44.4% for Maghribi and 84.7% / 76.6% for Egyptian. Sentiment preservation stayed around 91–99% and target preservation around 90–92%.
  • Correction rates were 95.17% (Saudi), 55.58% (Maghribi) and 31.04% (Egyptian). Maghribi had the most sentences failing three or more criteria (11.34%, against 5.02% Saudi and 2.97% Egyptian).
  • Each dialect had a single annotator, so no inter-annotator agreement was measured. Future work plans more annotators per dialect and adds Levantine and Yemeni.

Interesting ideas

  • I1. Instead of one overall score, translation quality is broken into separate binary checks (meaning, language variety, polarity, target), which makes each failure type visible on its own.
  • I2. The paper counts how many criteria fail at once in the same sentence, which separates outputs needing a full rewrite from those needing a small fix.
  • I3. Sentiment labels existed before translation, so the annotators' sentiment verdicts also serve as a consistency check on the annotators themselves.
  • I4. Several aspect-level polarities are merged into one sentence label by a small deterministic rule set (mixed becomes neutral, neutral yields to a polar label).
  • I5. Annotators save a corrected sentence and a corrected target next to every negative verdict, turning each error into a before/after pair.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): score the text → graph → text compressor and translation with independent yes/no checks for meaning, register, polarity and argument preservation, so a regression points at one specific layer.
  • I2 (learn): report per-sentence multi-failure rates in evaluation to tell deep parse/frame breakdowns apart from isolated local errors.
  • I3 (learn): seed LLM teacher batches with items whose answers are already known, as built-in controls that flag an unreliable teacher tier.
  • I4 (frames): use a similar explainable rule to aggregate evaluative stance across a sentence's frames into one sentence-level attitude, with the rule that fired recorded as the explanation.
  • I5 (learn): store human or teacher corrections as paired examples alongside error labels, so they can feed self-induced repair rules rather than serving only as scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Fahd University of Petroleum and MineralsDhahran · Saudi Arabia54 works · 4,666 citations🎓Lancaster UniversityLancaster · United Kingdom36 works · 1,615 citations🎓Jazan UniversityGazan · Saudi Arabia10 works · 98 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksSemitic languages · 3,066 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Maram I. Alharbi, Saad Ezzini, Hansi Hettiarachchi, Tharindu Ranasinghe and Ruslan Mitkov. 2025. ADOR: Dataset for Arabic Dialects in Hotel Reviews: A Human Benchmark for Sentiment Analysis. Proceedings of the First Workshop on Advancing NLP for Low-Resource Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alharbi2025ador,
  title = {{ADOR: Dataset for Arabic Dialects in Hotel Reviews: A Human Benchmark for Sentiment Analysis}},
  author = {Maram I. Alharbi, Saad Ezzini, Hansi Hettiarachchi, Tharindu Ranasinghe and Ruslan Mitkov},
  booktitle = {Proceedings of the First Workshop on Advancing NLP for Low-Resource Languages},
  year = {2025},
  doi = {10.26615/978-954-452-100-4-017},
  url = {https://doi.org/10.26615/978-954-452-100-4-017}
}

References

12 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ebtesam H. Almansor and Ahmed Al-Ani. 2017. Translating dialectal arabic as low resource language using word embedding. In International Conference Recent Advances in Natural Language Processing, RANLP, volume 2017-September, pages 52–57. Incoma Ltd. find
  2. Hanin Atwany, Nour Rabih, Ibrahim Mohammed, Abdul Waheed, and Bhiksha Raj. 2024. OSACT 2024 task 2: Arabic dialect to MSA translation. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 98–103, Torino, Italia. ELRA and ICCL.
  3. Sabri Boughorbel, MD Rizwan Parvez, and Majd Hawasly. 2024. Improving language models trained on translated data with continual pre-training and dictionary learning analysis.
  4. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. https://ai.facebook.com/ research/no-language-left-behind/ . Meta AI. ↗
  5. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab world. Commun. ACM, 64(4):72–81. find
  6. Mo El-Haj, Sultan Almujaiwel, Damith Premasiri, Tharindu Ranasinghe, and Ruslan Mitkov. 2024. DARES: Dataset for Arabic readability estimation of school materials. In Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024, pages 103–113, Torino, Italia. ELRA and ICCL. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md.
  7. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics.
  8. Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud Marı́a Jiménez-Zafra, and Gülşen Eryiğit. 2016. SemEval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 19–30, San Diego, California. Association for Computational Linguistics.
  9. Damith Premasiri, Tharindu Ranasinghe, Wajdi Zaghouani, and Ruslan Mitkov. 2022. DTW at qur’an QA 2022: Utilising transfer learning with transformers for question answering in a low-resource domain. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 88–95, Marseille, France. European Language Resources Association.
  10. Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural language processing for dialectical Arabic: A survey. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 36–48, Beijing, China. Association for Computational Linguistics.
  11. Silvana Yakhni and Ali Chehab. 2025. Can LLMs translate cultural nuance in dialects? a case study on Lebanese Arabic. In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 114–135, Abu Dhabi, UAE. Association for Computational Linguistics.
  12. Mohamed Zouidine and Mohammed Khalil. 2025. Large language models for arabic sentiment analysis and machine translation. Engineering, Technology and Applied Science Research, 15:20737–20742. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alharbi-2025-ador-dataset-arabic-dialects-hotel/