Elamur Research⌕
☰
home / papers / 2021 / Supporting Undotted Arabic with Pre-trained Language Models

Supporting Undotted Arabic with Pre-trained Language Models

Paper · 2021 · Aviad Rom and Kfir Bar

Summary

Some Arabic social-media users now strip the consonant-distinguishing dots (i'jaam) from letters so their posts slip past automatic content filters. Native readers can still follow the text, but Transformer models trained on normally dotted script cannot. The authors fine-tune CAMeLBERT-Mix on two tasks, sentiment classification and NER, and test it on undotted versions of the test sets. They undot text with a Unicode mapping to the closest dotless glyph, and performance collapses, mostly because the tokenizer marks the undotted pieces as unknown.

They try three fixes, none of which retrain the language model:

  • Undotted vocabulary: undot the whole tokenizer vocabulary.
  • Extended vocabulary: add undotted aliases that point to the original token IDs.
  • AReDotter: a Marian MT model fine-tuned to put dots back, run as a preprocessing step.

All three recover most of the lost accuracy. AReDotter does best on sentiment, while the tokenizer edits do best on NER. The work shows that a trivial, human-readable script change can act as an adversarial attack, and that cheap preprocessing or vocabulary edits can defend against it.

Key points

  • Model and vocabulary: the base model is CAMeLBERT-Mix (MSA, Classical and Dialectal Arabic), with a 30,000-token BERT tokenizer.
    • Undotting the vocabulary merges 5,852 tokens (19.52%).
    • Extending it adds 17,280 undotted aliases, so about 57% of IDs end up with two surface forms.
  • AReDotter training: Helsinki-NLP/opus-mt-ar-en is fine-tuned for two epochs on synthetic pairs built from the 10M unlabeled NADI tweets. Each pair is a tweet and its auto-undotted copy, with URLs, mentions and hashtags removed.
  • Sentiment results (ArSarcasm-v2, macro F1 over positive/negative):
    • clean text: 70.55
    • undotted text: 44.86
    • undotted vocabulary: 64.50
    • extended vocabulary: 65.03
    • AReDotter: 68.27
  • NER results (ANERcorp in the Obeid et al. release, micro F1):
    • clean text: 81.39
    • undotted text: 9.16
    • undotted vocabulary: 72.85
    • extended vocabulary: 71.68
    • AReDotter: 67.97
  • Explanation of NER gap: the authors attribute AReDotter's weaker NER score to the seq2seq model sometimes emitting out-of-context tokens. They propose a sequence tagger as future work.

Interesting ideas

  • I1. Variant spellings can be aliased onto existing vocabulary IDs, so a frozen model accepts both forms without any retraining; this is a cheap way to absorb systematic orthographic noise.
  • I2. A deterministic, lossy corruption function applied to a large unlabeled corpus yields unlimited parallel data for training a restorer, with no human annotation needed.
  • I3. Testing a clean-trained model on a systematically corrupted copy of its test set exposes brittleness that ordinary accuracy hides.
  • I4. The share of unknown tokens works as a quick diagnostic that explains the collapse before any task metric is computed.
  • I5. Generative restoration can hallucinate tokens, which hurts token-aligned tasks more than sentence-level ones; per-position tagging keeps the alignment intact.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): map spelling variants, curly vs straight quotes and deliberate obfuscations (leetspeak, inserted dots) to the same canonical token/lemma entry before tagging.
  • I2 (learn): generate noisy/clean pairs (apostrophes dropped, lowercased, punctuation stripped) to train a normalizer whose repairs can be logged and explained.
  • I3 (learn): add perturbed test sets as stress tests next to negative controls, to measure how parser and QA accuracy degrade under orthographic noise.
  • I4 (tokens): report the OOV/unknown rate per input as an explainable early warning that downstream frames may be unreliable.
  • I5 (tree): prefer token-aligned repair (tag-and-edit) over free rewriting in preprocessing, so UD tree nodes stay anchored to the original surface tokens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Reichman UniversityHerzliya · Israel8 works · 16 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Aviad Rom and Kfir Bar. 2021. Supporting Undotted Arabic with Pre-trained Language Models. Proceedings of the 4th International Conference on Natural Language and Speech Processing (ICNLSP 2021).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rom2021supporting,
  title = {{Supporting Undotted Arabic with Pre-trained Language Models}},
  author = {Aviad Rom and Kfir Bar},
  booktitle = {Proceedings of the 4th International Conference on Natural Language and Speech Processing (ICNLSP 2021)},
  year = {2021},
  url = {https://aclanthology.org/2021.icnlsp-1.11/}
}

References

16 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021a. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. In Proceedings of the ACL-IJCNLP 2021 Main Conference. Association for Computational Linguistics. ↗
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, Houda Bouamor, and Nizar Habash. 2021b. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  3. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  5. Yassine Benajiba, Paolo Rosso, and José Miguel BenedíRuiz. 2007. ANERsys: An Arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing, pages 143–153, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  6. Peter T. Daniels. 2014. The Type and Spread of Arabic Script, pages 25 – 39. Brill, Leiden, The Netherlands. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Gerald Drißner. 2021. Social media & palestine: Dotless Arabic outsmarts algorithms. find
  9. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In International Conference on Learning Representations. ↗
  10. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  11. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8018–8025. AAAI Press.
  12. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre-trained transformers for Arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734, Online. Association for Computational Linguistics.
  13. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  14. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199. ↗
  15. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal. find
  16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rom-2021-supporting-undotted-arabic-pre-trained/