Elamur Research⌕
☰
home / papers / 2021 / Automatic Romanization of Arabic Bibliographic Records

Automatic Romanization of Arabic Bibliographic Records

Paper · 2021 · Fadhl Eryani and Nizar Habash

Summary

Library standards ask cataloguers to add Romanized versions of records in non-Latin scripts, and this is done by hand. The paper presents what the authors describe as the first reported results on automatically converting undiacritized Arabic catalogue entries into the ALA-LC Romanization standard. The task is hard because the output must restore unwritten vowels and gemination, mark clitics with hyphens, drop most case endings, and capitalize proper nouns, which amounts to a kind of named-entity recognition. The authors built a parallel corpus of about 2.5M words from public MARC dumps (Library of Congress, University of Michigan, NYUAD's Arabic Collections Online). They compared four approaches: pure character rules, rules driven by MADAMIRA morphological analysis, a most-frequent-mapping lookup (MLE), and a character-level GRU Seq2Seq model. Gaps where the neural model dropped words are filled from the simpler systems. The best combinations reach 89.3% exact word accuracy on a blind test set. The data and code are released, which gives the field a new benchmark and could support a tool that assists librarians.

Key points

  • Corpus: 107,439 Arabic-tagged records (MARC 008 "ara") taken from about 11.2M records in total, giving 599,442 entries and roughly 2.5M words, split 80/10/10 by record. Entries were kept only if they used one of 33 safe MARC tags and had matching token counts.
  • Baselines without parallel data: Rules Simple (104 regexes, 13 exceptions, one capitalization rule) scores 16.1% exact on test. Rules Morph (MADAMIRA plus character rules) scores 67.4% exact and 83.6% case-insensitive.
  • Data-driven systems: MLE Morph scores 84.8% exact on test. Seq2Seq alone is far worse than MLE on small data: at 31K words MLE is about 10 times better, and the two only match at around 250K words.
  • Seq2Seq omits about 2% of output words. Filling these gaps via edit-distance alignment gives Seq2Seq+MLE Simple and Seq2Seq+MLE Morph, both at 89.3% exact on test.
  • Error analysis covered 100 Dev errors of the best model:
    • 52 were mistakes in the human references, and the model was actually right in 85% of those cases.
    • Among genuine system errors, about 75% were wrong vowels or gemination, and 21% were Seq2Seq hallucinations.

Interesting ideas

  • I1. Use a layered backoff: take the most frequent training mapping when one exists, fall back to morphological analysis, and finally to hand-written character rules, so every input still gets an output.
  • I2. When a neural model drops output units, realign its output to the input by minimizing character edit distance, then fill each hole with the answer of a simpler system.
  • I3. Report several nested metrics (exact, case-insensitive, case-and-punctuation-insensitive) so that each gap between them shows how much error comes from one cause.
  • I4. Measure each method across shrinking fractions of the training data to find where the simple rule or lookup systems beat the neural model.
  • I5. Manually check a sample of disagreements with the gold standard: here about half were reference errors, and the system was often right on them.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Lemma and morphology lookup in crate en could follow the same chain: a treebank-attested table first, then analyzer rules, then a surface-rule fallback for OOV words, with the active layer recorded for explainability.
  • I2 (tree): If gparse ever loses or merges tokens, an edit-distance realignment step could restore one-to-one token correspondence and patch the gaps with the perceptron parser's arcs.
  • I3 (learn): Evaluation could similarly report LAS, UAS and label-only scores side by side, so that parser regressions are attributed to attachment or to labeling rather than seen as one number.
  • I4 (learn): Learning curves over silver-data fractions would show when a learned component actually outperforms its rule-based counterpart and deserves to replace it.
  • I5 (learn): Disagreements between witness parsers and the treebank could be routed to an audit that flags suspected gold errors, keeping noisy references out of training and scoring.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Fadhl Eryani and Nizar Habash. 2021. Automatic Romanization of Arabic Bibliographic Records. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{eryani2021automatic,
  title = {{Automatic Romanization of Arabic Bibliographic Records}},
  author = {Fadhl Eryani and Nizar Habash},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.23/}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 11–16, San Diego, California.
  2. Mohamed Al-Badrashiny, Ramy Eskander, Nizar Habash, and Owen Rambow. 2014. Automatic transliteration of romanized Dialectal Arabic. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL), pages 30–38, Ann Arbor, Michigan.
  3. Yaser Al-Onaizan and Kevin Knight. 2002. Machine Transliteration of Names in Arabic Text. In Proceedings of the Workshop on Computational Approaches to Semitic Languages (CASL). find
  4. Kenneth R. Beesley. 1997. Romanization, Transcription and Transliteration. Http://www.xrce.xerox.com/Research-Development/Historical-projects/Linguistic-Demos/Arabic-Morphological-Analysis-and-Generation/Romanization-Transcription-and-Transliteration. find
  5. Fadi Biadsy, Nizar Habash, and Julia Hirschberg. 2009. Improving the Arabic Pronunciation Dictionary for Phone and Word Recognition with Linguistically-Based Pronunciation Rules. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 397–405, Boulder, Colorado.
  6. Tim Buckwalter. 2004. Buckwalter Arabic Morphological Analyzer Version 2.0. LDC catalog number LDC2004L02, ISBN 1-58563-324-0. ↗
  7. Achraf Chalabi and Hany Gerges. 2012. Romanized Arabic transliteration. In Proceedings of the Second Workshop on Advances in Text Input Methods, pages 89–96, Mumbai, India.
  8. The COLING 2012 Organizing Committee. find
  9. Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar. ↗
  10. Kareem Darwish. 2014. Arabizi Detection and Conversion to Arabic. In Proceedings of the Workshop for Arabic Natural Language Processing (WANLP), pages 217–224, Doha, Qatar. ↗
  11. Imane Guellil, cal Azouaou Fai Mourad Abbas, and Fatiha Sadat. 2017. Arabizi transliteration of Algerian Arabic dialect into Modern Standard Arabic. In Social MT 2017/First workshop on social media and user generated content machine translation. find
  12. Nizar Habash, Anas Shahrour, and Muhamed Al-Khalil. 2016. Exploiting Arabic diacritization for high quality automatic annotation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4298– 4304, Portorož, Slovenia. European Language Resources Association (ELRA).
  13. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. ↗
  14. Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Morgan & Claypool Publishers. ↗
  15. Library of Congress. 2012. Arabic Romanization Table. https://www.loc.gov/catdir/cpso/ romanization/arabic.pdf . ↗
  16. Library of Congress. 2017. ALA-LC Romanization Tables. https://www.loc.gov/catdir/cpso/ roman.html . ↗
  17. Thang Luong, Hieu Pham, and Christopher Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421, Lisbon, Portugal. ↗
  18. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  19. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 1094–1101, Reykjavik, Iceland.
  20. Ali Shazal, Aiza Usman, and Nizar Habash. 2020. A unified model for Arabizi detection and transliteration using sequence-to-sequence models. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 167–177, Barcelona, Spain (Online). Association for Computational Linguistics.
  21. Ahmad Taymūr. 1927? Qabr al-imām al-Suyūt.ı̄ wa-tah.qı̄q mawd.i‘ihi. Arabic Collections Online: http://hdl.handle.net/2333.1/m37pvs4b . ↗
  22. Jihene Younes, Emna Souissi, Hadhemi Achour, and Ahmed Ferchichi. 2018. A sequence-to-sequence based approach for the double transliteration of Tunisian dialect. Procedia Computer Science, 142:238 – 245. Arabic Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eryani-2021-automatic-romanization-arabic-bibliographic-records/