Elamur Research⌕
☰
home / papers / 2026 / DeformAR: A Visual Analytics Framework for Evaluation of Arabic Named Entity Recognition

DeformAR: A Visual Analytics Framework for Evaluation of Arabic Named Entity Recognition

Paper · 2026 · Ahmed Mustafa Younes

Summary

Standard evaluation of Arabic named entity recognition reports precision, recall and F1. These single numbers hide how data properties, learned representations and prediction behaviour jointly cause errors, and existing interpretability tools mostly target English or classification tasks. The author presents DeformAR, an open-source visual analytics framework that splits an NER system into data, model and evaluation subcomponents. Each subcomponent is described with metrics at several granularities, from corpus statistics down to per-token ambiguity, confidence, loss and embedding separability. An extraction phase fine-tunes a configurable pipeline and computes these metrics. A dashboard phase then connects them through linked interactive views, so users can drill down from system-level comparison to individual sentences. Applied to ANERCorp with a fine-tuned AraBERTv02 model, the tool exposes labelling errors, poorly calibrated predictions and side effects of subword tokenisation that aggregate scores do not reveal. This matters because it turns evaluation into diagnosis and helps decide whether to fix the data, the preprocessing or the model. The methodology is described as language- and model-agnostic.

Key points

  • Case study data: ANERCorp (Benajiba et al., 2007, in the CAMeL Lab standardised release) has about 150K words of Modern Standard Arabic tagged with PER, ORG, LOC and MISC in IOB2. It uses a sequential 5:1 split:

    • train: 125,102 words, of which 13,181 are NE words;
    • test: 25,008 words, of which 3,375 are NE words.

  • Setup: The model is AraBERTv02-base with a linear classification head, trained with AdamW (lr 5×10⁻⁵, batch size 16) for 4 epochs. With WordPiece, only the first subword of each word ("core token") gets a label and contributes to the loss, and the framework analyses how this shapes behaviour.

  • Per-entity results (precision/recall):

    • LOC: 0.893/0.934
    • PER: 0.860/0.844
    • ORG: 0.784/0.751
    • MISC: 0.772/0.634, the weakest and least frequent class (235 test examples)

  • Dashboard and evaluation tools: The dashboard has three views:

    • a Cross-Component View;
    • a Behavioural Analysis View, in which brushing and filtering over as many as six variables connect a UMAP projection, a scatter plot of model behaviour and a heatmap showing correlations between metrics;
    • an Instance-Level View, which offers a sentence viewer for IOB1/IOB2, token analysis and BERTviz attention comparison before and after fine-tuning.

    In addition, seqeval is extended to separate boundary, exclusion and type errors at span level.

  • Availability and limitations: The code is at github.com/ay94/DeformAR. The framework diagnoses but does not fix errors, ignores training dynamics and covers only one dataset and model configuration. Interpreting its findings also requires Arabic linguistic expertise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SussexBrighton · United Kingdom22 works · 3,332 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Ahmed Mustafa Younes. 2026. DeformAR: A Visual Analytics Framework for Evaluation of Arabic Named Entity Recognition. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.abjadnlp-1.34

@inproceedings{younes2026deformar,
  title = {{DeformAR: A Visual Analytics Framework for Evaluation of Arabic Named Entity Recognition}},
  author = {Ahmed Mustafa Younes},
  booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script},
  year = {2026},
  doi = {10.18653/v1/2026.abjadnlp-1.34},
  url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.34}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Betty van Aken, Benjamin Winter, Alexander Löser, and Felix A. Gers. 2019. find
  2. How Does BERT Answer Ques-Vasudev Lal, Arden Ma, Estelle Aflalo, Phillip Howard, Ana Simoes, Daniel Korat, Oren Pereg, Gadi Singer, and Moshe Wasserblat. 2021. InterpreT: An Interactive Visualization Tool for Interpreting Transformers. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 135–142, Online. Association for Computational Linguistics. find
  3. Dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8628–8645, Singapore. Association for Computational Linguistics. find
  4. Raymond Li, Wen Xiao, Lanjun Wang, Hyeju Jang, and Giuseppe Carenini. 2021. T3-Vis: visual analytic for Training and fine-Tuning Transformers in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 220–230, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  5. Khaled Shaalan. 2014. A Survey of Arabic Named Entity Recognition and Classification. Computational Linguistics, 40(2):469–510. Place: Cambridge, MA Publisher: MIT Press. ↗
  6. Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig. 2021. ExplainaBoard: An Explainable Leaderboard for NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 280–289, Online. Association for Computational Linguistics. ↗
  7. Shuheng Liu and Alan Ritter. 2023. Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8254–8271, Toronto, Canada. Association for Computational Linguistics. ↗
  8. Hiroki Nakayama. 2018. {{seqeval}: a python framework for sequence labeling evaluation}. {Software available from \url{https://github.com/chakki-works/seqeval}}. ↗
  9. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  10. Narendra Patwardhan, Stefano Marrone, and Carlo Sansone. 2023. Transformers in the Real World: A Survey on NLP Applications. Information, 14(4):242. Number: 4 Publisher: Multidisciplinary Digital Publishing Institute. ↗
  11. Xiaoye Qu, Yingjie Gu, Qingrong Xia, Zechang Li, Zhefeng Wang, and Baoxing Huai. 2023. A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends. arXiv preprint. ArXiv:2302.03512 [cs]. ↗
  12. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2022. Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2340–2354, Dublin, Ireland. Association for Computational Linguistics. find
  13. Xiaofei Sun, Diyi Yang, Xiaoya Li, Tianwei Zhang, Yuxian Meng, Han Qiu, Guoyin Wang, E. Hovy, and Jiwei Li. 2021. Interpreting Deep Learning Models in Natural Language Processing: A Review. ArXiv. ↗
  14. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275–9293, Online. Association for Computational Linguistics. find
  15. Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. 2020. The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models. arXiv preprint. ArXiv:2008.05122 [cs]. ↗
  16. Jesse Vig. 2019. A Multiscale Visualization of Attention in the Transformer Model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37– 42, Florence, Italy. Association for Computational Linguistics.
  17. Julia El Zini and Mariette Awad. 2023. On the Explainability of Natural Language Processing Deep Models. ACM Computing Surveys, 55(5):1–31. ArXiv:2210.06929 [cs]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/younes-2026-deformar-visual-analytics-framework-evaluation/