Elamur Research⌕
☰
home / papers / 2025 / Noor at BAREC Shared Task 2025: A Hybrid Transformer-Feature Architecture for Sentence-level Readability Assessment

Noor at BAREC Shared Task 2025: A Hybrid Transformer-Feature Architecture for Sentence-level Readability Assessment

Paper · 2025 · Nour Rabih

Summary

This BAREC Shared Task 2025 system predicts how hard an isolated Arabic sentence is to read, on a fine-grained 19-level scale based on the Taha/Arabi21 framework. The author entered the Strict track, which allows only the official training data. The model has two branches. One is a pretrained Arabic BERT encoder whose [CLS] vector represents the sentence. The other is a small MLP over 51 handcrafted indicators. These cover morphology from CAMeL Tools, syntax from CamelParser dependency trees, word and syllable counts, lexical difficulty from a lemma–POS dictionary enriched with the SAMER Lexicon, and a predicted conceptual-difficulty level. Both representations are concatenated and passed to a 19-way classifier. Training used gradual unfreezing of the encoder.

MARBERT was the best of five encoders, with a test QWK of 80.95% and a blind-set QWK of 83.1%, which ranked 9th. Compared with text-only baselines, the features gave a consistent gain only for MARBERT. The study is useful as evidence that explicit linguistic signals do not help every encoder equally.

Key points

  • Five encoders were compared: MARBERT, MARBERTv2, AraBERTv2, AraBERTv02 and CamelBERT-MSA. On test, MARBERT reached QWK 80.95, Acc19 54.45 and mean distance 1.11. MARBERTv2 was close behind at QWK 80.06.
  • On the blind leaderboard set, the submission scored QWK 83.10, Acc19 56.10, ±1 accuracy 72.50 and distance 1.00. This placed 9th in the Strict track.
  • The 51 features are grouped into five families:
    • morphological: affixes, tense, passive, broken and feminine plurals;
    • syntactic: binary markers from CamelParser parses, such as nominal vs. verbal sentence, vocatives, preposed predicates and coordination;
    • word/syllable counts;
    • vocabulary difficulty: from training data plus SAMER;
    • content level: one of 8 levels predicted by a fine-tuned AraBERT.
  • Training used AdamW at lr 2e-5, batch 16, 6 epochs, dropout 0.3, and early stopping on validation QWK with patience 3. Unfreezing was staged: embeddings frozen at the start, the top 4 layers released after epoch 1, and the full encoder after epoch 2.
  • Evaluation was ordinal. QWK is the main metric, alongside adjacent (±1) accuracy, mean absolute distance, and accuracy after collapsing the 19 levels into 7, 5 or 3 bins.

Interesting ideas

  • I1. Lexical difficulty is scored by the hardest lemma–POS pair in the sentence, not by an average. Each pair's level comes from its distribution over levels in the training data, with a relaxed 1% threshold to ignore rare outliers. This gives a simple, traceable difficulty signal that one rare word can trigger.
  • I2. Syntactic complexity is encoded as rule-derived yes/no flags read off dependency trees, such as coordination, vocatives and verbal clauses with or without objects. Each flag is interpretable and can be checked by hand.
  • I3. An ordinal 19-level task is scored at several granularities (7, 5 and 3 bins), plus off-by-one accuracy and mean distance. This shows whether errors are near-misses or gross mistakes.
  • I4. An ablation against text-only baselines showed that the hand-built features helped only one of five encoders. Adding explicit linguistic features should therefore be tested per model rather than assumed to help.
  • I5. Encoder layers are unfrozen in stages, top layers first and the full model later, while a freshly initialised side branch and head are trained. This protects the pretrained representations early in fine-tuning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course readers could rate passage and question difficulty by the hardest lemma they contain and point to that lemma as the explanation.
  • I2 (tree): UD parses could produce a small inventory of construction flags, usable as complexity diagnostics and as sanity checks that a parse contains the expected structures.
  • I3 (learn): Graded outputs such as difficulty, confidence tiers or teacher agreement levels could be reported with collapsed-bin accuracy and distance, so small slips are not mistaken for real failures.
  • I4 (learn): When gparse or the perceptron parser receives extra morphological or lexicon features, compare with and without them for each backbone as a control before keeping them.
  • I5 (tree): Gparse fine-tuning on a new treebank or converted UD dialect could unfreeze the transformer in stages, to reduce damage to pretrained knowledge on small data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Nour Rabih. 2025. Noor at BAREC Shared Task 2025: A Hybrid Transformer-Feature Architecture for Sentence-level Readability Assessment. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.46

@inproceedings{rabih2025noor,
  title = {{Noor at BAREC Shared Task 2025: A Hybrid Transformer-Feature Architecture for Sentence-level Readability Assessment}},
  author = {Nour Rabih},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.46},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.46}
}

References

19 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Hend S Al-Khalifa and Amani A Al-Ajlan. 2010. Automatic readability measurements of the arabic text: An exploratory study. Arabian Journal for Science and Engineering, 35(2 C):103–124. find
  3. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  4. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  6. Michael Heilman, Kevyn Collins-Thompson, Jamie Callan, and Maxine Eskenazi. 2007. Combining lexical and grammatical features to improve readability measures for first and second language texts. In Human language technologies 2007: the conference of the North American chapter of the association for computational linguistics; proceedings of the main conference, pages 460–467. find
  7. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  8. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.
  9. Safae Berrichi, Naoual Nassiri, Azzeddine Mazroui, and Abdelhak Lakhouaja. 2024. Exploring the impact of deep learning techniques on evaluating arabic l1 readability. In Artificial Intelligence, Data Science and Applications, pages 1–7, Cham. Springer Nature Switzerland. find
  10. Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. Linguistic features for readability assessment. arXiv preprint arXiv:2006.00377. ↗
  11. Khalid N. Elmadani, Bashar Alhafni, Hanada Taha, and Nizar Habash. 2025a. BAREC shared task 2025 on Arabic readability assessment. In Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China. Association for Computational Linguistics.
  12. Khalid N. Elmadani, Nizar Habash, and Hanada Taha-Thomure. 2025b. A large and balanced corpus for fine-grained Arabic readability assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16376–16400, Vienna, Austria. Association for Computational Linguistics.
  13. Ahmed Elshabrawy, Muhammed AbuOdeh, Go Inoue, and Nizar Habash. 2023. CamelParser2.0: A State-of-the-Art Dependency Parser for Arabic. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  14. Nizar Habash, Hanada Taha-Thomure, Khalid Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 359–376, Vienna, Austria. Association for Computational Linguistics.
  15. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2022. Arabic word-level readability visualization for assisted text simplification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 242–249, Abu Dhabi, UAE. Association for Computational Linguistics.
  16. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  17. Sarah E Petersen and Mari Ostendorf. 2009. A machine learning approach to reading level assessment. Computer speech & language, 23(1):89–106. ↗
  18. Yuxuan Sun, Keying Chen, Lin Sun, and Chenlu Hu. 2020. Attention-based deep learning model for text readability evaluation. 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find
  19.  ñ J‚Ó H. Taha. 2017. HAK J ’ JË é£ @XA Jë Q K AªÓ éJ  K. QªË@ ñ’ JË@ . ©K PñJË@ð Qå„ JÊË ø ñK. QË@ H. AJºË@ P@X . find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rabih-2025-noor-barec-shared-task-2025/