Elamur Research⌕
☰
home / papers / 2025 / STBW at BAREC Shared Task 2025: AraBERT-v2 with MSE-SoftQWK Loss for Sentence-Level Arabic Readability

STBW at BAREC Shared Task 2025: AraBERT-v2 with MSE-SoftQWK Loss for Sentence-Level Arabic Readability

Paper · 2025 · Saoussan Trigui

Summary

This system paper addresses sentence-level Arabic readability prediction in the BAREC Shared Task 2025, strict track. In this track only the BAREC training split may be used, and each sentence must be assigned one of 19 fine-grained levels. The author treats the problem as regression with AraBERT-v2 and a single-output head. Each input is one sequence that joins corpus metadata (document, book, author, domain, readership group), simple statistics (word count, diacritic frequency, mean and spread of word length), the raw sentence and its D3Tok morphological segmentation from CAMeL tools. Training runs in two stages. The first is an MSE warm-up. The second continues from that checkpoint with a differentiable soft QWK loss, so that optimisation matches the official metric. On the shared test split, QWK rises from about 83.9 (MSE only) to about 84.9, slightly above the 84.0 baseline. On the blind test set the system scores 85.6 QWK and places 4th of 16. The work matters because it shows that a simple, metric-aligned recipe stays competitive.

Key points

  • Data: the BAREC corpus has about 69K sentences and about 1M words. It covers three domains (Arts & Humanities, Social Sciences, STEM) and three readership groups. The official split is roughly 80/10/10, with 54,845 training sentences.
  • Phase 1 setup: MSE training for 6 epochs, batch size 64, AdamW, learning rate 2e-5 with a 10% linear warm-up, and mixed precision. The best validation checkpoint (epoch 3) then starts Phase 2 with SoftQWK.
  • Comparison with the baseline on the test split:
    • QWK 84.8% vs 84.0%.
    • Acc3 86.6% vs 75.9%, with gains at the 7- and 5-level granularities as well.
    • Exact 19-level accuracy is much lower (34.8% vs 43.1%).
    • Average distance is slightly worse (1.19 vs 1.13).
  • Blind test: 85.6 QWK, 4th of 16 entries. The code is public on GitHub.
  • Error analysis shows the largest errors at the top of the scale. Gold levels 18–19 are often predicted as 16, which the author attributes to class imbalance. Planned fixes are reweighting and adding the SAMER data.

Interesting ideas

  • I1. Train first with a plain, stable loss, then switch to a differentiable surrogate of the evaluation metric; the warm start keeps the harder objective from wandering early.
  • I2. A single scalar prediction can become a probability vector over ordered levels by placing a Gaussian bump at the predicted value; this lets regression outputs feed class-based losses and confusion matrices.
  • I3. Heterogeneous inputs (metadata, statistics, raw text, morphological segmentation) are packed into one sequence, and each field marker is a reserved token that the tokenizer never splits. This gives the encoder a structured, readable input layout.
  • I4. Optimising for QWK raised the headline score while exact 19-level accuracy dropped sharply; reporting several granularities and distance measures exposed this trade-off.
  • I5. Per-level error plots combined with a confusion matrix revealed that rare top classes collapse onto a nearby frequent class, a failure that one global metric would hide.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): gparse could first be trained with standard arc/label cross-entropy, then fine-tuned on a soft LAS-like objective, so that training targets the score we actually report.
  • I2 (answer): a reader that grades the difficulty of school-course texts or questions could output one number and still yield an explainable, distance-aware distribution over grade levels.
  • I3 (tree): gparse could receive lemmas, morphology tags or treebank-dialect markers as protected field tokens next to the words, so that the added cues remain inspectable rather than fused.
  • I4 (learn): evaluations should always report companion metrics alongside the target one, so that a repair rule or loss change cannot silently trade exactness for a better headline number.
  • I5 (tokens): per-tag error breakdowns for the tagger could reveal rare UPOS or morphology values being absorbed by frequent neighbours, which would show where to add silver data or loss weights.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Saoussan Trigui. 2025. STBW at BAREC Shared Task 2025: AraBERT-v2 with MSE-SoftQWK Loss for Sentence-Level Arabic Readability. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.50

@inproceedings{trigui2025stbw,
  title = {{STBW at BAREC Shared Task 2025: AraBERT-v2 with MSE-SoftQWK Loss for Sentence-Level Arabic Readability}},
  author = {Saoussan Trigui},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.50},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.50}
}

References

20 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  2. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association.
  4. Sadik Bessou and Ghozlane Chenni. 2021. Efficient measuring of readability to improve documents accessibility for arabic language learners. arXiv preprint arXiv:2109.08648. ↗
  5. Violetta Cavalli-Sforza, Hind Saddiki, and Naoual Nassiri. 2018. Arabic readability research: current state and future directions. Procedia computer science, 142:38–49. find
  6. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. find
  7. Edgar Dale and Jeanne S Chall. 1948. A formula for predicting readability: Instructions. Educational research bulletin, pages 37–54. find
  8. Jordi de la Torre, Domenec Puig, and Aida Valls. 2018. Weighted kappa loss function for multi-class classification of ordinal data in deep learning. Pattern Recognition Letters, 105:144–154. Machine Learning and Applications in Artificial Intelligence. find
  9. Raul Diaz and Amit Marathe. 2019. Soft labels for ordinal regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  10. Mahmoud El-Haj and Paul Rayson. 2016. Osman—a novel arabic readability metric. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 250–255. find
  11. Khalid N. Elmadani, Nizar Habash, and Hanada Taha-Thomure. 2025b. A large and balanced corpus for fine-grained Arabic readability assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16376–16400, Vienna, Austria. Association for Computational Linguistics.
  12. Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221. find
  13. Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 359–376, Vienna, Austria. Association for Computational Linguistics.
  14. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2022. Arabic word-level readability visualization for assisted text simplification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 242–249, Abu Dhabi, UAE. Association for Computational Linguistics.
  15. Joseph Marvin Imperial and Ekaterina Kochmar. 2023. Automatic readability assessment for closely related languages. arXiv preprint arXiv:2305.13478. ↗
  16. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  17. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.
  18. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  19. Sowmya Vajjala. 2021. Trends, limitations and open challenges in automatic readability assessment research. arXiv preprint arXiv:2105.00973. ↗
  20. Khalid N. Elmadani, Bashar Alhafni, Hanada Taha, and Nizar Habash. 2025a. BAREC shared task 2025 on Arabic readability assessment. In Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/trigui-2025-stbw-barec-shared-task-2025/