Elamur Research⌕
☰
home / papers / 2025 / SATLab at BAREC Shared Task 2025: Optimizing a Language-Independent System for Fine-Grained Readability Assessment

SATLab at BAREC Shared Task 2025: Optimizing a Language-Independent System for Fine-Grained Readability Assessment

Paper · 2025 · Yves Bestgen

Summary

SATLab tackled the BAREC Shared Task 2025, which asks systems to predict the readability of Arabic sentences and documents on a 19-level ordinal scale. A document's level is defined as the level of its hardest sentence. The author built a deliberately simple, language-independent system in the "strict" track. Character n-grams, plus a few length statistics and corpus metadata, feed a linear support vector regression (LibLinear).

Because the official metric is Quadratic Weighted Kappa (QWK), the continuous SVR outputs are not simply rounded. A small randomized search, called OptiK, adjusts the cut-points between categories to maximize QWK on out-of-sample predictions. For documents, a second SVR combines features computed from the sentence-level predictions.

On the blind test, the sentence system beats the organizers' BERT-based baseline, but only when the threshold tuning is applied. It remains about 5 QWK points behind the winning system.

The work measures how much heavy architectures and pretrained embeddings actually add over a cheap baseline. It also shows that metric-aware post-processing can matter as much as the model itself.

Key points

  • Corpus: BAREC contains 1,922 documents and 69,441 sentences, annotated on 19 levels.
    • Splits: Learning (54,845 sentences), Development (7,310), Public Test (7,286).
    • The Blind Test has 100 documents and 3,420 sentences.
  • Sentence features:
    • character 1–6-grams occurring at least 10 times, weighted with sublinear Tf-Idf and L2-normalised;
    • log character count and log number of distinct characters;
    • one-hot Annotator, Source, Domain and Text Class.
  • Training setup: LibLinear L2-regularized L2-loss SVR dual, with 9-fold cross-validation.
    • Folds are stratified by document and balanced by document length.
    • Implementation uses custom SAS code, with OptiK written in C.
  • Sentence results: OptiK added roughly 1.3–1.8 QWK points on the public sets.
    • Best Public Test QWK was 80.5, slightly above the baseline's 80.2.
    • On the blind test, SATLab scored 82.3 against 87.5 for the best system.
  • Rankings and document results (strict track):
    • Sentence level: 16th of 24 teams overall, 13th of 16 among teams with papers.
    • Document level: 4th of 8.
    • Document QWK was unstable: 67.1 on Public Test versus 73.3 on Blind Test, with large swings across folds.

Interesting ideas

  • I1. Tune the decision thresholds of a regression model directly for the evaluation metric. The search moves one randomly chosen cut-point at a time within its neighbours' range and prefers mid-interval values on ties, which is far simpler than redesigning the loss function.
  • I2. Fit thresholds only on out-of-fold or held-out predictions, never on predictions for the training data, since overfit scores there would distort the cut-points.
  • I3. Build cross-validation folds by whole document, balanced for size, so sentences from one text never sit on both sides of the split.
  • I4. Build document-level predictions from the distribution of sentence-level predictions: the maximum, the share of sentences at each level, and length statistics. This goes beyond simply taking the maximum, which performed clearly worse for the baseline.
  • I5. Use a cheap, language-independent character n-gram model as a reference point to measure what complex systems actually add.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova could calibrate the cut-offs that turn continuous parser or teacher confidence scores into accept/reject tiers for silver data, tuning them on held-out data rather than fixing them at 0.5.
  • I2 (learn): any tuning of repair-rule triggers or confidence thresholds should use predictions on unseen sentences, keeping the evaluation honest.
  • I3 (tree): when splitting treebanks or silver corpora for parser evaluation, group by source document to avoid inflated LAS from near-duplicate context.
  • I4 (answer): estimating the difficulty of a fable or school text could aggregate per-sentence parse or frame complexity into a profile, which helps choose suitable reading material for the reader agents.
  • I5 (learn): such a non-structural baseline can serve as a negative control, showing whether the UD-tree-and-frames pipeline beats surface statistics on a given task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓KU LeuvenLeuven · Belgium46 works · 7,833 citations🎓UCLouvainLouvain-la-Neuve · Belgium20 works · 3,741 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Yves Bestgen. 2025. SATLab at BAREC Shared Task 2025: Optimizing a Language-Independent System for Fine-Grained Readability Assessment. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.40

@inproceedings{bestgen2025satlab,
  title = {{SATLab at BAREC Shared Task 2025: Optimizing a Language-Independent System for Fine-Grained Readability Assessment}},
  author = {Yves Bestgen},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.40},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.40}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  2. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING, 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  3. Christopher Beckham and Christopher Pal. 2017. A simple squared-error reformulation for ordinal classification. Preprint, arXiv:1612.00775. ↗
  4. Yves Bestgen. 2017. Beyond single-word measures: L2 writing assessment, lexical richness and formulaic competence. System, 69:65–78. ↗
  5. Yves Bestgen. 2021a. LAST at CMCL 2021 shared task: Predicting gaze data during reading with a gradient boosting decision tree approach. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 90–96, Online. Association for Computational Linguistics. find
  6. Yves Bestgen. 2021b. A simple language-agnostic yet strong baseline system for hate speech and offensive content identification. In Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR Workshop Proceedings. CEUR-WS.org. find
  7. Edgar Dale and Jeanne Chall. 1948. A formula for predicting readability: Instructions. Educational Research Bulletin, pages 37–54. find
  8. Marc Damashek. 1995. Gauging similarity with ngrams: Language-independent categorization of text. Science, 267(5199):843–848. find
  9. Khalid N. Elmadani, Bashar Alhafni, Hanada Taha, and Nizar Habash. 2025a. BAREC shared task 2025 on Arabic readability assessment. In Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China. Association for Computational Linguistics.
  10. Khalid N. Elmadani, Nizar Habash, and Hanada Taha-Thomure. 2025b. A large and balanced corpus for fine-grained arabic readability assessment. In Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria. Association for Computational Linguistics.
  11. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9:1871–1874. ↗
  12. Tarek Naous, Michael J Ryan, Anton Lavrouk, Mohit Chandra, and Wei Xu. 2024. ReadMe++: Benchmarking multilingual language models for multi-domain readability assessment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12230–12266, Miami, Florida, USA. Association for Computational Linguistics. find
  13. Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5366– 5377, Marseille, France. European Language Resources Association. find
  14. Sowmya Vajjala and Ivana Lučić. 2018. OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304, New Orleans, Louisiana. Association for Computational Linguistics.
  15. Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 359–376, Vienna, Austria. Association for Computational Linguistics.
  16. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2022. Arabic word-level readability visualization for assisted text simplification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 242–249, Abu Dhabi, UAE. Association for Computational Linguistics.
  17. Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: A transformer meets handcrafted linguistic features. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10669– 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  18. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.
  19. Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2021. Supervised and unsupervised neural approaches to text readability. Computational Linguistics, 47(1):141–179. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bestgen-2025-satlab-barec-shared-task-2025/