Elamur Research⌕
☰
home / papers / 2025 / Syntaxa at BAREC Shared Task 2025: BERTnParse - Fusion of BERT and Dependency Graphs for Readability Prediction

Syntaxa at BAREC Shared Task 2025: BERTnParse - Fusion of BERT and Dependency Graphs for Readability Prediction

Paper · 2025 · Ahmed Bahloul

Summary

This system paper addresses Task 1 of the BAREC Shared Task 2025 (strict track): assigning one of 19 readability levels to a single Arabic sentence. The author argues that sentence difficulty depends on syntactic structure as well as vocabulary, and that this matters especially for a morphologically rich language like Arabic. BERTnParse parses each sentence with CamelParser2.0 and builds a dependency graph. Fine-tuned AraBERTv2 embeddings serve as node features, with learnable POS embeddings added to the nodes and dependency-relation embeddings on the edges. A 4-layer TransformerConv GNN processes the graph, and attentional pooling produces one sentence vector. That vector feeds two heads, one trained with an ordinal CORAL loss and one with a QWK loss. A key step aligns the two tokenizations at the word level, because the parser's clitic segmentation does not match BERT's WordPiece units. Training used only the BAREC data. The model raised QWK over an AraBERTv2 baseline on both the internal and the official blind test sets. Exact 19-level accuracy dropped, however, so the gain is in ordinal agreement rather than exact hits.

Key points

  • Data: the BAREC corpus, with 54,845 train, 7,310 dev, 7,286 test and 3,420 blind-test sentences, labeled on 19 levels. Under strict-track rules, other resources such as SAMER were excluded.

  • Pipeline:

    • AraBERTv2 (fine-tuned) supplies the embeddings, and CamelParser2.0 supplies POS tags and dependencies.
    • The GNN has 4 TransformerConv layers, each with 4 attention heads and hidden size 512.
    • Training used Adam with lr 1e-4, batch size 64, dropout 0.2 and early stopping.

  • Loss: an equal-weight sum of a CORAL loss (18 threshold logits) and a QWK loss (19 class logits).

  • Results:

    Test setMetricBaselineBERTnParse
    Internal testQWK80.283.7
    Internal testExact accuracy55.9%50.5%
    Official blindQWK81.584.3
    Official blindExact accuracy58.1%51.0%

    Mean distance was about 1.0 for both models.

  • Code: the implementation is released as ahmedehabb/BERTnParse on GitHub.

Interesting ideas

  • I1. Reconciling segmentations: subword embeddings are averaged into one vector per word, and parser nodes for clitics are merged into their host word, keeping the union of all their edges. This gives exactly one node and one vector per word, which makes neural features and the syntax graph line up whenever two segmentations disagree.
  • I2. Typed, directional edges: dependency labels become learned edge embeddings that modulate attention, and incoming and outgoing edges are aggregated separately before being fused into the node. The direction of a relation therefore carries information, instead of the tree being treated as an undirected bag of links.
  • I3. Root as a sentence anchor: the [CLS] token is placed into the graph as the syntactic root, so one node ties sentence-level context to the tree's head. This is a cheap way to inject global context into a purely local graph.
  • I4. Ordinal training objective: CORAL threshold logits are combined with a QWK-based loss on graded labels, so near misses cost less than distant ones. The results also show that this can raise agreement while lowering exact-match accuracy, so both metrics should be reported.
  • I5. Strong baseline as a control: the syntax-aware model is evaluated against the same pretrained encoder without the graph, which isolates what the structure itself contributes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could use this when mapping transformer subwords onto UD tokens, and the same logic fits multiword tokens like English "don't", where host and clitic need a consistent alignment.
  • I2 (frames): a role or coreference scorer over the UD tree could weight neighbours by relation type and direction, while staying explainable through the edge labels it attended to.
  • I3 (tree): an artificial ROOT node carrying a sentence-level vector could help the parser choose the main predicate in long or fragmentary sentences.
  • I4 (answer): if the readers estimate text or question difficulty for school-course tests, an ordinal loss plus reporting both QWK and exact accuracy would avoid hiding this trade-off.
  • I5 (learn): any tree-derived feature should be tested against a same-encoder, no-tree control, so that claimed gains from syntax are actually attributable to it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technical University of MunichMunich · Germany55 works · 48,474 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ahmed Bahloul. 2025. Syntaxa at BAREC Shared Task 2025: BERTnParse - Fusion of BERT and Dependency Graphs for Readability Prediction. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.35

@inproceedings{bahloul2025syntaxa,
  title = {{Syntaxa at BAREC Shared Task 2025: BERTnParse - Fusion of BERT and Dependency Graphs for Readability Prediction}},
  author = {Ahmed Bahloul},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.35},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.35}
}

References

22 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  2. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  5. Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. 2020. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 140:325–331. ↗
  6. Jordi de La Torre, Domenec Puig, and Aida Valls. 2018. Weighted kappa loss function for multi-class classification of ordinal data in deep learning. Pattern Recognition Letters, 105:144–154. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Khalid N. Elmadani, Bashar Alhafni, Hanada Taha, and Nizar Habash. 2025a. BAREC shared task 2025 on Arabic readability assessment. In Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China. Association for Computational Linguistics.
  9. Khalid N. Elmadani, Nizar Habash, and Hanada Taha-Thomure. 2025b. A large and balanced corpus for fine-grained Arabic readability assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16376–16400, Vienna, Austria. Association for Computational Linguistics. 359–376, Vienna, Austria. Association for Computational Linguistics.
  10. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2022. Arabic word-level readability visualization for assisted text simplification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 242–249, Abu Dhabi, UAE. Association for Computational Linguistics.
  11. Vladimir Vladimirovich Ivanov. 2022. Sentence-level complexity in russian: An evaluation of bert and graph neural networks. Frontiers in Artificial Intelligence, 5:1008411. ↗
  12. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. Preprint, arXiv:1609.02907. ↗
  13. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.
  14. Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506–1515, Copenhagen, Denmark. Association for Computational Linguistics.
  15. Tarek Naous, Michael J Ryan, Anton Lavrouk, Mohit Chandra, and Wei Xu. 2024. ReadMe++: Benchmarking multilingual language models for multi-domain readability assessment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12230–12266, Miami, Florida, USA. Association for Computational Linguistics. find
  16. Ahmed Elshabrawy, Muhammed AbuOdeh, Go Inoue, and Nizar Habash. 2023. CamelParser2.0: A state-ofthe-art dependency parser for Arabic. In Proceedings of ArabicNLP 2023, pages 170–180, Singapore (Hybrid). Association for Computational Linguistics. find
  17. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  18. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. ↗
  19. Xintong Shi, Wenzhi Cao, and Sebastian Raschka. 2023. Deep neural networks for rank-consistent ordinal regression based on conditional probabilities. Pattern Analysis and Applications, 26(3):941–955. ↗
  20. Nizar Habash, Abdelhadi Soudi, and Timothy Buckwalter. 2007. On Arabic Transliteration, pages 15–22. Springer Netherlands, Dordrecht. find
  21. Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. 2021. Masked label prediction: Unified message passing model for semi-supervised classification. Preprint, arXiv:2009.03509. ↗
  22. Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, and 12 others. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. Preprint, arXiv:1609.08144. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bahloul-2025-syntaxa-barec-shared-task-2025/