Elamur Research⌕
☰
home / papers / 2024 / Transformer Models in Education: Summarizing Science Textbooks with AraBART, MT5, AraT5, and mBART

Transformer Models in Education: Summarizing Science Textbooks with AraBART, MT5, AraT5, and mBART

Summary

The paper tackles automatic summarization of Arabic school science material. Arabic remains under-resourced for this task: its morphology is rich, its cursive script complicates segmentation, and annotated corpora are scarce. The authors compiled a new dataset from the Palestinian curriculum's biology textbooks for grades 11 and 12, paired with summaries supplied by an experienced biology teacher and the Palestinian Ministry of Education. They fine-tuned four pretrained sequence-to-sequence transformers on it: MT5, AraBART, AraT5 (a checkpoint referred to as Summarization-Arabic-English-News) and mBART50. Evaluation combined automatic ROUGE scores with expert grading on a 1–10 scale covering coherence, informativeness and relevance. The expert's overall rating was 7.17. Training and validation loss curves showed that AraBART and the AraT5 checkpoint generalized cleanly, while mBART50 and MT5 began to overfit. A qualitative case study on a passage about sponges compared each model's output with the expert summary. The work matters because it supplies an educational Arabic summarization resource and pairs metric scores with judgments from a domain teacher.

Key points

  • The dataset comes from Palestinian-curriculum biology textbooks (grades 11–12). Reference summaries and evaluations came from a teacher with 16 years of experience and the Palestinian Ministry of Education.
  • Four models were fine-tuned and compared: MT5, AraBART, AraT5 (Summarization-Arabic-English-News checkpoint) and mBART50.
  • Evaluation used ROUGE plus expert scores from 1 to 10 on coherence, informativeness and relevance. The reported overall expert rating was 7.17.
  • Loss curves showed steady, non-overfitting learning for AraBART and AraT5. mBART50 and MT5 had validation loss that plateaued or rose while training loss kept falling.
  • In the sponge case study:
    • AraBART was closest to the expert summary.
    • MT5 was thorough but wordy.
    • The AraT5 checkpoint covered structure but under-covered reproduction.
    • The authors propose LoRA as future work.

Interesting ideas

  • I1. Using a curriculum's own textbooks as a closed domain gives a coherent, graded body of text where a subject teacher can reliably judge output quality.
  • I2. Scoring output separately on coherence, informativeness and relevance, rather than with one overlap metric, shows where a system fails, not just how much.
  • I3. A per-passage case study shows what each model omits relative to an expert reference, revealing content gaps that ROUGE averages hide.
  • I4. Reading training and validation loss curves together is used as a cheap diagnostic to judge which models generalize on a small specialized dataset.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Graded school textbooks for one subject could serve as a staged test bed for the school-course readers, with a teacher-style rubric for scoring answers.
  • I2 (learn): Evaluation could report separate axes (e.g. faithfulness of extracted facts vs. coverage of key facts) so that repair rules target the specific weakness.
  • I3 (back): The text → graph → text compressor could be checked by diffing which world facts (e.g. reproduction vs. anatomy) survive into the generated summary against an expert's selection.
  • I4 (tree): When training gparse or the perceptron parser on small domain treebanks, tracking validation-loss divergence could flag overfitting before silver data is trusted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Summarization Semitic languages

 Institutions

🎓Columbia UniversityNew York · United States118 works · 44,744 citations🎓Arab American UniversityJenin · Palestine4 works · 6 citations

Cite

Cite as Sari Masri; Yaqeen Raddad; Fidaa Khandaqji; Huthaifa I. Ashqar; and Mohammed Elhenawy. 2024. Transformer Models in Education: Summarizing Science Textbooks with AraBART, MT5, AraT5, and mBART. arXiv:2406.07692.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.07692 · arXiv 2406.07692

@misc{ashqar2024transformer,
  title = {{Transformer Models in Education: Summarizing Science Textbooks with AraBART, MT5, AraT5, and mBART}},
  author = {Sari Masri; Yaqeen Raddad; Fidaa Khandaqji; Huthaifa I. Ashqar; and Mohammed Elhenawy},
  year = {2024},
  doi = {10.48550/arxiv.2406.07692},
  eprint = {2406.07692},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.07692}
}

References

2 references, 2 of them with a card of ours. ↗ leads to the work.

  1. A. P. Widyassari et al., “Review of automatic text summarization techniques & methods,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 4, pp. 1029–1046, Apr. 2022, doi: 10.1016/J.JKSUCI.2020.05.006. A. Gillioz, J. Casas, E. Mugellini, and O. A. Khaled, “Overview of the Transformer-based Models for NLP Tasks,” Proceedings of the 2020 Federated Conference on Computer Science and Information Systems, FedCSIS 2020, pp. 179–183, Sep. 2020, doi: 10.15439/2020F20. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | Request PDF.” Accessed: May 05, 2024. [Online]. Available: https://www.researchgate.net/publication/328230984_BERT_Pre-training_of_Deep_Bidirectional_Transformers_for_Language_Understanding Y. Tang et al., “Multilingual Translation with Extensible Multilingual Pretraining and Finetuning”. M. Kamal Eddine, N. Tomeh, N. Habash, J. Le Roux, M. Vazirgiannis, and É. Polytechnique, “AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization,” pp. 31–42, 2022, Accessed: May 05, 2024. [Online]. Available: https://huggingface.co/moussaKam/AraBART L. Xue et al., “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer”, Accessed: May 05, 2024. [Online]. Available: https://pypi.org/project/langdetect/ “marefa-nlp/summarization-arabic-english-news · Hugging Face.” Accessed: May 05, 2024. [Online]. Available: https://huggingface.co/marefa-nlp/summarization-arabic-english-news “‘"Al-Maleh, M., & Desouki, S. (2020). Arabic text summarization using deep learning approach. Journal of Big Data, 7, 1-17.’..".”. “T. El-Shishtawy and F. El-Ghannam, ‘'Keyphrase based Arabic summarizer (KPAS),’ ’ 2012 8th International Conference on Informatics and Systems (INFOS), Giza, Egypt, 2012, pp. NLP-7-NLP-14”. “Sobh, I. M. A. H., Darwish, N., & Fayek, M. (2009). An optimized dual classification system for Arabic extractive generic text summarization. Giza, ”. “Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. arXiv Prepr arXiv :1409.0473. 2014”. ↗ our card
  2. “Sutskever I, Vinyals O, Le QV. Sequence to sequence learning with neural networks. Adv Neural Inf Process Syst. 2014;4:3104–12”. “Rush AM, Chopra S, Weston J. A neural attention model for abstractive sentence summarization. arXiv Prepr arXiv :1509.00685 . 2015”. “A. Reda et al., ‘'A Hybrid Arabic Text Summarization Approach based on Transformers,’ ’ 2022 2nd International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), Cairo, Egypt, 2022, pp. 56-62, doi: 10.1109/MIUCC55081.2022.9781694”. “Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805”. “Abu Nada, A. M., Alajrami, E., Al-Saqqa, A. A., & Abu-Naser, S. S. (2020). Arabic text summarization using arabert model using extractive text summarization approach”. “Wissam Antoun*, Fady Baly*, Hazem Hajj ‘ ’AraBERT: Transformer based Model for Arabic Language Understanding, 30 Mar 2020”. “Kieuvongngam, V., Tan, B., & Niu, Y. (2020). Automatic text summarization of covid-19 medical research articles using bert and gpt-2. arXiv preprint arXiv:2006.01997”. “Bani-Almarjeh, M., & Kurdy, M. B. (2023). Arabic abstractive text summarization using RNN-based and transformer-based architectures. Information Processing & Management, 60(2), 103227”. “Kahla, M., Novák, A., & Yang, Z. G. (2023). Fine-tuning and multilingual pre-training for abstractive summarization task for the Arabic language. In Annales Mathematicae et Informaticae (Vol. 57, pp. 24-35).”. M. Bani-Almarjeh and M. B. Kurdy, “Arabic abstractive text summarization using RNN-based and transformer-based architectures,” Inf Process Manag, vol. 60, no. 2, p. 103227, Mar. 2023, doi: 10.1016/J.IPM.2022.103227. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/masri-2024-transformer-models-education-summarizing-science/