Elamur Research⌕
☰
home / papers / 2022 / Generative Approach for Gender-Rewriting Task with ArabicT5

Generative Approach for Gender-Rewriting Task with ArabicT5

Paper · 2022 · Sultan Alrowili and K. Vijay-Shanker

Summary

The paper addresses Arabic gender rewriting. The task is to change a sentence so that first- and second-person references (I/you) match a requested gender combination, using the APGC corpus. The prior state of the art, the multi-step model of Alhafni et al., chains four components: gender identification, a corpus-based bigram rewriter, a CAMeL Tools morphological rewriter, and a character-level GRU encoder-decoder. The authors propose a simpler alternative that handles the task as a single text-to-text problem. They pre-train ArabicT5 (base, large and xlarge sizes) on 17GB of fairly formal Arabic text. They then fine-tune it on inputs that contain the sentence plus a target-gender tag (FF/FM/MF/MM). On the APGC v2.1 dev set, ArabicT5 XLarge beats mT5 and three AraT5 variants and has the highest recall of all systems. However, its precision and F0.5 remain well below the multi-step pipeline. The study shows that a single seq2seq model can approach this task, though not yet match a dedicated pipeline. It also suggests that the domain of the pre-training corpus can matter more than its size.

Key points

  • The roughly 17GB pre-training corpus for ArabicT5 combined the Marefa encyclopedia, Hindawi Books, the El-Khair (2016) news collection, and the Arabic edition of Wikipedia. It uses a 32K SentencePiece vocabulary and the efficient T5 configurations of Tay et al.
  • Model sizes:
    • Base: hidden size 512, 8 heads, 20 layers, trained for 256K steps on TPUv3-32.
    • Large: hidden size 768, 12 heads, 16 layers, trained for 512K steps on TPUv3-128.
    • XLarge: like Large but with 36 layers.
    • All three are released on Hugging Face.
  • Dev results on APGC v2.1, scored with the M2 scorer and BLEU:
    • ArabicT5 XLarge: P 73.0, R 87.1, F0.5 75.4, BLEU 98.0.
    • Multi-step model: P 88.8, R 86.8, F0.5 88.3, BLEU 98.1.
    • mT5 Base: F0.5 73.4.
    • AraT5 Base: F0.5 74.7.
  • AraT5 was trained on far more data (248GB, mostly Twitter), yet ArabicT5 at larger sizes outperforms it. The authors credit this to ArabicT5's encyclopedic and news-domain corpus.
  • Fine-tuning ran for 70 epochs at learning rate 1e-4 with PyTorch XLA on TPUv3-8, using the shared task's normalization script.

Interesting ideas

  • I1. The desired output condition (the target-gender label) is placed as an explicit tag inside the input, so one model learns all four rewriting directions. This is a cheap way to turn a family of related transformations into one conditional generator.
  • I2. BLEU barely separates the systems (97.5–98.1) because only a few letters change per sentence, while the edit-based M2 scorer exposes large precision gaps. When outputs mostly copy the input, the metric should score the edits, not the whole string.
  • I3. The pipeline wins on precision while the single seq2seq model wins on recall. This suggests a hybrid in which a neural generator proposes changes and explicit morphological or rule components filter them.
  • I4. A smaller, cleaner, domain-appropriate pre-training corpus outperformed a much larger one dominated by Twitter text. Corpus domain fit can outweigh raw size.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Generation from the meaning graph could take explicit control tags (person, gender, tense, register) alongside the graph, letting one realizer produce controlled variants instead of maintaining separate generators.
  • I2 (learn): Evaluations of repair rules or text→graph→text round-trips should score only the changed spans, against an unchanged-copy baseline as a negative control, so near-copy outputs do not look deceptively good.
  • I3 (tokens): A morphology layer could act as a precision gate on neurally proposed inflection changes, accepting only edits that the explicit analyzer can justify, which keeps every change explainable.
  • I4 (learn): When gathering silver data or teacher outputs for school texts and fables, it is worth prioritizing register-matched sources over sheer volume, and checking this with ablations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of DelawareNewark · United States20 works · 1,056 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Sultan Alrowili and K. Vijay-Shanker. 2022. Generative Approach for Gender-Rewriting Task with ArabicT5. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.55

@inproceedings{alrowili2022generative,
  title = {{Generative Approach for Gender-Rewriting Task with ArabicT5}},
  author = {Sultan Alrowili and K. Vijay-Shanker},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.55},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.55}
}

References

16 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022a. The Arabic parallel gender corpus 2.0: Extensions and analyses. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1870–1884, Marseille, France. European Language Resources Association.
  2. Bashar Alhafni, Nizar Habash, and Houda Bouamor. 2022b. User-centric gender rewriting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 618– 631, Seattle, United States. Association for Computational Linguistics. ↗
  3. Bashar Alhafni, Nizar Habash, Houda Bouamor, Ossama Obeid, Sultan Alrowili, Daliyah Alzeer, Khawlah M. Ashnqiti, Ahmed ElBakry, Muhammad ElNokrashy, Mohamed Gabr, Abderrahmane Issam, Abdelrahim Qaddoumi, K. Vijay-Shanker, and Mahmoud Zyate. 2022c. The shared task on gender rewriting. In Proceedings of the Seventh Arabic Natural Language Processing Workshop, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  4. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. ↗
  5. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR. find
  6. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568–572, Montréal, Canada. Association for Computational Linguistics. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  9. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  10. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations.
  11. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  12. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  13. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. ↗
  14. Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2021. Scale efficiently: Insights from pre-training and fine-tuning transformers. ↗
  15. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗
  16. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alrowili-2022-generative-approach-gender-rewriting-task/