Elamur Research⌕
☰
home / papers / 2023 / Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin

Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin

Paper · 2023 · Pin-Jie Lin, Muhammed Saeed, Ernie Chang et al.

Summary

Nigerian Pidgin (Naija) has around 75 million speakers in Nigeria but very little parallel text. It also has no standard spelling, so models perform poorly on tasks such as machine translation and sentiment analysis. The authors address this in two ways. First, they build an English–Pidgin resource from aligned Bible verses, JW300, the Naija Treebank and several monolingual Pidgin sources. A baseline translation system then generates about 5 million synthetic sentence pairs. Second, they propose a two-stage cross-lingual adaptation framework:

  • Continual adaptive training (CAT): a pre-trained model keeps learning from unlabeled Pidgin text through masked language modeling.
  • Task adaptive training (TAT): the model also trains on synthetic pairs created by back-translation in both directions.

Experiments cover sentiment classification on NaijaSenti (BERT, RoBERTa) and translation (Transformer, T5, mT5). TAT improves BLEU over the baseline. The English-only T5 also outperforms multilingual mT5, which suggests that English, as Pidgin's lexifier, gives a stronger starting point. The released data and code support further work on Naija NLP.

Key points

  • Parallel sources: Bible (29,737 aligned verse pairs), JW300 (20,218) and the Naija Treebank (9,240; a UD treebank of transcribed spoken Pidgin with English translations).
  • Monolingual sources: NaijaSenti, the Afri-BERTa Pidgin subset (~176K sentences), BBC Pidgin, ASR and PidginUNMT for Pidgin, plus IWSLT'15 and WMT14 for English.
  • Released resources: 29.73K English–Pidgin sentence pairs and 5M synthetic pairs, with code in the CLaT GitHub repository.
  • Spelling variation: the authors identify four systematic types, all rooted in pronunciation:
    • sound alternation (carry → karry)
    • digraph conversion (our → awa)
    • phonetic transcription (trouble → trobol)
    • silent-letter deletion (he → e)
  • Translation results: TAT with bidirectional back-translation adds +2.28 BLEU for Pidgin→English and +1.69 BLEU for English→Pidgin over the baseline. T5 (base) beats mT5 by 2.38 BLEU on English→Pidgin.
  • Sentiment classification: NaijaSenti has 3 classes (6.7K/0.6K/1.2K split) and is scored with F1. CAT applied to BERT and RoBERTa is reported to improve over direct fine-tuning; exact scores are in the part of the text not provided.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksLow-resource languages · 267 works

Cite

Cite as Pin-Jie Lin, Muhammed Saeed, Ernie Chang and Merel Scholman. 2023. Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin. arXiv:2307.00382.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.00382 · arXiv 2307.00382

@misc{lin2023low,
  title = {{Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin}},
  author = {Pin-Jie Lin, Muhammed Saeed, Ernie Chang and Merel Scholman},
  year = {2023},
  doi = {10.48550/arxiv.2307.00382},
  eprint = {2307.00382},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.00382}
}

References

29 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H. C. Lent, E. Bugliarello, M. de Lhoneux, C. Qiu, and A. Søgaard, “On language models for creoles,” CoRR, vol. abs/2109.06074, 2021. [Online]. Available: https://arxiv.org/abs/ 2109.06074 ↗
  2. E. Chang, D. I. Adelani, X. Shen, and V. Demberg, “Unsupervised pidgin text generation by pivoting english data and self-training,” 2021.
  3. S. H. Muhammad, D. I. Adelani, I. S. Ahmad, I. Abdulmumin, B. S. Bello, M. Choudhury, C. C. Emezue, A. Aremu, S. Abdul, and P. Brazdil, “Naijasenti: A Nigerian Twitter sentiment corpus for multilingual sentiment analysis,” arXiv preprint arXiv:2201.08277, 2022.
  4. O. Ahia and K. Ogueji, “Towards supervised and unsupervised neural machine translation baselines for nigerian pidgin,” arXiv preprint arXiv:2003.12660, 2020. — [arXiv:2003.12660](https://arxiv.org/abs/2003.12660) find
  5. T. Bergmanis, A. Stafanovics, and M. Pinnis, “Robust neural machine translation: Modeling orthographic and interpunctual variation,” CoRR, vol. abs/2009.05460, 2020. [Online]. Available: https://arxiv.org/abs/2009.05460 — [arXiv:2009.05460](https://arxiv.org/abs/2009.05460) ↗
  6. I. Feldman and R. Coto-Solano, “Neural machine translation models with back-translation for the extremely low-resource indigenous language Bribri,” in Proceedings of the 28th International Conference on Computational Linguistics. Barcelona, Spain (Online): International Committee on Computational Li… find
  7. B. R. Chakravarthi, P. Rani, M. Arcan, and J. P. McCrae, “A survey of orthographic information in machine translation,” SN computer science, vol. 2, no. 4, p. 330, 2021. find
  8. W. F. Oyewusi, O. Adekanmbi, and O. Akinsande, “Semantic enrichment of nigerian pidgin english for contextual sentiment classification,” CoRR, vol. abs/2003.12450, 2020. [Online]. Available: https://arxiv.org/abs/2003.12450 ↗
  9. E. Chang, J. O. Alabi, D. I. Adelani, and V. Demberg, “Dialogue pidgin text adaptation via contrastive fine-tuning,” in 3rd Workshop on African Natural Language Processing, 2022. [Online]. Available: https://openreview.net/forum?id= HAzG99MV8-5 ↗
  10. J. O. Alabi, D. I. Adelani, M. Mosbach, and D. Klakow, “Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning,” in Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju, Republic of Korea: International Committee on Computat…
  11. E. Chang, J. O. Alabi, D. I. Adelani, and V. Demberg, “Few-shot pidgin text adaptation via contrastive fine-tuning,” in Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju, Republic of Korea: International Committee on Computational Linguistics, Oct. 2022, pp. 428… find
  12. A. Mohammadshahi, V. Nikoulina, A. Berard, C. Brun, J. Henderson, and L. Besacier, “Small-100: Introducing shallow multilingual machine translation model for low-resource languages,” arXiv preprint arXiv:2210.11621, 2022. find
  13. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu et al., “Exploring the limits of transfer learning with a unified text-to-text transformer.” J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67, 2020.
  14. Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 726–742, 2020. [Online]. Available: https://aclanthology.org/202… ↗
  15. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
  16. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  17. I. Abdulmumin, B. S. Galadanci, and A. Isa, “Enhanced back-translation for low resource neural machine translation using self-training,” in Information and Communication Technology and Applications: Third International Conference, ICTA 2020, Minna, Nigeria, November 24–27, 2020, Revised Selected Pap… find
  18. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
  19. Ž. Agić and I. Vulić, “JW300: A wide-coverage parallel corpus for low-resource languages,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 3204–3210. [Online]. Available: https:/…
  20. B. Caron, M. Courtin, K. Gerdes, and S. Kahane, “A surface-syntactic UD treebank for Naija,” in TLT 2019, Treebanks and Linguistic Theories, Syntaxfest, 2019. find
  21. K. Ogueji, Y. Zhu, and J. Lin, “Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages,” in Proceedings of the 1st Workshop on Multilingual Representation Learning. Punta Cana, Dominican Republic: Association for Computational Linguisti…
  22. D. Ajisafe, O. Adegboro, E. Oduntan, and T. Arulogun, “Towards end-to-end training of automatic speech recognition for nigerian pidgin,” arXiv preprint arXiv:2010.11123, 2020. — [arXiv:2010.11123](https://arxiv.org/abs/2010.11123) find
  23. K. Ogueji and O. Ahia, “Pidginunmt: Unsupervised neural machine translation from west african pidgin to english,” arXiv preprint arXiv:1912.03444, 2019. — [arXiv:1912.03444](https://arxiv.org/abs/1912.03444) find
  24. M.-T. Luong and C. D. Manning, “Stanford neural machine translation systems for spoken language domains,” in Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign, 2015. find
  25. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  26. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” arXiv preprint arXiv:1904.01038, 2019.
  27. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” CoRR, vol. abs/2010.11934, 2020. [Online]. Available: https://arxiv.org/abs/ 2010.11934 ↗
  28. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State-of-the-art natural language processing,” i…
  29. K. Ogueji, Y. Zhu, and J. Lin, “Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages,” in Proceedings of the 1st Workshop on Multilingual Representation Learning, 2021, pp. 116–126.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2023-low-resource-cross-lingual-adaptive/