📄 AfriCaption: Establishing a New Paradigm for Image Captioning in African Languages

Summary

Vision-language research relies mostly on English-centric resources such as Flickr8k and MS COCO, so African languages have almost no image captioning support. AfriCaption tackles this gap with a dataset and a model covering 20 African languages plus English. The authors translated Flickr8k captions using a cascade of machine translation systems (M2M-100, NLLB-200 and Azure Translate). When several systems handle the same language, LaBSE embedding similarity picks the best output. The pipeline is designed so that a stronger MT model added later can replace older translations. Low-similarity outputs are discarded with a threshold set through manual inspection. Translation quality was then checked with NLLB200 back-translation and with human evaluation. The captioning model has about 0.5B parameters and pairs a SigLIP image encoder with an NLLB decoder, trained in two stages. It outperforms the multilingual model Pangea on the two languages both systems share. Results vary widely, however: better-resourced languages such as Hausa and Yoruba score well, while Ewe and Dinka lag behind. The work provides a scalable starting point and benchmark for multimodal AI in underrepresented African languages.

Key points

• Language coverage: the dataset spans Afro-Asiatic, Niger-Congo and Nilo-Saharan languages, including Hausa, Igbo, Yoruba, Ewe, Luganda, Kinyarwanda, Amharic, Dinka and Kabyle. The authors say no prior caption corpus covers this many African languages.
• Quality filtering: translations are kept only if their LaBSE cosine similarity falls between 0.53 and 0.98. The 0.53 lower bound comes from a manual review of sample translations.
• Architecture and training: the model is built on a modified Hugging Face VisionEncoderDecoderModel and trained on an L4 GPU with lr 2.0e-5 and batch size 16.
  • Stage one, which aligns image and text, runs for 40 epochs.
  • Stage two, multimodal pretraining for captioning, runs for 30 epochs with the image encoder left unfrozen.
• BLEU results: the highest scores are Hausa ≈0.857, Igbo ≈0.851, Afrikaans ≈0.839 and Yoruba ≈0.821. The lowest is Kabyle ≈0.102. Pangea scores close to zero on Amharic and Igbo.
• Future direction: strong results for Luganda suggest that related Bantu languages, such as Zulu and Xhosa, could be added through transfer from closely related languages.