📄 Extension Multi30K: Multimodal Dataset for Integrated Vision and Language Research in Ukrainian

Summary

Authors from Kharkiv National University of Radio Electronics built a Ukrainian version of the multimodal corpus Multi30K. The corpus consists of images from Flickr30K and their text descriptions. The work aims to give Ukrainian, a low-resource language, high-quality parallel data for machine translation and for tasks that combine vision and language. The English descriptions were first translated automatically with Google Cloud Translator. A team of eight students and lecturers then proofread the translations, checking them against both the English original and the image itself. About half of the sentences needed corrections. The corpus was used in three directions. The first was fine-tuning MarianMT for translation from English. The second was testing multilingual semantic image search based on CLIP. The third was evaluating translation quality without a reference text, using cosine similarity of embeddings. The text model aligned with visual embeddings handled idioms and slang noticeably better than a text-only model. The work's value lies in an open Ukrainian multimodal resource. It also shows what existing multilingual models can do for Ukrainian and where they make mistakes.

Key points

• Corpus size: 31 014 sentences. Token counts: English 357 172, German 333 833, Ukrainian 276 520. There are fewer Ukrainian tokens partly because the language has no articles. About 51% of the machine translations were corrected by hand. The corpus is open: Hugging Face turuta/Multi30k-uk and GitHub researchlabs/multi30k-uk.
• Quality check: cosine similarity was computed with the distiluse-base-multilingual-cased-v2 model on a sample of 5000 pairs. Only 3 pairs scored below 0.4. After the manual corrections, the share of pairs with high similarity increased.
• Machine translation: MarianMT (Helsinki NLP, OPUS) was fine-tuned on Multi30K together with other collected datasets. On 5000 TATOEBA texts, the model achieved METEOR 0.3810 and BERTScore F1 0.9232. Multi30K on its own did not substantially improve quality, but it improved generalization.
• Semantic search (CLIP + DistilBERT, clip-ViT-B-32-multilingual-v1): the correct image was found for 44.5% of English descriptions and only 29.55% of Ukrainian ones. The model recognizes objects well but often confuses actions and background details.
• Reference-free translation evaluation: the model aligned with CLIP gave no pair a score below 0.5. For "walkie talkie" → «рація», the score rose from 0.32 to 0.7456. For idioms such as "Keep the change", it rose from about 0.25 to 0.95. The authors stress that this approach still needs to be validated.