Elamur Research⌕
☰
home / papers / 2023 / PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation

PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation

Summary

Image translation (IT) converts text embedded in an image from one language into another. Cascaded OCR+MT pipelines accumulate errors and are slow. End-to-end models struggle because visual text and textual MT inputs are represented very differently. The authors propose PEIT, which uses pre-trained components to narrow this modality gap. Its parts are a visual encoder (ResNet or CRNN), a Transformer encoder-decoder shared across modalities, a contrastive aligner on the encoder side, and a cross-modal regularizer on the decoder side. The regularizer pushes the decoder to produce the same output whether the input is an image or plain text.

Training happens in two pre-training stages followed by fine-tuning:

  1. An NMT model is trained on parallel text.
  2. The full framework is trained on synthetic images rendered from that text.

For real-world evaluation, the team built ECOIT, a Chinese-English corpus of Taobao product images. PEIT outperforms cascaded systems and the earlier end-to-end model ItNet, while using fewer parameters and decoding faster. The work therefore contributes both a stronger method and a scarce public benchmark.

Key points

  • ECOIT was built from about 600K crawled images. Text was recognised with EasyOCR, and the authors then filtered and corrected the results by hand. Translations came from the Google Translate API, and crowd workers post-edited over 80% of them. The final splits are 477,490 training, 2,000 validation and 1,020 test pairs.
  • Ablation experiments confirm that removing the aligner or the regularizer, fully or partially, lowers quality.
  • Cross-contrastive learning, which uses an external pre-trained text encoder, slightly beats self-contrastive learning on Chinese-English: 45.9 vs. 45.3 BLEU and 67.5 vs. 67.0 METEOR.
  • The authors also built synthetic En-Fr and En-Ru test sets from UNv1.0-6way and WMT newstest data. On these, PEIT reaches 24.8 BLEU (En-Fr) and 16.8 BLEU (En-Ru), compared with ItNet's 19.7 and 12.6.
  • Code is available at github.com/lishangjie1/PEIT. The authors note two limitations: the multi-stage training is long, and ECOIT may contain "machine translationese".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

19 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 18 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.751

@inproceedings{zhu2023peit,
  title = {{PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation}},
  author = {Zhu, Shaolin and Li, Shangjie and Lei, Yikun and Xiong, Deyi},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.751},
  url = {https://doi.org/10.18653/v1/2023.acl-long.751}
}

References

28 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. 2022. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th Annual Meeting of the Association f… find
  2. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2022. XLS-R: self-supervised cross-lingual speech representation learning at scale. In Interspeech 2022… find our card
  3. Iacer Calixto, Daniel Stein, Evgeny Matusov, Sheila Castilho, and Andy Way. 2017a. Human evaluation of multi-modal neural machine translation: A case-study on e-commerce listing titles. In Proceedings of the Sixth Workshop on Vision and Language, VL@EACL 2017, Valencia, Spain, April 4, 2017, pages 3… find
  4. Iacer Calixto, Daniel Stein, Evgeny Matusov, Pintu Lohar, Sheila Castilho, and Andy Way. 2017b. Using images to improve machine-translating e-commerce product listings. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia… find
  5. Roldano Cattoni, Mattia Antonino Di Gangi, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Must-c: A multilingual corpus for end-to-end speech translation. Comput. Speech Lang., 66:101155. find
  6. Michael J. Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26-27, 2014, Baltimore, Maryland, USA, pages 376–380. The Association for Compute… find our card
  7. Qianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2021. Consecutive decoding for speech-to-text translation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleven… find
  8. Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, and Haoshuang Wang. 2020. PP-OCR: A practical ultra lightweight OCR system. CoRR, abs/2009.09941. — [arXiv:2009.09941](https://arxiv.org/abs/2009.09941) find
  9. Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. STEMM: self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 20… find
  10. Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-text translation. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 2214–2225. Association for… find
  11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society. find
  12. Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2261–2269. IEEE Computer Society. find
  13. Hirofumi Inaguma, Tatsuya Kawahara, and Shinji Watanabe. 2021. Source and target bidirectional knowledge distillation for end-to-end speech translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  14. Puneet Jain, Orhan Firat, Qi Ge, and Sihang Liang. 2021. Image translation network. find
  15. Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and Jingbo Zhu. 2022a. On vision features in multimodal machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 202… find
  16. Yaoyiran Li, Fangyu Liu, Nigel Collier, Anna Korhonen, and Ivan Vulic. 2022b. Improving word translation via two-stage contrastive learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022,… find
  17. Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-end speech translation with knowledge distillation. arXiv preprint arXiv:1904.08075. — [arXiv:1904.08075](https://arxiv.org/abs/1904.08075) find
  18. Elman Mansimov, Mitchell Stern, Mia Xu Chen, Orhan Firat, Jakob Uszkoreit, and Puneet Jain. 2020. Towards end-to-end in-image neural machine translation. CoRR, abs/2010.10648. find
  19. Baoguang Shi, Xiang Bai, and Cong Yao. 2017. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(11):2298–2304. find
  20. Yuqing Song, Shizhe Chen, Qin Jin, Wei Luo, Jun Xie, and Fei Huang. 2021. Product-oriented machine translation with cross-modal cross-lingual pre-training. In MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, pages 2843–2852. ACM. find
  21. Pengcheng Yang, Boxing Chen, Pei Zhang, and Xu Sun. 2020. Visual agreement regularized training for multi-modal machine translation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020,… find
  22. Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to-end speech translation via cross-modal progressive training. arXiv preprint arXiv:2104.10380. — [arXiv:2104.10380](https://arxiv.org/abs/2104.10380) find
  23. Umut Sulubacak, Ozan Caglayan, Stig-Arne Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, and Jörg Tiedemann. 2020. Multimodal machine translation through visuals and speech. Machine Translation, 34(2–3):97–147. find
  24. Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. 2022. Unified speech-text pre-training for speech translation and recognition. In Proceedings of the 60th Annual Meeting of the Association for Computa… find
  25. Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. 2021. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2… find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,… find
  27. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019… find
  28. Huiyun Yang, Huadong Chen, Hao Zhou, and Lei Li. 2022. Enhancing cross-lingual transfer by manifold mixup. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2023-peit-bridging-modality-gap-pre/