Elamur Research⌕
☰
home / papers / 2023 / Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation

Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation

Paper · 2023 · Zhiqi Huang, Puxuan Yu and James Allan

Summary

Cross-language retrieval for low-resource languages faces two problems. Multilingual encoders such as mBERT and XLM-R are pre-trained on unevenly distributed data, so they represent these languages poorly. Relevance-labelled CLIR training data for such languages also barely exists. The authors propose OPTICAL, which handles document matching and translation separately. First, a ColBERT-style bi-encoder teacher, initialised from a multilingual encoder, learns query–document matching from English MS MARCO triples. Then, for the setting where non-English queries search English documents, the teacher's document encoder is kept and a new student query encoder is trained. The student learns to place tokens of a non-English sentence near the teacher's tokens for its English parallel sentence. This alignment is framed as an optimal transport problem whose costs are cross-lingual cosine distances, and the resulting transport plan acts as a soft token alignment. Since only bitext is required, the approach suits languages without retrieval data. Across seven language pairs, OPTICAL clearly beats strong baselines on low-resource languages, including a translate-then-retrieve NMT pipeline, while using relatively little parallel text.

Key points

  • The distillation loss is the Frobenius inner product of the optimal transport plan and the token cost matrix. The plan is computed with a Sinkhorn-type entropic OT solver.
  • The evaluation covers seven language pairs: four low-resource languages from different families and three medium- or high-resource languages for comparison. Results are reported in MAP.
  • On low-resource languages, OPTICAL improves MAP by 13.7% over an NMT-based Translate-Test baseline.
  • The method is data-efficient. With bitext from CCAligned (capped at 2M pairs in the main results), Swahili overtakes Translate-Test with only 100K pairs and Tagalog does so from 500K. More bitext keeps improving reranking.
  • In a second experiment, French and German were made to simulate low-resource conditions. OPUS-MT-style NMT models trained on only 5M or 10M OPUS pairs caused clear drops for Translate-Test, which shows that the NMT route needs much more data. t-SNE plots show that distillation moves Tagalog tokens toward their English translations, while French tokens were already mixed with English ones. The paper appeared at WSDM '23.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Massachusetts AmherstAmherst Center · United States106 works · 66,692 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksInformation retrieval & search · 473 worksLow-resource languages · 267 works

Cite

Cite as Zhiqi Huang, Puxuan Yu and James Allan. 2023. Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation. arXiv:2301.12566.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3539597.3570468 · arXiv 2301.12566

@misc{huang2023improving,
  title = {{Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation}},
  author = {Zhiqi Huang, Puxuan Yu and James Allan},
  year = {2023},
  doi = {10.1145/3539597.3570468},
  eprint = {2301.12566},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3539597.3570468}
}

References

47 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sawsan Alqahtani, Garima Lalwani, Yi Zhang, Salvatore Romeo, and Saab Mansour. 2021. Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings. arXiv preprint arXiv:2110.02887 (2021). find
  2. Hamed Bonab, James Allan, and Ramesh Sitaraman. 2019. Simulating CLIR Translation Resource Scarcity Using High-Resource Languages. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval (Santa Clara, CA, USA) (ICTIR ’19). Association for Computing Machinery,… find
  3. Hamed Bonab, Sheikh Muhammad Sarwar, and James Allan. 2020. Training Effective Neural CLIR by Bridging the Translation Gap. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 9–18. find
  4. Luiz Henrique Bonifacio, Israel Campiotti, Vitor Jeronymo, Roberto Lotufo, and Rodrigo Nogueira. 2021. mmarco: A multilingual version of the ms marco passage ranking dataset. arXiv preprint arXiv:2108.13897 (2021). — [arXiv:2108.13897](https://arxiv.org/abs/2108.13897) find
  5. Martin Braschler. 2002. CLEF 2002—Overview of results. In Workshop of the Cross-Language Evaluation Forum for European Languages. Springer, 9–27. find
  6. Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019. Improving sequence-to-sequence learning via optimal transport. arXiv preprint arXiv:1901.06283 (2019). — [arXiv:1901.06283](https://arxiv.org/abs/1901.06283) find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  8. Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems 26 (2013). find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  10. Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020. CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 5960–5969.
  11. R Flamary, N Courty, D Tuia, and A Rakotomamonjy. 2016. Optimal transport for domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell 1 (2016). find
  12. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision 129, 6 (2021). find
  13. Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In CIKM’16. ACM, 55–64. find
  14. Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2, 7 (2015).
  15. Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving efficient neural ranking models with cross-architecture knowledge distillation. arXiv preprint arXiv:2010.02666 (2020). — [arXiv:2010.02666](https://arxiv.org/abs/2010.02666) find
  16. Zhiqi Huang, Hamed Bonab, Sheikh Muhammad Sarwar, Razieh Rahimi, and James Allan. 2021. Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. find
  17. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast Neural Machine Translation in C++. In Proceedings of ACL 2018, System…
  18. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 39–48.
  19. Yulong Li, Martin Franz, Md Arafat Sultan, Bhavani Iyer, Young-Suk Lee, and Avirup Sil. 2022. Learning Cross-Lingual IR from an English Retriever. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Associ… find
  20. Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang. 2022. Knowledge Distillation via the Target-Aware Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10915–10924. find
  21. Robert Litschko, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2018. Unsupervised cross-lingual information retrieval using monolingual data only. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 1253–1256. find
  22. Robert Litschko, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2018. Unsupervised cross-lingual information retrieval using monolingual data only. In SIGIR’18. 1253–1256. find
  23. Robert Litschko, Goran Glavaš, Ivan Vulic, and Laura Dietz. 2019. Evaluating Resource-Lean Cross-Lingual Embedding Models in Unsupervised Retrieval. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Paris, France) (SIGIR’19). 1109–111… find
  24. Su Lu, Han-Jia Ye, and De-Chuan Zhan. 2022. Faculty Distillation with Optimal Transport. arXiv preprint arXiv:2204.11526 (2022). — [arXiv:2204.11526](https://arxiv.org/abs/2204.11526) find
  25. Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. find
  26. Rongrong Ma, Guansong Pang, Ling Chen, and Anton van den Hengel. 2022. Deep Graph-level Anomaly Detection by Glocal Knowledge Distillation. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. find
  27. Suraj Nair, Eugene Yang, Dawn Lawrie, Kevin Duh, Paul McNamee, Kenton Murray, James Mayfield, and Douglas W Oard. 2022. Transfer learning approaches for building cross-language dense retrieval models. In European Conference on Information Retrieval. Springer, 382–396. find
  28. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPs.
  29. Thong Thanh Nguyen and Anh Tuan Luu. 2022. Improving Neural Cross-Lingual Abstractive Summarization via Employing Optimal Transport Distance for Knowledge Distillation. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  30. Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. 2020. Cosda-ml: Multilingual code-switching data augmentation for zero-shot cross-lingual nlp. arXiv preprint arXiv:2006.06402 (2020). — [arXiv:2006.06402](https://arxiv.org/abs/2006.06402) find
  31. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813 (2020).
  32. Shadi Saleh and Pavel Pecina. 2020. Document Translation vs. Query Translation for Cross-Lingual Information Retrieval in the Medical Domain. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 6849–6860. find
  33. Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. 2018. Improving GANs using optimal transport. arXiv preprint arXiv:1803.05573 (2018). find
  34. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019). find
  35. Shota Sasaki, Shuo Sun, Shigehiko Schamoni, Kevin Duh, and Kentaro Inui. 2018. Cross-lingual learning-to-rank with shared representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Shor… find
  36. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT – Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. European Association for Machine Translation, Lisboa, Portugal, 479–480. find
  37. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008). find
  38. Ivan Vulić and Marie-Francine Moens. 2015. Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 363–372. find
  39. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending Multilingual BERT to Low-Resource Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 2649–2656. https://doi.org/10.18653/v1/2020.findings-emnl… ↗
  40. Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Peter Vajda, and Joseph E Gonzalez. 2021. Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation. arXiv preprint arXiv:2112.09445 (2021). — [arXiv:2112.09445](https://arxiv.org/abs/2112.09445) find
  41. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? arXiv preprint arXiv:2005.09093 (2020).
  42. Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10687–10698. find
  43. Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. 2020. A Fast Proximal Point Method for Computing Exact Wasserstein Distance. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference (Proceedings of Machine Learning Research, Vol. 115), Ryan P. Adams and Vibhav Gogate (E… find
  44. Puxuan Yu and James Allan. 2020. A study of neural matching models for cross-lingual IR. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1637–1640. find
  45. Puxuan Yu, Hongliang Fei, and Ping Li. 2021. Cross-lingual language model pretraining for retrieval. In Proceedings of the Web Conference 2021. 1029–1039. find
  46. Mingkuan Yuan and Yuxin Peng. 2019. CKD: Cross-task knowledge distillation for text-to-image synthesis. IEEE Transactions on Multimedia 22, 8 (2019), 1955–1968. find
  47. Lingjun Zhao, Rabih Zbib, Zhuolin Jiang, Damianos Karakos, and Zhongqiang Huang. 2019. Weakly supervised attentional model for low resource ad-hoc cross-lingual information retrieval. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019). 259–264. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2023-improving-cross-lingual-information-retrieval/