Elamur Research⌕
☰
home / papers / 2022 / Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

Paper · 2022 · Suraj Nair, Eugene Yang, Dawn Lawrie et al.

Summary

Neural rankers trained on large English collections such as MS MARCO have far surpassed lexical models like BM25 in monolingual search. Cross-language information retrieval (CLIR), especially dense retrieval, has not kept pace. The authors propose ColBERT-X, which adapts the multi-representation late-interaction model ColBERT to CLIR by replacing its English BERT encoder with the multilingual XLM-R. They compare two training regimes. In zero-shot, the model is fine-tuned only on English MS MARCO triples, and queries are machine-translated at search time. In translate-train, English MS MARCO queries are paired with machine-translated relevant and non-relevant passages, so the model learns to match across languages directly. Retrieval runs end to end through term-level approximate nearest neighbor search, followed by MaxSim reranking, with no BM25 first stage. Experiments used English queries over news in Chinese, Persian, French, German, Italian, Russian and Spanish (HC4 and CLEF). Trained ColBERT-X models significantly beat lexical query-translation baselines. The work matters because it gives a practical recipe, plus released code and translated training data, for dense CLIR in languages that lack native relevance judgments.

Key points

  • Evaluation collections: HC4 (Chinese, Persian) and CLEF (French, German, Italian, Russian, Spanish). These range from 16k to 646k documents, split into 0.1M–3.6M passages.
  • Encoder choice: XLM-R initialization significantly outperforms mBERT. One factor is tokenization: for Chinese, mBERT splits text into characters, while XLM-R uses sentencepiece subwords, which also gives smaller indexes.
  • Pseudo-relevance feedback: Feedback built from clustered ColBERT-X term embeddings raises translate-train MAP over BM25 with RM3 in every reported language except Spanish. Examples are German (0.406 vs 0.321) and French (0.459 vs 0.410).
  • Query length: Combining title and description, after removing "stop structures" to fit the 32-token query limit, beats title-only or description-only queries. French MAP reaches 0.507, compared with 0.422 for title queries.
  • Costs and resources: Each token is stored as a 128-dimensional 16-bit vector (256 bytes), so indexes range from 4.7GB (CLEF Russian) to 154GB (HC4 Chinese). Code and new MS MARCO translations into Chinese, Persian and Russian are released at github.com/hltcoe/ColBERT-X.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States258 works · 108,081 citations🎓University of Maryland, College ParkCollege Park · United States151 works · 29,733 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 worksInformation Systems · 1,888 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 888 worksInformation retrieval & search · 561 works

Cite

Cite as Suraj Nair, Eugene Yang, Dawn Lawrie, Kevin Duh, Paul McNamee, Kenton Murray, James Mayfield and Douglas W. Oard. 2022. Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models. ECIR 2022.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2201.08471

@inproceedings{nair2022transfer,
  title = {{Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models}},
  author = {Suraj Nair, Eugene Yang, Dawn Lawrie, Kevin Duh, Paul McNamee, Kenton Murray, James Mayfield and Douglas W. Oard},
  booktitle = {ECIR 2022},
  year = {2022},
  doi = {10.1007/978-3-030-99736-6\_26},
  eprint = {2201.08471},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1007/978-3-030-99736-6_26}
}

References

4 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Allan, J., Callan, J., Croft, W.B., Ballesteros, L., Byrd, D., Swan, R., Xu, J.: INQUERY does battle with TREC-6. NIST Special Publication 500-240 pp. 169– 206 (1998) 2. Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., Mc-Namara, A., Mitra, B., Nguyen, T., et al.: MS… — MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
  2. S. Nair et al. 14. Lawrie, D., Mayfield, J., Oard, D.W., Yang, E.: HC4: A new suite of test collections for ad hoc CLIR. In: Proceedings of the 44th European Conference on Information Retrieval (2021) 15. Lin, J., Nogueira, R., Yates, A.: Pretrained transformers for text ranking: Bert and beyond. Sy… — RoBERTa: A Robustly Optimized BERT Pretraining Approach
  3. Springer (2005) 21. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. pp. 311–318. Association for Computational Linguistics, Philadelphia, Pennsy… — ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
  4. Cross-Language Dense Retrieval 32. Wicks, R., Post, M.: A unified approach to sentence segmentation of punctuated text in many languages. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proces… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nair-2022-transfer-learning-approaches-building/