Elamur Research⌕
☰
home / papers / 2021 / Adaptive Nearest Neighbor Machine Translation

Adaptive Nearest Neighbor Machine Translation

Paper · 2021 · Xin Zheng, Zhirui Zhang, Junliang Guo et al.

Summary

kNN-MT improves a pre-trained NMT model by looking up similar decoder states in a datastore of cached contexts and their target tokens. It uses the same number of neighbors k for every token. When the datastore lacks relevant entries, the retrieved neighbors are noisy, and translation quality depends heavily on the chosen k. The authors propose Adaptive kNN-MT, which retrieves up to K neighbors and adds a tiny Meta-k Network. This network looks at the neighbors' distances and at how many distinct target tokens appear among them. It then weights several candidate k values (0 and powers of two up to K), and the weighted mix replaces the fixed interpolation weight λ. Only the Meta-k Network is trained, on a few hundred to a few thousand sentences, while the NMT model stays frozen. In German-English domain adaptation across four domains, the method consistently beats vanilla kNN-MT and is far less sensitive to K. It also transfers across domains, making retrieval-augmented translation easier to deploy without per-domain tuning.

Key points

  • Setup:
    • The base model is the WMT'19 German-English winner (Ng et al., 2019), implemented with fairseq and faiss.
    • Domains are IT, Medical, Koran and Law from the Khandelwal et al. (2020a) multi-domain data, scored with SacreBLEU.
    • The Meta-k Network is a two-layer FFN with hidden size 32, trained on the dev set (about 2k sentences) for about 5k steps.
  • Main results:
    • For K ≥ 4, average gains over vanilla kNN-MT range from 1.44 to 2.97 BLEU.
    • At K = 32, the average is 46.97 BLEU, compared with 44.00 for vanilla, 44.81 for uniform kNN-MT and 35.02 for base NMT.
  • Generality: A Meta-k Network trained only on IT and applied to the other domains averages 46.68 BLEU, close to the 46.97 obtained with in-domain training.
  • Robustness (domain mismatch, K = 32):
    • In the IT⇒Medical setting, vanilla kNN-MT falls to 25.82 BLEU, while Adaptive kNN-MT keeps 37.78.
    • In the Medical⇒IT setting, the scores are 15.79 and 30.09 respectively.
  • Ablation (IT, K = 8):
    • The full model scores 48.04 BLEU; removing the value-count feature lowers this to 46.76, and removing the distance feature lowers it to 45.60.
    • Even when trained on just 100 sentences, or shrunk to a hidden size of 8 (about 0.6k parameters), the method outperforms vanilla kNN-MT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China26 works · 111,012 citations🎓Nanjing UniversityNanjing · China28 works · 7,072 citations🏢Alibaba DAMO AcademyHangzhou · China28 works · 6,115 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang, Boxing Chen, Weihua Luo and Jiajun Chen. 2021. Adaptive Nearest Neighbor Machine Translation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-short.47

@inproceedings{zheng2021adaptive,
  title = {{Adaptive Nearest Neighbor Machine Translation}},
  author = {Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang, Boxing Chen, Weihua Luo and Jiajun Chen},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.47},
  url = {https://doi.org/10.18653/v1/2021.acl-short.47}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  2. Ankur Bapna and Orhan Firat. 2019. Non-parametric adaptation for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1921–1931, Minneapol… find
  3. Zi-Yi Dou, Junjie Hu, Antonios Anastasopoulos, and Graham Neubig. 2019. Unsupervised domain adaptation for neural machine translation with domain-aware feature embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer… find
  4. Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. 2018. Search engine guided neural machine translation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposiu… find
  5. Hany Hassan, Anthony Aue, C. Chen, Vishal Chowdhary, J. Clark, C. Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, W. Lewis, M. Li, Shujie Liu, T. Liu, Renqian Luo, Arul Menezes, Tao Qin, F. Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and M. Zhou. 2018… — Achieving Human Parity on Automatic Chinese to English News Translation
  6. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. CoRR, abs/1702.08734.
  7. Nora Kassner and Hinrich Schütze. 2020. BERT-kNN: Adding a kNN search component to pretrained language models for better QA. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3424–3430, Online. Association for Computational Linguistics. find
  8. Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020a. Nearest neighbor machine translation. CoRR, abs/2010.00710. — [arXiv:2010.00710](https://arxiv.org/abs/2010.00710) find
  9. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020b. Generalization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  10. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  11. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, NMT@ACL 2017, Vancouver, Canada, August 4, 2017, pages 28–39. Association for Computational Linguistics. find
  12. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR’s WMT19 news translation task submission.
  13. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314–319, Florence, Italy. Association for Computational Linguistics. find
  14. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr…
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  16. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
  17. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104–3112. find
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
  19. Hao-Ran Wei, Zhirui Zhang, Boxing Chen, and Weihua Luo. 2020. Iterative domain-repaired back-translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5884–5893, Online. Association for Computational Linguistics. find
  20. Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  21. Datastore Size Hard Disk Space (Datastore) Hard Disk Space (faiss index) IT Medical Koran Laws 3, 613, 350 6.9 Gb 266 Mb 6, 903, 320 15 Gb 492 Mb 524, 400 1.1 Gb 54 Mb 19, 070, 000 37 Gb 1.3 Gb *Table 7: Statistics of datastore in different domains.* ms / sent find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2021-adaptive-nearest-neighbor-machine-translation/