Elamur Research⌕
☰
home / papers / 2021 / Augmenting Transformers with KNN-Based Composite Memory for Dialog

Augmenting Transformers with KNN-Based Composite Memory for Dialog

Paper · 2021 · Angela Fan, Claire Gardent, Chloé Braud et al.

Summary

Dialog systems that generate their own replies often need outside information, such as encyclopedic facts or images, to stay on topic and say something substantive. Earlier memory-augmented networks mostly attend over every memory slot, which becomes expensive and less effective as memories grow. Fan, Gardent, Braud and Bordes propose KNN-based Information Fetching (KIF) modules for generative Transformers.

Each external source is encoded once and then frozen. The model learns only a mapping that turns the dialog context into a query. Nearest-neighbour search, implemented with faiss, uses that query to pick the most relevant items. The selected items are re-encoded, merged through a gated weighted sum, and concatenated with the encoder output.

KIF is evaluated on two benchmarks, Wizard of Wikipedia and Engaging ImageChat, using memories built from Wikipedia sentences, related images paired with their dialogs, and utterances from the training set. Among generative models, KIF reaches the best reported results. Human judges found no statistically significant gap in interestingness or preference against top retrieval systems. Because each retrieved item is readable text, the model's choices are also easy to inspect.

Key points

  • The memories are pre-computed and never updated by backpropagation. This lets KIF search hundreds of thousands of candidates. In contrast, the Generative Transformer MemNet baseline handles larger knowledge pools poorly.
  • Combining sources helps on Wizard of Wikipedia. F1 is 18.1 with training utterances alone, 23.9 with Wikipedia knowledge alone, and 25.9 with both. The corresponding Engaging ImageChat scores are 13.9, 13.8 and 14.4.
  • Richer search queries raise validation F1 on Wizard of Wikipedia. Scores go from 24.6 with the previous utterance only, to 26.4 with full dialog context, to 27.4 with turn embeddings. Engaging ImageChat gains in a similar way, rising from 13.3 to about 15.1.
  • The gating mechanism can suppress an unrelated source. When PersonaChat turns are added as noise, gating recovers nearly the performance of a model without that source.
  • Multi-hop retrieval brought no gain on Wizard of Wikipedia; 2–3 fetches scored 25.9–26.9 F1. Retrieving too many neighbours (large k) also hurts performance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States183 works · 228,558 citations🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🔬Laboratoire Lorrain de Recherche en Informatique et ses ApplicationsNancy · France30 works · 3,899 citations🔬Institut de Recherche en Informatique de ToulouseToulouse · France35 works · 1,872 citations🎓Université de LorraineNancy · France20 works · 945 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Angela Fan, Claire Gardent, Chloé Braud and Antoine Bordes. 2021. Augmenting Transformers with KNN-Based Composite Memory for Dialog. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00356

@article{fan2021augmenting,
  title = {{Augmenting Transformers with KNN-Based Composite Memory for Dialog}},
  author = {Angela Fan, Claire Gardent, Chloé Braud and Antoine Bordes},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00356},
  url = {https://doi.org/10.1162/tacl_a_00356}
}

References

45 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4177–4187. find
  2. Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. find
  3. Angela Fan, David Grangier, and Michael Auli. 2018a. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45–54.
  4. Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiao-jiang Liu, and Shuming Shi. 2019. Retrieval-guided dialogue response generation via a matching-to-generation framework. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference o… find
  5. Angela Fan, Mike Lewis, and Yann Dauphin. 2018b. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898.
  6. Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2017a. Efficient softmax approximation for GPUs. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1302–1310.
  7. Sarath Chandar, Sungjin Ahn, Hugo Larochelle, Pascal Vincent, Gerald Tesauro, and Yoshua Bengio. 2016. Hierarchical memory networks. CoRR, abs/1605.07427. — [arXiv:1605.07427](https://arxiv.org/abs/1605.07427) find
  8. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017b. Improving neural language models with a continuous cache. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. find
  9. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879. DOI: https://doi.org/10.18653/v1/P17-1171, PMCID: P… ↗
  10. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural Turing machines. arXiv preprint arXiv:1410.5401. find
  11. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the International Conference on Machine Learning, pages 5695–5704.
  12. Wenlin Chen, David Grangier, and Michael Auli. 2016. Strategies for training large vocabulary neural language models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1975–1985. DOI: https://doi.org/10.18653/v1/P16 -1186 ↗
  13. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations. find
  14. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of Wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
  15. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data. DOI: https://doi.org/10.1109/TBDATA .2019.2921572 ↗
  16. Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale seq2seq models to multi-document inputs. In Proceedings of the 2019 Conference on Empirical Methods Armand Joulin and Tomas Mikolov. 2015. Inferring algorithmic patterns with stack-a…
  17. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. find
  18. Andriy Mnih and Geoffrey Hinton. 2009. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems, pages 1081–1088. find
  19. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  20. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  21. Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2019. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559. find
  22. Tobias Plötz and Stefan Roth. 2018. Neural nearest neighbors networks. In Advances in Neural Information Processing Systems, pages 1087–1098. find
  23. Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087. — [arXiv:1909.03087](https://arxiv.org/abs/1909.03087) find
  24. Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, and Jianfeng Gao. 2019. Conversing by reading: Contentful neural conversation with on-demand machine reading. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages… find
  25. Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. 2019. Learning to select knowledge for response generation in dialog systems. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5081–5087. AAAI Press. find
  26. Jack Rae, Jonathan J. Hunt, Ivo Danihelka, Timothy Harley, Andrew W. Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. 2016. Scaling memory-augmented neural networks with sparse reads and writes. In Advances in Neural Information Processing Systems, pages 3621–3629. find
  27. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. 2018. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), pages 181–196. DOI: https://doi.org/… find
  28. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305.
  29. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. DOI: https://doi.org/10.18653/v1/P16 -1162 ↗
  30. Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. parl.ai: A Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase inde…
  31. Advances in Neural Information Processing Systems, pages 2440–2448. find
  32. Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. YFCC100M: The new data in multimedia research. Communications of the ACM, 59(2):64–73. DOI: https://doi.org/10 .1145/2812802 find
  33. Iulian V. Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2016a. Generative deep neural networks for dialogue: A short review. arXiv preprint arXiv:1611.06216. — [arXiv:1611.06216](https://arxiv.org/abs/1611.06216) find
  34. Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016b. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence.
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  36. Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. 2020. Image-chat: Engaging grounded conversations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2414–2429. DOI: https://doi.org/10.18653/v1/2020.acl-main.219 ↗
  37. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  38. Jason Weston, Emily Dinan, and Alexander Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, pages 87–92. DOI: https://doi.org/10 .18653/v1/W18-5713
  39. Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. 2019. Engaging image captioning via personality. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12516–12526. DOI: https://doi.org/10.1109/CVPR.2019.01280 — [doi:10.1109/cvpr.2019.01280](https://doi.org/10.1109/cvpr.2019.01280) ↗
  40. Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500. find
  41. Haoyu Song, Yan Wang, Wei-Nan Zhang, Xiaojiang Liu, and Ting Liu. 2020. Generate, delete and rewrite: A three-stage framework for improving persona consistency of dialogue generation. arXiv preprint arXiv:2004.07672. DOI: https://doi.org/10.18653/v1/2020.acl-main.516, PMID: 32249355 ↗
  42. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213. DOI: h…
  43. Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, and Ming Zhang. 2016. Two are better than one: An ensemble of retrieval-and generation-based dialog systems. arXiv preprint arXiv:1610.07149. — [arXiv:1610.07149](https://arxiv.org/abs/1610.07149) find
  44. Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. 2019. Augmenting self-attention with persistent memory. https://arxiv.org/abs/1907 .01470 ↗
  45. Yutao Zhu, Zhicheng Dou, Jian-Yun Nie, and Ji-Rong Wen. 2020. ReBoost: A retrieval-boosted sequence-to-sequence model for neural response generation. Information Retrieval Journal, 23(1):27–48. DOI: https://doi.org/10.1007/s10791-019-09364-x Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 20… — [doi:10.1007/s10791-019-09364-x](https://doi.org/10.1007/s10791-019-09364-x) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fan-2021-augmenting-transformers-knn-based-composite/