Elamur Research
☰
home / papers / 2021 / Perceiver: General Perception with Iterative Attention

Perceiver: General Perception with Iterative Attention

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2103.03206

Summary

Most deep perception models are tailored to a single modality. ConvNets, for example, rely on the 2D grid structure of images, so each new input type or sensor combination needs a redesigned architecture. Transformers make fewer assumptions about their inputs, but their cost grows quadratically with input length, which rules out feeding them raw pixels or audio samples. The Perceiver, from DeepMind, addresses this by having a small learned latent array repeatedly cross-attend to the large input array. A deep stack of self-attention layers then processes only the latents. This makes the cost linear in input size and separates network depth from input size. Position and modality are supplied through Fourier-feature encodings rather than built-in structure. With essentially the same architecture, the model was trained on ImageNet, AudioSet (audio, video and both) and ModelNet40. It matched ResNet-50 and ViT on ImageNet without convolutions and was competitive on AudioSet. The work points toward general-purpose perception models that can handle arbitrary sensor configurations.

Key points

  • Architecture: Cross-attention layers project a large "byte array" (e.g. 50,176 pixels for a 224×224 ImageNet image) into a much smaller latent array (512 latents on ImageNet). Cross-attention alternates with latent Transformer blocks. With optional weight sharing, the model behaves like an RNN unrolled in depth.
  • ImageNet: The model attends directly to roughly 50,000 pixels with no 2D convolutions and performs on par with ResNet-50 and ViT.
  • AudioSet (about 1.7M examples): On raw audio the Perceiver reaches 38.4 mAP. This beats most ConvNets, though not CNN-14 trained with AugMix and class balancing; without those additions CNN-14 scores 37.5. Audio+video fusion improves on either modality alone but still trails the best late-fusion system.
  • ModelNet40 (9,843 train / 2,468 test, about 2,000 points per object): The Perceiver reaches 85.7% top-1. This beats generic baselines such as a Transformer at 82.1%, ViT-B-2 at 78.9% and ResNet-50 at 66.3%, all using Fourier features. It remains below the specialized PointNet++ at 91.9%.
  • Limitations: The model still uses modality-specific augmentation and position encodings, and overfitting is a major concern. The authors propose large-scale pre-training and fully modality-agnostic learning as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 113 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2103.03206 · arXiv 2103.03206

@misc{jaegle2021perceiver,
  title = {{Perceiver: General Perception with Iterative Attention}},
  author = {Jaegle, Andrew and Gimeno, Felix and Brock, Andrew and Zisserman, Andrew and Vinyals, Oriol and Carreira, Joao},
  year = {2021},
  doi = {10.48550/arxiv.2103.03206},
  eprint = {2103.03206},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2103.03206}
}

References

109 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adelson, E. H. and Bergen, J. R. Spatiotemporal energy models for the perception of motion. Journal of the Optical Society of America A, 2(2):284–299, 1985. find
  2. Alayrac, J.-B., Recasens, A., Schneider, R., Arandjelović, R., Ramapuram, J., De Fauw, J., Smaira, L., Dieleman, S., and Zisserman, A. Self-supervised multimodal versatile networks. In Proceedings of Neural Information Processing Systems (NeurIPS), 2020. find
  3. Arandjelovic, R. and Zisserman, A. Objects that sound. In Proceedings of the European conference on computer vision (ECCV), 2018. find
  4. Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. find our card
  5. Babuschkin, I., Baumli, K., Bell, A., Bhupatiraju, S., Bruce, J., Buchlovsky, P., Budden, D., Cai, T., Clark, A., Danihelka, I., Fantacci, C., Godwin, J., Jones, C., Hennigan, T., Hessel, M., Kapturowski, S., Keck, T., Kemaev, I., King, M., Martens, L., Mikulik, V., Norman, T., Quan, J., Papamakario… find
  6. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedings of International Conference on Learning Representations (ICLR), 2015. find our card
  7. Bello, I., Zoph, B., Vaswani, A., Shlens, J., and Le, Q. V. Attention augmented convolutional networks. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2019. find
  8. Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document Transformer. arXiv preprint arXiv:2004.05150, 2020. find
  9. Borenstein, E., Sharon, E., and Ullman, S. Combining top-down and bottom-up segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2004. find
  10. Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax. ↗
  11. Brock, A., De, S., Smith, S. L., and Simonyan, K. High-performance large-scale image recognition without normalization. arXiv preprint arXiv:2102.06171, 2021. — [arXiv:2102.06171](https://arxiv.org/abs/2102.06171) find
  12. Bronstein, M., Bruna, J., LeCun, Y., Szlam, A., and Vandergheynst, P. Geometric deep learning: Going beyond Euclidean data. IEEE Signal Processing Magazine, 34 (4):18–42, 2017. find
  13. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with Transformers. In Proceedings of European Conference on Computer Vision (ECCV), 2020. find our card
  14. Carreira, J., Agrawal, P., Fragkiadaki, K., and Malik, J. Human pose estimation with iterative error feedback. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. find
  15. Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In Proceedings of International Conference on Machine Learning (ICML), 2020a. find
  16. Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. UNITER: Learning UNiversal Image-TExt Representations. In Proceedings of European Conference on Computer Vision (ECCV), 2020b. find
  17. Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse Transformers. arXiv preprint arXiv:1904.10509, 2019. find
  18. Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., Belanger, D., Colwell, L., and Weller, A. Rethinking attention with Performers. In Proceedings of International Conference on Learning Representations (ICLR), 2021. find
  19. Cireşan, D. C., Meier, U., Masci, J., Gambardella, L. M., and Schmidhuber, J. High-performance neural networks for visual object classification. Technical report, IDSIA, 2011. find
  20. Cohen, T. and Welling, M. Group equivariant convolutional networks. In Proceedings of International Conference on Machine Learning (ICML), 2016. find
  21. Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In Proceedings of International Conference on Learning Representations (ICLR), 2020. find
  22. Correia, G. M., Niculae, V., and Martins, A. F. Adaptively sparse Transformers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. find
  23. Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020. find
  24. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Annual Meetings of the Association for Computational Linguistics, 2019. find
  25. Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, L. Universal Transformers. In Proceedings of International Conference on Learning Representations (ICLR), 2019. find
  26. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. find
  27. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019. find
  28. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference on Learning Representations (ICLR)… find
  29. Esteves, C., Allen-Blanchette, C., Makadia, A., and Daniilidis, K. Learning SO(3) equivariant representations with spherical CNNs. In Proceedings of European Conference on Computer Vision (ECCV), 2018. find
  30. Fayek, H. M. and Kumar, A. Large scale audiovisual learning of sounds with weakly labeled data. In Proceedings of International Joint Conference on Artificial Intelligence, 2020. find
  31. Felleman, D. J. and Essen, D. C. V. Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1(1):1–47, 1991. find
  32. Ford, L., Tang, H., Grondin, F., and Glass, J. R. A deep residual network for large-scale acoustic scene analysis. In Proceedings of Interspeech, pp. 2568–2572, 2019. find
  33. Fukushima, K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, (36):193—-202, 1980. find
  34. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. Convolutional sequence to sequence learning. In Proceedings of International Conference on Machine Learning (ICML), 2017. find our card
  35. Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio Set: An ontology and human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017. find
  36. Girdhar, R., Carreira, J., Doersch, C., and Zisserman, A. Video action Transformer network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. find
  37. Goyal, A., Didolkar, A., Lamb, A., Badola, K., Ke, N. R., Rahaman, N., Binas, J., Blundell, C., Mozer, M., and Bengio, Y. Coordination among neural modules through a shared global workspace. arXiv preprint arXiv:2103.01197, 2021. — [arXiv:2103.01197](https://arxiv.org/abs/2103.01197) find
  38. Graves, A., Mohamed, A., and Hinton, G. Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2013. find
  39. Graves, A., Wayne, G., and Danihelka, I. Neural Turing machines. arXiv preprint arXiv:1410.5401, 2014. find
  40. Guo, M.-H., Cai, J.-X., Liu, Z.-N., Mu, T.-J., Martin, R. R., and Hu, S.-M. Pct: Point Cloud Transformer. arXiv preprint arXiv:2012.09688, 2020. — [arXiv:2012.09688](https://arxiv.org/abs/2012.09688) find
  41. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. find
  42. Hendrycks, D. and Gimpel, K. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016. find
  43. Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. AugMix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019. — [arXiv:1912.02781](https://arxiv.org/abs/1912.02781) find
  44. Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T. Axial attention in multidimensional Transformers. arXiv preprint arXiv:1912.12180, 2019. — [arXiv:1912.12180](https://arxiv.org/abs/1912.12180) find
  45. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  46. Hu, P. and Ramanan, D. Bottom-up and top-down reasoning with hierarchical rectified gaussians. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. find
  47. Kaiser, L., Gomez, A. N., Shazeer, N., Vaswani, A., Parmar, N., Jones, L., and Uszkoreit, J. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017. — [arXiv:1706.05137](https://arxiv.org/abs/1706.05137) find
  48. Kandel, E., Schwartz, J., Jessell, T., Siegelbaum, S., and Hudspeth, A. Principles of Neural Science. McGraw-Hill Education, Fifth edition, 2012. find
  49. Kant, I. Critique of Pure Reason. 1781. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. find
  50. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. Large-scale video classification with convolutional neural networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. find
  51. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are RNNs: Fast autoregressive Transformers with linear attention. In Proceedings of International Conference on Machine Learning (ICML), 2020. find
  52. Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient Transformer. In Proceedings of International Conference on Learning Representations (ICLR), 2020. find our card
  53. Köhler, W. Gestalt psychology. Psychologische Forschung, 31(1):XVIII–XXX, 1967. find
  54. Kong, Q., Xu, Y., Wang, W., and Plumbley, M. D. Audio set classification with attention model: A probabilistic perspective. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018. find
  55. Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., and Plumbley, M. D. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020. find
  56. Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012. find
  57. Kumar, M. P., Ton, P., and Zisserman, A. Obj cut. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005. find
  58. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite BERT for self-supervised learning of language representations. In Proceedings of International Conference on Learning Representations (ICLR), 2020. find
  59. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. find
  60. LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Nature, 521(7553):436–444, 2015. find
  61. Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. Set Transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of International Conference on Machine Learning (ICML), 2019. find
  62. Lee, M. A., Zhu, Y., Zachares, P., Tan, M., Srinivasan, K., Savarese, S., Fei-Fei, L., Garg, A., and Bohg, J. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3), 2020. find
  63. Lin, X., Ma, L., Liu, W., and Chang, S.-F. Context-gated convolution. In Proceedings of European Conference on Computer Vision (ECCV), 2020. find
  64. Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. In Proceedings of Neural Information Processing Systems (NeurIPS), 2020. find
  65. Lollo, V. D., Enns, J. T., and Rensink, R. A. Competition for consciousness among visual events: The psychophysics of reentrant visual processes. Journal of Experimental Psychology, 129(4):481–507, 2000. find
  66. Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorth, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of European Conference on Computer Vision (ECCV), 2020. find
  67. Nyquist, H. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 47(2):617–644, 1928. find
  68. Olshausen, B. A., Anderson, C. H., and Van Essen, D. C. A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. Journal of Neuroscience, 13(11):4700–4719, 1993. find
  69. Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V. SpecAugment: A simple data augmentation method for automatic speech recognition. In Proceedings of Interspeech, 2019. find our card
  70. Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. Image Transformer. In Proceedings of International Conference on Machine Learning (ICML), 2018. find
  71. Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N., and Kong, L. Random feature attention. In Proceedings of International Conference on Learning Representations (ICLR), 2021. find
  72. Qi, C. R., Yi, L., Su, H., and Guibas, L. J. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Proceedings of Neural Information Processing Systems (NeurIPS), 2017. find
  73. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. find
  74. Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. In Proceedings of International Conference on Learning Representations (ICLR), 2020. find our card
  75. Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in vision models. In Proceedings of Neural Information Processing Systems (NeurIPS), 2019. find
  76. Roy, A., Saffar, M., Vaswani, A., and Grangier, D. Efficient content-based sparse attention with routing Transformers. In Transactions of the Association for Computational Linguistics (TACL), 2020. find our card
  77. Santoro, A., Faulkner, R., Raposo, D., Rae, J., Chrzanowski, M., Weber, T., Wierstra, D., Vinyals, O., Pascanu, R., and Lillicrap, T. Relational recurrent neural networks. In Proceedings of Neural Information Processing Systems (NeurIPS), 2018. find
  78. Shi, J. and Malik, J. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):888–905, 2000. find
  79. Simoncelli, E. P. and Heeger, D. J. A model of neuronal responses in visual area MT. Vision Research, 38(5): 743–761, 1998. find
  80. Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Proceedings of International Conference on Learning Representations (ICLR), 2015. find our card
  81. Srinivas, A., Lin, T.-Y., Parmar, N., Shlens, J., Abbeel, P., and Vaswani, A. Bottleneck Transformers for visual recognition. arXiv preprint arXiv:2101.11605, 2021. — [arXiv:2101.11605](https://arxiv.org/abs/2101.11605) find
  82. Stanley, K. O. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8(20):131 – 162, 2007. find
  83. Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in Transformers. In Annual Meetings of the Association for Computational Linguistics, 2019. find
  84. Sutton, R. The bitter lesson. http://www. incompleteideas. net/IncIdeas/BitterLesson, 2019. find
  85. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. find
  86. Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., and Ng, R. Fourier features let networks learn high frequency functions in low dimensional domains. In Proceedings of Neural Information Processing Systems (NeurIPS), 2020. find
  87. Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. Efficient Transformers: A survey. arXiv preprint arXiv:2009.06732, 2020. find our card
  88. Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., and Zheng, C. Synthesizer: Rethinking self-attention in Transformer models. In Proceedings of International Conference on Machine Learning (ICML), 2021a. find
  89. Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long Range Arena: A benchmark for efficient Transformers. In Proceedings of International Conference on Learning Representations (ICLR), 2021b. find our card
  90. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image Transformers & distillation through attention. arXiv preprint arXiv:2012.12877, 2020. find our card
  91. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. Learning spatiotemporal features with 3D convolutional networks. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2015. find
  92. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of Neural Information Processing Systems (NeurIPS), 2017. find
  93. Wang, H., Zhu, Y., Green, B., Adam, H., Yuille, A., and Chen, L.-C. Axial-DeepLab: Stand-alone axial-attention for panoptic segmentation. In Proceedings of European Conference on Computer Vision (ECCV), 2020a. find
  94. Wang, H., Zhu, Y., Adam, H., Yuille, A., and Chen, L.-C. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. find
  95. Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020b. find
  96. Wang, W., Tran, D., and Feiszli, M. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12695–12705, 2020c. find
  97. Wang, X., Girshick, R., Gupta, A., and He, K. Non-local neural networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. find
  98. Weston, J., Chopra, S., and Bordes, A. Memory networks. In Proceedings of International Conference on Learning Representations (ICLR), 2015. find
  99. Wolfe, J. M., Reinecke, A., and Brawn, P. Why don’t we see changes? the role of attentional bottlenecks and limited visual memory. Visual Cognition, 14(4–8):749– 780, 2006. find
  100. Wu, B., Xu, C., Dai, X., Wan, A., Zhang, P., Yan, Z., Tomizuka, M., Gonzalez, J., Keutzer, K., and Vajda, P. Visual Transformers: Token-based image representation and processing for computer vision. arXiv preprint arXiv:2006.03677, 2020. — [arXiv:2006.03677](https://arxiv.org/abs/2006.03677) find
  101. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. 3D shapenets: A deep representation for volumetric shapes. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. find
  102. Xiao, F., Lee, Y. J., Grauman, K., Malik, J., and Feichtenhofer, C. Audiovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020. — [arXiv:2001.08740](https://arxiv.org/abs/2001.08740) find
  103. Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V. Nyströmformer: A Nyström-based algorithm for approximating self-attention. In Proceedings of AAAI Conference on Artificial Intelligence, 2021. find
  104. Yang, Y., Zhong, Z., Shen, T., and Lin, Z. Convolutional neural networks with alternately updated clique. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. find
  105. Ye, Z., Guo, Q., Gan, Q., Qiu, X., and Zhang, Z. BP-Transformer: Modelling long-range context via binary partitioning. arXiv preprint arXiv:1911.04070, 2019. find
  106. You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. In Proceedings of International Conference on Learning Representations (ICLR), 2020. find
  107. Yu, C., Barsim, K. S., Kong, Q., and Yang, B. Multi-level attention model for weakly supervised audio classification. In DCASE2018 Workshop on Detection and Classification of Acoustic Scenes and Events, 2018. find
  108. Zhao, H., Jia, J., and Koltun, V. Exploring self-attention for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. find
  109. Zoran, D., Chrzanowski, M., Huang, P.-S., Gowal, S., Mott, A., and Kohli, P. Towards robust image classification using sequential attention models. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. find