Elamur Research⌕
☰
home / papers / 2022 / MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

Paper · 2022 · Zhengyan Zhang, Yankai Lin, Zhiyuan Liu et al.

Summary

Feed-forward networks (FFNs) hold roughly two-thirds of a Transformer's parameters, yet how they compute has received far less study than attention. The authors observe that FFN activations are highly sparse. In a fine-tuned T5-Large using ReLU, 90% of inputs switch on fewer than 5% of neurons. They ask whether pre-trained FFNs therefore contain functional partitions, analogous to regions of the brain.

To test this, they propose MoEfication, which rewrites an existing model as a Mixture-of-Experts while keeping its parameters. It has two steps:

  1. Expert construction: neurons that tend to fire together are grouped into equal-sized experts, using either balanced K-Means over weight vectors or a co-activation graph split.
  2. Expert selection: a router chooses, for each input, the experts likely to contain its active neurons.

Experiments with T5 and BERT on GLUE and QA benchmarks show that activating only 10–30% of FFN parameters keeps over 95% of the original performance. This yields practical inference speedups, especially on CPU. It also gives a finer-grained way to inspect how FFNs operate internally.

Key points

  • Sparse activation: for a fine-tuned T5-Large (~700M parameters), 90% of inputs activate under 5% of FFN neurons. A neuron counts as active when its ReLU output is positive.
  • Speedup: using 25% of FFN parameters gives about 2x speedup in FLOPS and on CPU (Intel Broadwell), but only about 1.2x on GPU (NVIDIA Tesla V100). Gains shrink at lower ratios because attention cost stays fixed and becomes the bottleneck.
  • Recommended ratio: the authors suggest about 20% of FFN parameters as a good balance between speed and accuracy.
  • Routing imbalance: in MoEfied T5-Small (64 experts per encoder layer, 20% selected, SST-2), some "general" experts serve over 80% of inputs. Others are rarely used (under 10%) and handle tightly clustered, input-specific inputs.
  • Code: released at github.com/thunlp/MoEfication.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🏢TencentShenzhen · China66 works · 17,945 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China21 works · 5,843 citations🔬Beijing Academy of Artificial IntelligenceBeijing · China16 works · 7,166 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun and Jie Zhou. 2022. MoEfication: Transformer Feed-forward Layers are Mixtures of Experts. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-acl.71

@inproceedings{zhang2022moefication,
  title = {{MoEfication: Transformer Feed-forward Layers are Mixtures of Experts}},
  author = {Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun and Jie Zhou},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.71},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.71}
}

References

57 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael R. Lyu, and Irwin King. 2021. Binarybert: Pushing the limit of BERT quantization. In Proceedings of ACL/IJCNLP, pages 4334–4348.
  2. Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Maxout networks. In Proceedings of ICML, pages 1319– 1327. find
  3. Yoshua Bengio. 2013. Deep learning of representations: Looking forward. In Proceedings of SLSP, pages 1–37. find
  4. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Ch…
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment Charles G Gross. 2002. Genealogy of the “grandmother cell”. The Neuroscientist, 8(5):512–518. find
  6. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-Trained… find
  7. Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. 2021. FastMoE: A fast Mixture-of-Expert training system. arXiv preprint 2103.13262. find
  8. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (GELUs). arXiv preprint 1606.08415. 885 9 find
  9. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  10. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint 1207.0580. find
  11. Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020. Dynabert: Dynamic BERT with adaptive width and depth. In Proceedings of NeurIPS. find
  12. Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. 2018. Multi-Scale dense networks for resource efficient image classification. In Proceedings of ICLR. find
  13. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural Comput., 3(1):79–87. find
  14. Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. BASE layers: Simplifying training of large, sparse models. arXiv preprint 2103.16716. find
  15. Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2021. CascadeBERT: Accelerating inference of pre-trained language models via calibrated complete models cascade. In Findings of EMNLP, pages 475–486. find
  16. Mikko I. Malinen and Pasi Fränti. 2014. Balanced kmeans for clustering. In Proceedings of SSSPR, volume 8621, pages 32–41. find
  17. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proc. Natl. Acad. Sci. USA, 117(48):30046–30054. find
  18. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual knowledge in gpt. arXiv preprint arXiv:2202.05262. find
  19. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of ACL, pages 3651– 3657.
  20. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of EMNLP, pages 4163–4174. find
  21. George Karypis and Vipin Kumar. 1998. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput., 20(1):359–392. find
  22. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of ICLR.
  23. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of EMNLP-IJCNLP, pages 4365–4374.
  24. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of NeurIPS, pages 1106–1114. find
  25. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of EMNLP, pages 785–794.
  26. Dmitry Lepikhin, Hyoukjoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. GShard: Scaling giant models with conditional computation and automatic sharding. In Proceedings of ICLR. find
  27. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In Proceedings of ICLR.
  28. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Proceedings of NeurIPS, pages 14014–14024.
  29. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of ICML, pages 807–814. find
  30. Bruno A Olshausen and David J Field. 1996. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609. find
  31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified Text-to-Text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  32. Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation ai scale. arXiv preprint arXiv:2201.05596.
  33. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383–2392.
  34. Sahana Ramnath, Preksha Nema, Deep Sahni, and Mitesh M. Khapra. 2020. Towards interpreting BERT for reading comprehension based QA. In Proceedings of EMNLP, pages 3236–3242. 886 10 find
  35. Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. 2021. Hash layers for large sparse models. arXiv preprint arXiv:2106.04426. — [arXiv:2106.04426](https://arxiv.org/abs/2106.04426) find
  36. Wenxuan Wang and Zhaopeng Tu. 2020. Rethinking the value of transformer components. In Proceedings of COLING, pages 6019–6029. find
  37. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint 1910.01108. find
  38. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. TACL, 7:625–641. find
  39. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, pages 1112–1122. find
  40. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The Sparsely-Gated Mixture-of-Experts layer. In Proceedings of ICLR. find
  41. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631–1642. find
  42. Xavier Suau, Luca Zappella, and Nicholas Apostoloff. 2020. Finding experts in transformer models. arXiv preprint arXiv:2005.07647. — [arXiv:2005.07647](https://arxiv.org/abs/2005.07647) find
  43. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Proceedings of EMNLP, pages 4323– 4332.
  44. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact Task-Agnostic BERT for Resource-Limited devices. In Proceedings of ACL, pages 2158–2170.
  45. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of ACL, pages 4593–4601.
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, and Jakob Uszkoreit. 2017. Attention is all you need. In Proceedings of NeurIPS, pages 5998–6008.
  47. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing Multi-Head Self-Attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of ACL, pages 5797–5808.
  48. Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, Yi Yang, and Shilei Wen. 2020. Dynamic inference: A new approach toward efficient video action recognition. arXiv preprint 2002.03342. — [arXiv:2002.03342](https://arxiv.org/abs/2002.03342) find
  49. Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of ACL, pages 2246–2251.
  50. Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. 2021. TR-BERT: dynamic token reduction for accelerating BERT inference. In Proceedings of NAACL-HLT, pages 5798–5809. find
  51. Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8BERT: Quantized 8bit BERT. arXiv preprint 1910.06188. — [arXiv:1910.06188](https://arxiv.org/abs/1910.06188) find
  52. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Proceedings of NeurIPS.
  53. Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. TernaryBERT: Distillation-aware ultra-low bit BERT. In Proceedings of EMNLP, pages 509–521.
  54. Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Qun Liu, and Maosong Sun. 2021. Know what you don’t need: Single-Shot Meta-Pruning for attention heads. AI Open, 2:36–42. find
  55. Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. 2019. AllenNLP interpret: A framework for explaining predictions of NLP models. In Proceedings of EMNLP-IJCNLP, pages 7–12.
  56. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR.
  57. Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. 887 11 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2022-moefication-transformer-feed-forward-layers/