Elamur Research⌕
☰
home / papers / 2026 / Component Transfer Can Exceed Full Model Performance: Investigating Post-Trained Mixture-of-Experts

Component Transfer Can Exceed Full Model Performance: Investigating Post-Trained Mixture-of-Experts

Paper · 2026 · Rabin Tiwari

Summary

Post-trained language models are usually compared only by aggregate benchmark scores, which hides which internal parts carry the improvements. This paper examines that question for Mixture-of-Experts models using two instruction-tuned OLMoE-1B-7B releases. One is an older SFT+DPO checkpoint (0924). The other is a newer checkpoint built with the Tulu 3 recipe (0125). With the older model as the base, the author swaps in routers, attention blocks and the 64 expert MLPs per layer from the Tulu3 model. Each component is transferred alone and in every combination. Exact-match accuracy is measured on GSM8K (chain-of-thought), BoolQ (zero-shot) and ARC-Challenge (zero-shot and 5-shot). The gains turn out to be highly localized. Experts drive the improvements in math and few-shot science. Attention matters more for the general yes/no task. Routers on their own add little or reduce accuracy. Some hybrid models outperform both originals with no extra training. The author concludes that aggregate evaluation is not enough and that component-level analysis can guide selective model merging.

Key points

  • On BoolQ, combining Tulu3 attention and experts gives 79.2% ± 1.4%. This beats the Tulu3 checkpoint (75.4%, p < 0.001) and the SFT+DPO checkpoint (78.0%, p = 0.013).
  • On ARC-Challenge 5-shot, transferring only the experts reaches 62.5%. The Tulu3 checkpoint scores 60.9% and the SFT+DPO checkpoint 60.4%.
  • On GSM8K, Tulu3 far outperforms SFT+DPO (68.5% vs 38.3%). The attention+experts hybrid nearly closes that gap at 67.6%.
  • Prompting reverses the expert effect on ARC-Challenge. Expert transfer costs about 7.4 points zero-shot but adds about 4.6 points with 5-shot prompting (both measured on 500-item samples).
  • Evaluation protocol:
    • Most configurations are scored on 500 random test items per task, with full test sets reserved for key configurations.
    • Significance uses two-proportion z-tests and 10,000-sample bootstrap confidence intervals.
    • A proposed component-selection heuristic picks the best configuration in 3 of 4 conditions.
    • The study covers a single model family.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Rabin Tiwari. 2026. Component Transfer Can Exceed Full Model Performance: Investigating Post-Trained Mixture-of-Experts. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.7

@inproceedings{tiwari2026component,
  title = {{Component Transfer Can Exceed Full Model Performance: Investigating Post-Trained Mixture-of-Experts}},
  author = {Rabin Tiwari},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.7},
  url = {https://doi.org/10.18653/v1/2026.gem-main.7}
}

References

14 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
  2. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? ArXiv, abs/1905.10650.
  3. Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Daniel Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, and 5 others. 2024. O… find
  4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457.
  5. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill. Https://distill.pub/2020/circuits/zoom-in. find
  6. Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plap-Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie… — Training language models to follow instructions with human feedback
  7. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model.
  8. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. ArXiv, abs/1707.06347. find
  9. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. find
  10. Sheng Shen, Le Hou, Yan-Quan Zhou, Nan Du, S. Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Y. Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. 2023. Mixture-of-experts meets instruction…
  11. Kyuhong Shim, Jungwook Choi, and Wonyong Sung. 2022. Understanding the role of self attention for efficient speech recognition. In International Conference on Learning Representations. find
  12. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
  13. Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing in…
  14. Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. 2024. Investigating layer importance in large language models. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 469–479, Miami, Florida, US. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tiwari-2026-component-transfer-can-exceed-full/