Elamur Research⌕
☰
home / papers / 2026 / Expert Calibration Lens for Pruning Mixture of Experts

Expert Calibration Lens for Pruning Mixture of Experts

Paper · 2026 · Luis Frentzen Salim, Chia-Chun Wu, Tran Van Nhiem et al.

Summary

Expert pruning makes Mixture-of-Experts (MoE) models cheaper to deploy, but its results depend heavily on the calibration set. The effects of changing that set are hard to predict: adding more samples may not help, and data in another language can work about as well. Normally each such question requires a full pruning run. The authors present Expert Calibration Lens, a demo system built for gpt-oss-20b. It hooks into each MoE layer and records dense router probabilities, plus an optional "feature energy" estimate of how much each expert contributes. It then compares datasets through these activation statistics in a web interface, so no pruning is needed. Case studies use C4, MMLU ProX and a proprietary Traditional Chinese FoxBrain corpus. Evaluation runs on MMLU and MathQA through LM Evaluation Harness. The analysis shows that the set of selected experts stabilizes after a few dozen sequences. It also shows that Chinese and English calibration choose largely the same experts, and the pruned models score similarly downstream. Practitioners can therefore check calibration choices cheaply before committing to expensive pruning runs.

Key points

  • Metrics: The tool computes metrics for single datasets (sparsity, variance, effective number of experts via inverse participation ratio, Shannon entropy). It also compares pairs of datasets (Jensen-Shannon divergence, Jaccard overlap of top-r experts, Pearson correlation of feature energy, and asymmetric retained mass A→B).
  • Calibration size: Jaccard similarity of the retained experts converges after roughly 32–64 sequences. Larger calibration sets brought no consistent gains (tested with 24 retained experts).
  • Cross-lingual calibration: Across 7 MMLU ProX languages (ar, de, en, fr, ja, ru, zh), calibrating in another language performed comparably to calibrating in the evaluation language.
  • Downstream scores (Table 1): The unpruned baseline scores 0.365 MathQA / 0.563 MMLU. After pruning, C4 calibration gives 0.3889 / 0.617, the private Chinese set 0.405 / 0.611, and a MATH-derived set 0.408 / 0.585.
  • Availability and scope: The system is part of the FoxBrain project (Hon Hai) and is currently proprietary. It supports only the GPT-OSS architecture, and the public demo is limited to three precomputed activation sets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Institute of Information Science, Academia SinicaTaipei · Taiwan7 works · 51 citations🎓National Taiwan University of Science and TechnologyTaipei · Taiwan4 works · 6 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Luis Frentzen Salim, Chia-Chun Wu, Tran Van Nhiem, Lun-Wei Ku and Yung-Hui Li. 2026. Expert Calibration Lens for Pruning Mixture of Experts. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-demo.72

@inproceedings{salim2026expert,
  title = {{Expert Calibration Lens for Pruning Mixture of Experts}},
  author = {Luis Frentzen Salim, Chia-Chun Wu, Tran {Van Nhiem}, Lun-Wei Ku and Yung-Hui Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.72},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.72}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computation…
  2. Lucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu, and Nanyun Peng. 2025. Multilingual routing in mixture-of-experts. In The Fourteenth International Conference on Learning Representations. find
  3. Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. 2022. Task-specific expert pruning for sparse mixture-of-experts. Preprint, arXiv:2206.00277. — [arXiv:2206.00277](https://arxiv.org/abs/2206.00277) find
  4. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language mo… find
  5. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. Preprint, arXiv:2104.08758. find
  6. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In The Ninth International Conference on Learning Representations (ICLR). ArXiv:2009.03300.
  8. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  9. Slimmoe: Structured compression of large moe models via expert slimming and distillation. Preprint, arXiv:2506.18349. find
  10. Neel Nanda and Joseph Bloom. 2022. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens. ↗
  11. Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. Preprint, arXiv:2201.05596.
  12. Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021. Scaling vision with sparse mixture of experts. Preprint, arXiv:2106.05974. find
  13. Luis Frentzen Salim, Lun-Wei Ku, and Hsing-Kuo Kenneth Pao. 2026. Positional cognitive specialization: Where do llms learn to comprehend and speak your language? Proceedings of the AAAI Conference on Artificial Intelligence, 40(39):32875–32883. find
  14. Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. 2020. The language interpretability tool: Extensible, interactive visualizations and analysis for nlp models. Preprint, arXiv:2008.0… find
  15. Igor Tufanov, Karen Hambardzumyan, Javier Ferrando, and Elena Voita. 2024. LM transparency tool: Interactive tool for analyzing transformer language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 51–60, Bang… find
  16. Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. 2024. Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router. Preprint, arXiv:2410.12013. — [arXiv:2410.12013](https://arxiv.org/abs/2410.12013) find
  17. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, and 13 others. 2025. Mmlu-prox: A multilingual benchmark for advanced large l…
  18. Zichong Li, Chen Liang, Zixuan Zhang, Ilgee Hong, Young Jin Kim, Weizhu Chen, and Tuo Zhao. 2025. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salim-2026-expert-calibration-lens-pruning-mixture/