Elamur Research⌕
☰
home / papers / 2026 / MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification

Paper · 2026 · Bo Zheng, Yudong Chen, Zihua Xiong et al.

Summary

Industrial tables used in finance and similar high-stakes domains are wide, sparse, and mostly unlabeled. Their missing values often carry meaning, yet standard pipelines built around GBDTs such as XGBoost usually drop or impute them. The authors propose MaskTab, a self-supervised pretraining framework for this kind of data, built on a Transformer encoder. Feature names are embedded with a frozen BERT, and values get type-specific encoders. Absent values are represented by learnable [MISS] tokens, which share their initialization with the [MASK] tokens used for masked-value reconstruction. Training follows a twin-path hybrid scheme that combines reconstruction with supervised classification. A Mixture-of-Experts reconstruction head distributes capacity across heterogeneous features. On industrial benchmarks, MaskTab beats prior methods by 5.04% AUC and 8.28% KS in relative terms. The authors study scaling along three axes: unlabeled data volume, feature count, and model size. They also distill the largest model into a compact student restricted to 500 interpretable features. The work suggests tabular data can be handled like other foundation-model domains, while still meeting production limits on latency and auditability.

Key points

  • Scaling with a 500-feature, 25.2M-parameter setup: adding unlabeled data on top of 330K labeled rows helped up to a 20× ratio, reaching OOT KS 0.3594. Going to 40× made results worse.
  • Further scaling: KS rose with feature count up to 2,000 (KS 0.3754). It then rose with model size up to MaskTab-L at 134M parameters (KS 0.4145). MaskTab-XL (201M) and 2,500 features both showed saturation.
  • Missing-value handling: learnable mask embeddings outperformed zero/mode filling, HyperImpute, and ReMasker. They achieved AUC 0.8698 on HomeCredit and RMSE 0.2345 on Sberbank.
  • Distillation: MaskTab-L-2000 was distilled into MaskTab-Distill-500. The student improves KS by 4.04% over MaskTab-Base-500 and runs inference 9.3× faster. Compared with traditional models, it gains 2.55% AUC and 4.85% KS.
  • Robustness to time shift: models were trained on 2024-01–06 and tested on 2024-07–12. XGBoost showed a large train–OOT KS gap (>0.60 vs ≈0.35), while MaskTab-Distill had the smallest gap.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations🏢Ant GroupHangzhou · China51 works · 1,211 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEfficient models & compression · 297 works

Cite

Cite as Bo Zheng, Yudong Chen, Zihua Xiong, Shuai Fang, Peidong He, Yang Yang and Sheng Guo. 2026. MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2053

@inproceedings{zheng2026masktab,
  title = {{MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification}},
  author = {Bo Zheng, Yudong Chen, Zihua Xiong, Shuai Fang, Peidong He, Yang Yang and Sheng Guo},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2053},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2053}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sercan Ö Arik and Tomas Pfister. 2021. TabNet: Attentive interpretable tabular learning. In AAAI. find
  2. Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wei Chen, and Tien-Hao Chang. 2023. Trompt: Towards a better deep neural network for tabular data. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 4392–4434. PMLR. find
  3. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In SIGKDD. find
  4. Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, and 1 others. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional Transformers for language understanding. In NAACL.
  6. Tianyu Du, Luca Melis, and Ting Wang. 2024. Remasker: Imputing tabular data with masked autoencoding. In ICLR. find
  7. Yury Gorishniy, Ivan Rubachev, and Artem Babenko. 2022. On embeddings for numerical features in tabular deep learning. In NeurIPS. find
  8. Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. 2024. Tabr: Tabular deep learning meets nearest neighbors in 2023. In ICLR. find
  9. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021. Revisiting deep learning models for tabular data. In NeurIPS. find
  10. Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, Mihir Manium, Rosen Yu, Felix Jablonski, Shi Bin Hoo, Anurag Garg, Jake Robertson, Magnus Bühler, Vladyslav Moroshan, Lennart Purucker, and 7 others. 202… — [arXiv:2511.08667](https://arxiv.org/abs/2511.08667) find
  11. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  12. Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. 2024. Tabred: A benchmark of tabular machine learning in-the-wild. arXiv, 2406.19380. find
  13. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, and 1 others. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
  14. Ruoxi Wang, Rakesh Shivanna, Derek Z. Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. 2020. Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. arXiv, 2008.13535v2. find
  15. Daniel Jarrett, Bogdan Cebere, Tennison Liu, Alicia Curth, and Mihaela van der Schaar. 2022. Hyperimpute: Generalized iterative imputation with automatic model selection. In ICML. find
  16. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  17. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146–3154. find
  18. Guolin Ke, Zhenhui Xu, Jia Zhang, Jiang Bian, and Tie-Yan Liu. 2019. Deepgbm: A deep learning framework distilled by gbdt for online prediction tasks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 384–394. find
  19. Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. In NeurIPS. find
  20. Weijiang Lai, Beihong Jin, Jiongyan Zhang, Yiyuan Zheng, Jian Dong, Jia Cheng, Jun Lei, and Xingxing Wang. 2025. Exploring scaling laws of ctr model for online performance improvement. In Proceedings of the Nineteenth ACM Conference on Recommender Systems, pages 114–123. find
  21. Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L Caterini. 2024. Tabdpt: Scaling tabular foundation models. arXiv preprint arXiv:2410.18164. — [arXiv:2410.18164](https://arxiv.org/abs/2410.18164) find
  22. Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2018. Catboost: unbiased boosting with categorical features. In NeurIPS. find
  23. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, and 1 others. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. find
  24. Zifeng Wang and Jimeng Sun. 2022. TransTab: Learning transferable tabular Transformers across tables. In NeurIPS. find
  25. Qingren Yao, Chao-Han Huck Yang, Renhe Jiang, Yuxuan Liang, Ming Jin, and Shirui Pan. 2024. Towards neural scaling laws for time series foundation models. arXiv preprint arXiv:2410.12360. — [arXiv:2410.12360](https://arxiv.org/abs/2410.12360) find
  26. Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. 2024. Towards cross-table masked pretraining for web data mining. In WWW. find
  27. Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104–12113. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-masktab-scalable-masked-tabular-pretraining/