Elamur Research⌕
☰
home / papers / 2026 / BOLT: Benchmarking Open-World Learning for Text Classification

BOLT: Benchmarking Open-World Learning for Text Classification

Paper · 2026 · Chuan Qin, Xi Chen, Jinpeng Li et al.

Summary

Open-world text classification requires models to handle classes absent from training. Evaluation in this area has been fragmented. Most studies test on a few English intent datasets such as Banking and CLINC, and papers differ in how many classes are treated as known, how much data is labeled, and which metrics they report.

BOLT offers a single benchmark and toolkit for two tasks:

  • Open-set Text Classification (OSTC): ID inputs must be assigned to known classes and OOD inputs rejected.
  • Generalized Category Discovery (GCD): both known and new categories must be recovered from partially labeled data.

The authors assemble 12 public datasets spanning topic, intent and ontology scenarios in several languages. They run 22 methods under a shared protocol that varies the known class ratio (KCR) and the labeled ratio (LAR). Most existing approaches fit the training distribution too closely and transfer poorly to classes they have not seen. Lightweight LLM-based variants, by contrast, look promising. The resource makes fair, reproducible comparison possible in this emerging area.

Key points

  • Datasets: 20NG, THUCNews, Yahoo, X-Topic, DBPedia, TREC, CLINC, HWU, BANKING, StackOverflow, MCID and ECDT.
    • Average document length ranges from 8 to 1,856 words.
    • Label counts range from 10 to 219.
    • Languages are EN, CN, ES, JA and EL.
  • Methods: 15 OSTC methods (e.g., DOC, DeepUnk, ADB, KNNCon, DyEn) and 7 GCD methods (e.g., DAL, GeoID, SDC, TAN, LOOP, ALUP).
    • The OSTC set includes the authors' LLM variants: LLM-OpenMax, LLM-Energy and LLM-MaxLogit.
  • Settings: KCR takes values of 0.25, 0.5 and 0.75, and LAR takes 0.1, 0.5 and 1.0.
    • Metrics include ACC, F1, K-F1, N-F1, ARI and NMI.
  • Backbone scale: Scaling the Qwen2.5 backbone of LLM-MaxLogit (3B/7B/32B) does not help monotonically.
    • 32B reaches the best average N-F1 (65.86).
    • 3B has the best average K-F1 (52.89).
    • 7B mostly lags behind both.
  • Label sampling: Choosing semantically related known classes ("Similarity" sampling) consistently beats random class splits. Code and data are available at github.com/CNIC-DSL/BOLT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🔬Computer Network Information CenterBeijing · China3 works · 22 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Chuan Qin, Xi Chen, Jinpeng Li and Hengshu Zhu. 2026. BOLT: Benchmarking Open-World Learning for Text Classification. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.667

@inproceedings{qin2026bolt,
  title = {{BOLT: Benchmarking Open-World Learning for Text Classification}},
  author = {Chuan Qin, Xi Chen, Jinpeng Li and Hengshu Zhu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.667},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.667}
}

References

61 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wenbin An, Haonan Lin, Jiahao Nie, Feng Tian, Wenkai Shi, Yaqiang Wu, Qianying Wang, and Ping Chen. 2025. Unleashing the potential of model bias for generalized category discovery. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  2. Wenbin An, Wenkai Shi, Feng Tian, Haonan Lin, QianYing Wang, Yaqiang Wu, Mingxiang Cai, Luyan Wang, Yan Chen, Haiping Zhu, and Ping Chen. 2024a. Generalized category discovery with large language models in the loop. In Findings of the Association for Computational Linguistics: ACL 2024. find
  3. Wenbin An, Feng Tian, Wenkai Shi, Yan Chen, Yaqiang Wu, Qianying Wang, and Ping Chen. 2024b. Transfer and alignment network for generalized category discovery. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  4. Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, QianYing Wang, and Ping Chen. 2023. Generalized category discovery with decoupled prototypical network. In Proceedings of the AAAI conference on artificial intelligence. find
  5. Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, and Jose Camacho-Collados. 2024. Multilingual topic classification in x: Dataset and analysis. arXiv preprint arXiv:2410.03075. find
  6. Abhinav Arora, Akshat Shrivastava, Mrinal Mohit, Lorena Sainz-Maza Lecanda, and Ahmed Aly. 2020a. Cross-lingual transfer learning for intent detection of covid-19 utterances. find
  7. Abhinav Arora, Akshat Shrivastava, Mrinal Mohit, Lorena Sainz-Maza Lecanda, and Ahmed Aly. 2020b. Cross-lingual transfer learning for intent detection of covid-19 utterances. In The EMNLP 2020 Workshop NLP-COVID Submission. find
  8. Steven Basart, Mazeika Mantas, Mostajabi Mohammadreza, Steinhardt Jacob, and Song Dawn. 2022. Scaling out-of-distribution detection for real-world settings. In International Conference on Machine Learning. find
  9. Nathanaël Beau and Benoit Crabbé. 2024. CodeInsight: A curated dataset of practical coding solutions from Stack Overflow. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5935– 5947, Bangkok, Thailand. Association for Computational Linguistics. find
  10. Abhijit Bendale and Terrance E. Boult. 2016. Towards open set deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  11. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45, Online. Association for Computational Linguistics.
  12. Robin Chan, Matthias Rottmann, and Hanno Gottschalk. 2021. Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation. In Proceedings of the ieee/cvf international conference on computer vision. find
  13. Junyang Chen, Zhiguo Gong, and Weiwen Liu. 2020. A dirichlet process biterm-based mixture model for short text stream clustering. Applied Intelligence. find
  14. Xi Chen, Chuan Qin, Chuyu Fang, Chao Wang, Chen Zhu, Fuzhen Zhuang, Hengshu Zhu, and Hui Xiong. 2024. Job-sdf: A multi-granularity dataset for job skill demand forecasting and benchmarking. Advances in Neural Information Processing Systems, 37:129329–129356. find
  15. Xi Chen, Chuan Qin, Ziqi Wang, Shasha Hu, Chao Wang, Hengshu Zhu, and Hui Xiong. 2026. Beyond the known: An unknown-aware large language model for open-set text classification. In The Fourteenth International Conference on Learning Representations. find
  16. Geli Fei and Bing Liu. 2016. Breaking the closed world assumption in text classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies. find
  17. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning. PMLR. find
  18. Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations. find
  19. Xiaohan Huang, Meng Xiao, Chuan Qin, Qingqing Long, Jinmiao Chen, Yuanchun Zhou, and Hengshu Zhu. 2026. Scihorizon-gene: Benchmarking llm for life sciences inference from gene knowledge to functional understanding. arXiv preprint arXiv:2601.12805. — [arXiv:2601.12805](https://arxiv.org/abs/2601.12805) find
  20. Feihu Jiang, Chuan Qin, Kaichun Yao, Chuyu Fang, Fuzhen Zhuang, Hengshu Zhu, and Hui Xiong. 2024. Enhancing question answering for enterprise knowledge bases using large language models. In International Conference on Database Systems for Advanced Applications, pages 273–290. Springer. find
  21. Konstantin Kirchheim, Marco Filax, and Frank Ort-meier. 2022. Pytorch-ood: A library for out-of-distribution detection based on pytorch. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. find
  22. Harold W Kuhn. 2004. The hungarian method for the assignment problem. Naval Research Logistics (NRL). find
  23. Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning. find
  24. Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Confer… find
  25. Fayin Li and Harry Wechsler. 2005. Open set face recognition using transduction. IEEE transactions on pattern analysis and machine intelligence. find
  26. Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
  27. Jinggui Liang, Lizi Liao, Hao Fei, Bobo Li, and Jing Jiang. 2024. Actively learn from LLMs with uncertainty propagation for generalized category discovery. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  28. Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
  29. Bo Liu, Liming Zhan, Zexin Lu, Yujie Feng, Lei Xue, and Xiao-Ming Wu. 2023. How good are llms at out-of-distribution detection? arXiv preprint arXiv:2308.10261. find
  30. Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020. Energy-based out-of-distribution detection. Advances in neural information processing systems. find
  31. Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2021. Benchmarking natural language understanding services for building conversational agents. In Increasing naturalness and flexibility in spoken dialogue interaction: 10th international workshop on spoken dialogue systems, pages 165… find
  32. Petr Lorenc, Tommaso Gargiani, Jan Pichl, Jakub Konrád, Petr Marek, Ondřej Kobza, and Jan Šedivỳ. 2022. Metric learning and adaptive boundary for out-of-domain detection. In International Conference on Applications of Natural Language to Information Systems. Springer. find
  33. Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep learning–based text classification: a comprehensive review. ACM computing surveys (CSUR). find
  34. Sridhama Prakhya, Vinodini Venkataram, and Jugal Kalita. 2017. Open set text classification using cnns. In Proceedings of the 14th International Conference on Natural Language Processing (ICON-2017). find
  35. Chuan Qin, Xin Chen, Chengrui Wang, Pengmin Wu, Xi Chen, Yihang Cheng, Jingyi Zhao, Meng Xiao, Xiangchao Dong, Qingqing Long, and 1 others. 2025a. Scihorizon: Benchmarking ai-for-science readiness from scientific data to large language models. In Proceedings of the 31st ACM SIGKDD Conference on Know… find
  36. Chuan Qin, Chuyu Fang, Kaichun Yao, Xi Chen, Fuzhen Zhuang, and Hengshu Zhu. 2025b. Cotr: Efficient job task recognition for occupational information systems with class-incremental learning. ACM Transactions on Management Information Systems, 16(2):1–30. find
  37. Chuan Qin, Le Zhang, Yihang Cheng, Rui Zha, Dazhong Shen, Qi Zhang, Xi Chen, Ying Sun, Chen Zhu, Hengshu Zhu, and 1 others. 2025c. A comprehensive survey of artificial intelligence techniques for talent analytics. Proceedings of the IEEE. find
  38. Wenkai Shi, Wenbin An, Feng Tian, Yan Chen, Yaqiang Wu, Qianying Wang, and Ping Chen. 2024. A unified knowledge transfer network for generalized category discovery. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  39. Lei Shu, Hu Xu, and Bing Liu. 2017. Doc: Deep open classification of text documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find
  40. Lihua Wang, Huiting Yang, Feng Li, Wenzhong Yang, and Zhenwan Zou. 2022. Intent detection model based on dual-channel feature fusion. In 2022 IEEE 6th Information Technology and Mechatronics Engineering Conference (ITOEC), volume 6, pages 1862– 1867. IEEE. find
  41. Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. 2022. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review. find
  42. Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. 2022. Mitigating neural network overconfidence with logit normalization. In International conference on machine learning. PMLR. find
  43. Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021a. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. arXiv preprint arXiv:2105.14289. find
  44. Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021b. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisti… find
  45. Zhiheng Song, Jingshuai Zhang, Chuan Qin, Chao Wang, Chao Chen, Longfei Xu, Kaikui Liu, Xiangxiang Chu, and Hengshu Zhu. 2026. Mobilitybench: A benchmark for evaluating route-planning agents in real-world mobility scenarios. arXiv preprint arXiv:2602.22638. — [arXiv:2602.22638](https://arxiv.org/abs/2602.22638) find
  46. Feng Zhang, Wei Chen, Fei Ding, Meng Gao, Tengjiao Wang, Jiahui Yao, and Jiabin Zheng. 2024a. From discrimination to generation: Low-resource intent detection with language model instruction tuning. In Findings of the Association for Computational Linguistics ACL 2024. find
  47. Maosong Sun, Jingyang Li, Zhipeng Guo, Zhao Yu, Yabiing Zheng, Xiance Si, and Zhiyuan Liu. 2016. Thuctc: An efficient chinese text classifier. find
  48. Hanlei Zhang, Hua Xu, and Ting-En Lin. 2021a. Deep open intent classification with adaptive decision boundary. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  49. Kai Tang, Junbo Zhao, Xiao Ding, Runze Wu, Lei Feng, Gang Chen, and Haobo Wang. 2024. Learning geometry-aware representations for new intent discovery. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. find
  50. Zhenyu Tong, Chuan Qin, Chuyu Fang, Kaichun Yao, Xi Chen, Jingshuai Zhang, Chen Zhu, and Hengshu Zhu. 2025. From missteps to mastery: Enhancing low-resource dense retrieval through adaptive query generation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1… find
  51. Fengjun Wang, Moran Beladev, Ofri Kleinfeld, Elina Frayerman, Tal Shachar, Eran Fainman, Karen Last-mann Assaraf, Sarai Mizrachi, and Benjamin Wang. 2023. Text2topic: Multi-label text classification system for efficient topic detection in user generated content with zero-shot capabilities. In Procee… find
  52. Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021b. Discovering new intents with deep aligned clustering. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  53. Hanlei Zhang, Hua Xu, Xin Wang, Fei Long, and Kai Gao. 2023. A clustering framework for unsupervised and semi-supervised new intent discovery. IEEE Transactions on Knowledge and Data Engineering. find
  54. Shun Zhang, Chaoran Yan, Jian Yang, Changyu Ren, Jiaqi Bai, Tongliang Li, and Zhoujun Li. 2024b. Ronid: new intent discovery with generated-reliable labels and cluster-friendly representations. In International Conference on Database Systems for Advanced Applications. find
  55. Wei-Nan Zhang, Zhigang Chen, Wanxiang Che, Guoping Hu, and Ting Liu. 2017. The first evaluation of chinese human-computer dialogue technology. arXiv preprint arXiv:1709.10217. — [arXiv:1709.10217](https://arxiv.org/abs/1709.10217) find
  56. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015a. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems.
  57. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
  58. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
  59. Qi Zhou, Xi Chen, Chuyu Fang, Jianji Wang, Chuan Qin, and Fuzhen Zhuang. 2025. Enhancing dual-target cross-domain recommendation via similar user bridging. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 4487–4497. find
  60. Yunhua Zhou, Peiju Liu, and Xipeng Qiu. 2022. Knn-contrastive learning for out-of-domain intent classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find
  61. Yunhua Zhou, Jianqiang Yang, Pengyu Wang, and Xipeng Qiu. 2023. Two birds one stone: Dynamic ensemble for ood intent classification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qin-2026-bolt-benchmarking-open-world-learning/