Elamur Research⌕
☰
home / papers / 2026 / TeleAI at SemEval-2026 Task 6: A Confidence-Aware Multi-Stage Reasoning Framework with Chain-of-Thought

TeleAI at SemEval-2026 Task 6: A Confidence-Aware Multi-Stage Reasoning Framework with Chain-of-Thought

Paper · 2026 · Lingling Shi, Haoyu Jin, Shiquan Wang et al.

Summary

The paper presents TeleAI's winning system for SemEval-2026 Task 6 (CLARITY). The task asks whether answers in English political interviews actually respond to a given sub-question. SubTask 1 uses three coarse clarity labels, and SubTask 2 uses nine fine-grained evasion categories. The authors propose CAMSR-CoT, a prompting pipeline built on DeepSeek-V3 that splits the decision into stages:

  • A confidence-scoring Gate first screens for the three Non-Reply classes and releases only the clear-cut refusals directly.
  • Uncertain Non-Reply candidates receive a 9-class correction review.
  • All remaining answers are labelled within a reduced 6-class space.

Every stage uses Chain-of-Thought prompting with refined label definitions, guidance on frequently confused label pairs, and contrastive few-shot examples. These examples were mined from the model's zero-shot confusion matrix on the 3,448-sample training set. SubTask 1 labels are not predicted separately; they are obtained from the fine-grained prediction through a fixed mapping. The approach placed first on the hidden test set in both subtasks. It also shows that staged label-space reduction and selecting examples by their decision boundaries can help with highly ambiguous discourse labels.

Key points

  • Official ranking: 1st on the hidden test set, with Macro-F1 of 0.89 (SubTask 1) and 0.68 (SubTask 2). The margins over second place were about 4 and 7 points.
  • Development set (308 multi-reference samples): CAMSR-CoT reached 0.812 / 0.617 Macro-F1. The single-step CoT baseline scored 0.710 / 0.490, and direct 9-class prompting scored 0.662 / 0.421.
  • Ablations: Confidence-aware Gate routing gave the largest SubTask 1 gain (+0.101). Boundary few-shot examples in Step 3 gave the largest SubTask 2 gain (+0.097). All three Non-Reply classes reached precision 1.000 on dev.
  • Cost and alternative backbones:
    • The pipeline averages 1.94 model calls per sample.
    • With other backbones, Qwen3-235B-A22B and TeleChat3-36B-Thinking scored lower than DeepSeek-V3.
    • Fine-tuned Qwen2.5-7B models (SFT, SFT + distillation) also stayed below the prompting approach.
  • Main weakness: The Dodging class (recall 0.179) is often confused with General and Deflection, mirroring disagreement among human annotators. Only 40.6% of dev samples had unanimous labels. Code is available on GitHub (ther7777/semeval-2026-task6-camsr-cot).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢China TelecomBeijing · China7 works · 16 citations🏢China Telecom (China)Changsha · China6 works · 15 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Lingling Shi, Haoyu Jin, Shiquan Wang, Fang Yu, Shuangyong Song and Xuelong Li. 2026. TeleAI at SemEval-2026 Task 6: A Confidence-Aware Multi-Stage Reasoning Framework with Chain-of-Thought. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.206

@inproceedings{shi2026teleai,
  title = {{TeleAI at SemEval-2026 Task 6: A Confidence-Aware Multi-Stage Reasoning Framework with Chain-of-Thought}},
  author = {Lingling Shi, Haoyu Jin, Shiquan Wang, Fang Yu, Shuangyong Song and Xuelong Li},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.206},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.206}
}

References

40 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hongjun An, Wenhan Hu, Sida Huang, Siqi Huang, Ruanjun Li, Yuanzhi Liang, Jiawei Shao, Yiliang Song, Zihan Wang, Cheng Yuan, Chi Zhang, Hongyuan Zhang, Wenhao Zhuang, and Xuelong Li. 2026. Ai flow: perspectives, scenarios, and approaches. Vicinagearth, 3(1). find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  3. Peter Bull. 1994. On identifying questions, replies, and non-replies in political interviews. Journal of Language and Social Psychology, 13(2):115–131. find
  4. Wenhan Chang, Tianrui Zhu, Yue Zhao, Shuangyong Song, Peng Xiong, Wenhao Zhou, and Yongxiang Li. 2025. Chain-of-lure: A synthetic narrative-driven approach to compromise large language models. Preprint, arXiv:2505.17519. — [arXiv:2505.17519](https://arxiv.org/abs/2505.17519) find
  5. Jinyang Chen, Haolun Wu, Jianhong Pang, Yihua Wang, Dell Zhang, and Changzhi Sun. 2025. Tool learning with language models: a comprehensive survey of methods, pipelines, and benchmarks. Vicinagearth, 2(1). find
  6. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… find
  7. Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. 2024. Active prompting with chain-of-thought for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1330–1350, Bangkok, Thailand. A… find
  8. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, Miami…
  9. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  11. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. Dspy: Compiling declarative language model calls into self-improving pipelines. I…
  12. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
  13. Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Chao Wang, Xinzhang Liu, Zihan Wang, Yu Zhao, Xin Wang, Yuyao Huang, Shuangyong Song, Yongxiang Li, Zhongjiang He, Xuelong Li, and 1 others. 2024. Tele-FLM technical report. Preprint, arXiv:2404.16645. — [arXiv:2404.16645](https://arxiv.org/abs/2404.16645) find
  14. Zhongqiu Li, Shiquan Wang, Ruiyu Fang, Mengjiao Bao, Zhenhe Wu, Shuangyong Song, Yongxiang Li, and Xuelong Li. 2025. MR-UIE: Multi-perspective reasoning with reinforcement learning for universal information extraction. Vicinagearth, 2(1). find
  15. Xinzhang Liu, Chao Wang, Zhihao Yang, Zhuo Jiang, Xuncheng Zhao, Haoran Wang, Lei Li, Dongdong He, Luobin Liu, Kaizhe Yuan, Han Gao, Zihan Wang, Yitong Yao, Sishi Xiong, Wenmin Deng, Haowei He, Kaidong Yu, Yu Zhao, Ruiyu Fang, and 35 others. 2025. Training report of telechat3-moe. Preprint, arXiv:25… — [arXiv:2512.24157](https://arxiv.org/abs/2512.24157) find
  16. Uche Christina Okwuonu, Helen Kwipnchep Njoya, and Uzoigwe Tracy Peremoboere. 2025. Enhancing math reasoning ability of large language models via computation logic graphs. Knowledge-Based Systems. find
  17. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — Training language models to follow instructions with human feedback
  18. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511, Baltimore, Maryland. Association for Computational Linguistics.
  19. Qwen Team. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  20. Parameswary Rasiah. 2010. A framework for the systematic analysis of evasion in parliamentary discourse. Journal of Pragmatics, 42(3):664–680. find
  21. Yiqing Shen and Dell Zhang. 2025. A survey of language-guided video object segmentation: from referring to reasoning. Vicinagearth, 2(1). find
  22. Lingling Shi, Haoyu Jin, Ruiyu Fang, Shuangyong Song, Jinsong Su, Yongxiang Li, and Xuelong Li. 2026. CEMT: Chain-of-thought enhanced machine translation. In Findings of the Association for Computational Linguistics: ACL 2026. Association for Computational Linguistics. find
  23. Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, and Giorgos Stamou. 2024. “I never said that”: A dataset, taxonomy and baselines on response clarity classification. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5204–5233, Miami, Florida… find
  24. Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, and Giorgos Stamou. 2026. Semeval-2026 task 6: CLARITY – unmasking political question evasions. Preprint, arXiv:2603.14027.
  25. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. The Journal of Artificial Intelligence Research, 72:1385–1470. find
  26. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023a. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… find
  27. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171.
  28. Zihan Wang, Xinzhang Liu, Shixuan Liu, Yitong Yao, Yuyao Huang, Zhongjiang He, Xuelong Li, and 1 others. 2024a. Telechat technical report. Preprint, arXiv:2401.03804.
  29. Zihan Wang, Xinzhang Liu, Shixuan Liu, Yitong Yao, and 1 others. 2024b. TeleChat: An open-source bilingual large language model. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10). Association for Computational Linguistics. find
  30. Zihan Wang, Xinzhang Liu, Yitong Yao, Chao Wang, Yu Zhao, Zhihao Yang, Wenmin Deng, Kaipeng Jia, Jiaxin Peng, Yuyao Huang, Sishi Xiong, Zhuo Jiang, Kaidong Yu, Xiaohui Hu, Fubei Yao, Ruiyu Fang, Zhuoru Jiang, Ruiting Song, Qiyi Xie, and 19 others. 2025. Technical report of telechat2, telechat2.5 and… — [arXiv:2507.18013](https://arxiv.org/abs/2507.18013) find
  31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processi…
  32. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, pages 11809–11822. Curran Associates, Inc. find
  33. Yiqun Yao, Xiang Li, Xin Jiang, and 1 others. 2024. 52B to 1T: Lessons learned via Tele-FLM series. Preprint, arXiv:2407.02783. — [arXiv:2407.02783](https://arxiv.org/abs/2407.02783) find
  34. Wei Zhang, Hongcheng Guo, Jian Yang, and 1 others. 2024. Lemur: Log parsing with entropy sampling and chain-of-thought merging. Preprint, arXiv:2402.18205. — [arXiv:2402.18205](https://arxiv.org/abs/2402.18205) find
  35. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410, Bangkok, Thail…
  36. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. Preprint, arXiv:2205.10625.
  37. Di Wu, Ruiyu Fang, Liting Jiang, Shuangyong Song, Xiaomeng Huang, Shiquan Wang, Zhongqiu Li, Lingling Shi, Mengjiao Bao, Yongxiang Li, and Hao Huang. 2025. Multi-intent spoken language understanding: a survey of methods, trends, and challenges. Vicinagearth, 2(1). find
  38. Hongrui Xing, Xinzhang Liu, Zhuo Jiang, Zhihao Yang, Yitong Yao, Zihan Wang, and 1 others. 2025. LLMSR@XLLM25: A language model-based pipeline for structured reasoning data construction. In Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025). Association… find
  39. Sishi Xiong, Dakai Wang, Yu Zhao, Jie Zhang, Changzai Pan, Haowei He, Xiangyu Li, Wenhan Chang, Xuelong Li, and 1 others. 2025a. TableReasoner: Advancing table reasoning framework with large language models. Preprint, arXiv:2507.08046. — [arXiv:2507.08046](https://arxiv.org/abs/2507.08046) find
  40. Sishi Xiong, Dakai Wang, Yu Zhao, Jie Zhang, Changzai Pan, Haowei He, Xiangyu Li, Wenhan Chang, Xuelong Li, and 1 others. 2025b. TeleAI at SemEval-2025 task 8: Advancing table reasoning framework with large language models. In Proceedings of the 19th International Workshop on Semantic Evaluation (Se… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-teleai-semeval-2026-task-6/