Elamur Research⌕
☰
home / papers / 2026 / Confidence over Time: Confidence Calibration with Temporal Logic for Large Language Model Reasoning

Confidence over Time: Confidence Calibration with Temporal Logic for Large Language Model Reasoning

Paper · 2026 · Zhenjiang Mao, Anirudhh Venkat, Artem Bisliouk et al.

Summary

Most confidence estimators for LLMs reduce a whole multi-step answer to one number, such as an average token probability. This hides how certainty changes during reasoning. It also makes scores depend on length and filler text, so a wrong answer with one brief confidence dip can still look reliable.

The authors instead split each response into reasoning steps, average token probabilities within each step, and treat the result as a time series. They describe these series with Signal Temporal Logic (STL), whose robustness scores measure how well a trajectory matches a temporal pattern. A discriminative mining procedure finds STL formulas that separate correct from incorrect answers. A hypernetwork then adjusts each formula's numeric thresholds per question, while the logical structure stays fixed and readable.

Across several reasoning benchmarks and backbone models, the approach gives better ECE and Brier scores than logit-, consistency- and self-evaluation baselines. It needs only one sample per answer. This makes calibrated, auditable confidence cheap enough for abstention, candidate selection or early stopping.

Key points

  • Average ECE is 0.051, about 83% lower in relative terms than the AveLogit baseline. Inference takes about 0.55 s per example, a speedup of roughly 10× to 10,000× compared with self-evaluation, SAR and Self-Consistency.
  • The hypernetwork has 2.67M parameters (≈0.03% of the LLM) and trains on one GPU. InternalInspector, by comparison, needs 12–57M parameters and multi-GPU training.
  • Ablations show that failure-mode STL patterns transfer across tasks better than full pattern reuse. In-domain mining still works best, and per-question parameter adaptation helps most on CLadder and BBH.
  • Steps are segmented with a regex cascade keyed on "Step N:" markers, giving about 8.7 segments on average. Ten balanced STL templates gave the best trade-off across ECE, Brier score and AUROC.
  • Limitations: the method depends on step-structured outputs and is evaluated only on benchmarks with clear correctness labels. Open-ended generation and hallucination detection are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MannheimMannheim · Germany29 works · 8,347 citations🎓University of FloridaGainesville · United States33 works · 364 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Zhenjiang Mao, Anirudhh Venkat, Artem Bisliouk, Sindhura Kumbakonam Subramanian, Akshat Kothiyal, Saithej Singhu and Ivan Ruchkin. 2026. Confidence over Time: Confidence Calibration with Temporal Logic for Large Language Model Reasoning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.484

@inproceedings{mao2026confidence,
  title = {{Confidence over Time: Confidence Calibration with Temporal Logic for Large Language Model Reasoning}},
  author = {Zhenjiang Mao, Anirudhh Venkat, Artem Bisliouk, Sindhura Kumbakonam Subramanian, Akshat Kothiyal, Saithej Singhu and Ivan Ruchkin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.484},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.484}
}

References

52 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, and 1 others. 2021. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:… find
  2. Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhu Chen, and William Yang Wang. 2024. Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6416–6432.
  3. Artur d’Avila Garcez, Marco Gori, Luis C Lamb, Luciano Serafini, Michael Spranger, and Son N Tran. 2019. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. arXiv preprint arXiv:1905.06088. find
  4. Amos Azaria and Tom Mitchell. 2023. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, Singapore. Association for Computational Linguistics.
  5. Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. 2018. Conditional neural processes. In International conference on machine learning, pages 1704–1713. PMLR. find
  6. Daniel Beck, Lucia Specia, and Trevor Cohn. 2016. Exploring prediction uncertainty in machine translation quality estimation. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 208–218, Berlin, Germany. Association for Computational Linguistics. find
  7. Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, and Lifu Huang. 2024. InternalIn-spector i 2 : Robust confidence estimation in LLMs through internal states. In Findings of the Association for Computational Linguistics: EMNLP 2024, pa… find
  8. Glenn W. Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3. find
  9. Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295–302, Online. Association for Computational Linguistics. find
  10. Li Dong, Chris Quirk, and Mirella Lapata. 2018. Confidence modeling for neural semantic parsing. arXiv preprint arXiv:1805.04604. find
  11. Alexandre Donzé and Oded Maler. 2010. Robust satisfaction of temporal logic over real-valued signals. In International conference on formal modeling and analysis of timed systems, pages 92–106. Springer. find
  12. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Com… find
  13. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the Association for Computational Linguistics, 2:477–490.
  14. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  15. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR. find
  16. David Ha, Andrew Dai, and Quoc V Le. 2016. Hypernetworks. arXiv preprint arXiv:1609.09106. find
  17. Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2024. Decomposing uncertainty for large language models through input clarification ensembling. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find
  18. Susmit Jha, Ashish Tiwari, Sanjit A Seshia, Tuhin Sahai, and Natarajan Shankar. 2019. Telex: learning signal temporal logic from positive examples using tightness metric. Formal Methods in System Design, 54(3):364–387. find
  19. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962–977. find
  20. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, and 1 others. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. find
  21. Adharsh Kamath, Sishen Zhang, Calvin Xu, Shubham Ugare, Gagandeep Singh, and Sasa Misailovic. 2025. Enforcing temporal constraints for llm agents. arXiv preprint arXiv:2512.23738. — [arXiv:2512.23738](https://arxiv.org/abs/2512.23738) find
  22. Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30.
  23. Michael Kirchhof, Luca Füger, Adam Golinski, Eeshan Gunesh Dhekane, Arno Blaas, and Sinead Williamson. 2025. Self-reflective uncertainties: Do llms know their internal answer distribution? In ICML 2025 Workshop on Reliable and Responsible Foundation Models. find
  24. Zhaodan Kong, Austin Jones, Ana Medina Ayala, Ebru Aydin Gol, and Calin Belta. 2014. Temporal logic inference for classification and prediction from data. In Proceedings of the 17th international conference on Hybrid systems: computation and control, pages 273–282. find
  25. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations. find
  26. Jiawei Li, Akshayaa Magesh, and Venugopal V Veeravalli. 2025a. Principled detection of hallucinations in large language models via multiple testing. arXiv preprint arXiv:2508.18473. find
  27. Yinghao Li, Rushi Qiang, Lama Moukheiber, and Chao Zhang. 2025b. Language model uncertainty quantification with attention chain. In Second Conference on Language Modeling. find
  28. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research. find
  29. Alexis Linard and Jana Tumova. 2020. Active learning of signal temporal logic specifications. In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pages 779–785. IEEE. find
  30. Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. 2025. Uncertainty quantification and confidence calibration in large language models: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6107–6117.
  31. Zhenjiang Mao, Artem Bisliouk, Rohith Nama, and Ivan Ruchkin. 2025a. Temporalizing confidence: Evaluation of chain-of-thought reasoning with signal temporal logic. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 882–890, Vienna, Au… find
  32. Zhenjiang Mao, Carson Sobolewski, and Ivan Ruchkin. 2024. How safe am i given what i see? calibrated prediction of safety chances for image-controlled autonomy. In 6th Annual Learning for Dynamics & Control Conference, pages 1370–1387. PMLR. find
  33. Zhenjiang Mao, Mrinall Eashaan Umasudhan, and Ivan Ruchkin. 2025b. How safe will i be given what i saw? calibrated prediction of safety chances for image-controlled autonomy. arXiv preprint arXiv:2508.09346. — [arXiv:2508.09346](https://arxiv.org/abs/2508.09346) find
  34. Zhenjiang Mao and Anirudhh Venkat. 2026. Recurrent confidence chain: Temporal-aware uncertainty quantification in large language models. arXiv preprint arXiv:2601.13368. — [arXiv:2601.13368](https://arxiv.org/abs/2601.13368) find
  35. Daniele Nicoletti, Samuele Germiniani, and Graziano Pravadelli. 2024. Mining signal temporal logic specifications for hybrid systems. In 2024 Forum on Specification & Design Languages (FDL), pages 1–8. IEEE. find
  36. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3956–3965. PMLR.
  37. Amir Pnueli. 1977. The temporal logic of programs. In 18th annual symposium on foundations of computer science (sfcs 1977), pages 46–57. ieee. find
  38. Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Neuro-Logic decoding: (un)supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingu… find
  39. Gwenyth Portillo Wightman, Alexandra Delucia, and Mark Dredze. 2023. Strength in numbers: Estimating confidence of large language models by prompt agreement. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 326–362, Toronto, Canada. Association for… find
  40. Oded Maler and Dejan Nickovic. 2004. Monitoring temporal properties of continuous signals. In International symposium on formal techniques in real-time and fault-tolerant systems, pages 152–166. Springer. find
  41. Vasumathi Raman, Mehdi Maasoumy, and Alexandre Donzé. 2014. Model predictive control from signal temporal logic specifications: A case study. In Proceedings of the 4th ACM SIGBED international workshop on design, modeling, and evaluation of cyber-physical systems, pages 52–55. find
  42. Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations. find
  43. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 9004– 9017. find
  44. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118. find
  45. Carson Sobolewski, Zhenjiang Mao, Kshitij Maruti Ve-jre, and Ivan Ruchkin. 2025. Generalizable image repair for robust visual control. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9971–9978. IEEE. find
  46. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. Preprint, arXiv:2305.14975. find
  47. Ke Wang, Yangbin Shi, Jiayi Wang, Yuqi Zhang, Yu Zhao, and Xiaolin Zheng. 2021. Beyond glass-box features: Uncertainty quantification enhanced quality estimation for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4687–4698, Punta Cana, Dom… find
  48. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  49. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  50. Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations. find
  51. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. find
  52. Jingjin Yu, Ming C. Lin, Karen Leung, Nikos Aréchiga, and Marco Pavone. 2023. Backpropagation through signal temporal logic specifications: Infusing logical structure into gradient-based methods. Int. J. Rob. Res., 42(6):356–370. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mao-2026-confidence-over-time-confidence-calibration/