Elamur Research⌕
☰
home / papers / 2026 / CAST: Achieving Stable LLM-based Text Analysis for Data Analytics

CAST: Achieving Stable LLM-based Text Analysis for Data Analytics

Paper · 2026 · Jinxiang Xie, Zihao Li, Wei He et al.

Summary

When LLMs process free-text columns in tables, their outputs become keys for filtering, grouping and aggregation. If repeated runs on the same input yield different labels or themes, downstream analytics stop being reproducible. The authors frame this setting as Text Analysis for Data Analysis (TADA), built on two operators: summarization, which condenses a column into a few themes, and tagging, which labels each row. They trace instability to diffuse, unconstrained reasoning paths. Their framework, CAST, narrows these paths in two ways. Algorithmic Prompting encodes a procedural workflow in the prompt. Thinking-before-Speaking makes the model commit to intermediate states, such as domain, topic schema and clusters, before giving its answer. The authors also propose two stability metrics, CAST-S for bulleted summaries and CAST-T for tags, and check them against human judgments. On public benchmarks with several LLM backbones, CAST is the most stable of all compared methods, and quality holds steady or improves. Because it needs only one call, it costs far less than sampling-based alternatives.

Key points

  • Across backbones (GPT-5.2, Gemini-3-Flash, DeepSeek-V3.2), CAST raises the Stability Score by as much as 16.2% over baselines including Few-shot CoT, Tree of Thoughts and Self-Consistency.
  • The two metrics work differently:
    • CAST-S pairs an LLM-based semantic score with a positional score built on Kendall's Tau, so it is sensitive to bullet order.
    • CAST-T clusters equivalent tags across runs and measures convergence toward one dominant meaning.
  • Self-Consistency proved the least stable approach and also requires 3× or more compute.
  • CAST's overhead is small. Latency was 6.60s versus 6.06s on Gemini-3-Flash, and each call uses only several dozen additional tokens.
  • In ablations, combining AP and TbS generally beats either part alone. In summarization, summary recall rose to 0.879 versus 0.854 for Zero-shot CoT, with precision perfect for both.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksGeneral Social Sciences · 254 works

Cite

Cite as Jinxiang Xie, Zihao Li, Wei He, Rui Ding, Shi Han and Dongmei Zhang. 2026. CAST: Achieving Stable LLM-based Text Analysis for Data Analytics. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.113

@inproceedings{xie2026cast,
  title = {{CAST: Achieving Stable LLM-based Text Analysis for Data Analytics}},
  author = {Jinxiang Xie, Zihao Li, Wei He, Rui Ding, Shi Han and Dongmei Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.113},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.113}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. 2024. LLM Stability: A detailed analysis with some surprises. Preprint, arXiv:2408.04667. — [arXiv:2408.04667](https://arxiv.org/abs/2408.04667) find
  2. Ansar Aynetdinov and Alan Akbik. 2024. Semscore: Automated evaluation of instruction-tuned llms based on semantic textual similarity. Preprint, arXiv:2401.17072. — [arXiv:2401.17072](https://arxiv.org/abs/2401.17072) find
  3. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resour…
  4. Suiyao Chen, Jing Wu, Naira Hovakimyan, and Handong Yao. 2023. Recontab: Regularized contrastive representation learning for tabular data. Preprint, arXiv:2310.18541. — [arXiv:2310.18541](https://arxiv.org/abs/2310.18541) find
  5. Wuxinlin Cheng, Yupeng Cao, Jinwen Wu, Koduvayur Subbalakshmi, Tian Han, and Zhuo Feng. 2025. SALMAN: Stability Analysis of Language Models Through the Maps Between Graph-based Manifolds. Preprint, arXiv:2508.18306. — [arXiv:2508.18306](https://arxiv.org/abs/2508.18306) find
  6. Emma Croxford, Yanjun Gao, Nicholas Pellegrino, Karen Wong, Graham Wills, Elliot First, Frank Liao, Cherodeep Goswami, Brian Patterson, and Majid Afshar. 2025. Current and future state of evaluation of large language models for medical summarization tasks. npj Health Systems, 2(1):6. find
  7. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summarization Evaluation. Preprint, arXiv:2007.12626. find
  8. Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2022. Massive: A 1m-example multilingual natural language… find
  9. Vanessa Glenny, Jonathan Tuke, Nigel Bean, and Lewis Mitchell. 2019. A framework for streamlined statistical prediction using topic models. Preprint, arXiv:1904.06941. find
  10. Zhouhong Gu, Haoning Ye, Xingzhou Chen, Zeyang Zhou, Hongwei Feng, and Yanghua Xiao. 2025. StrucText-Eval: Evaluating Large Language Model’s Reasoning Ability in Structure-Rich Text. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p… find
  11. Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2024. Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling. Preprint, arXiv:2311.08718. — [arXiv:2311.08718](https://arxiv.org/abs/2311.08718) find
  12. Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Structgpt: A general framework for large language model to reason over structured data. Preprint, arXiv:2305.09645. find
  13. Mirella Lapata. 2006. Automatic Evaluation of Information Ordering: Kendall’s Tau. Computational Linguistics, 32(4):471–484.
  14. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  15. Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023. Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation. In Proceedings of the 61st Annual Meeting of the…
  16. Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, and Lluís Màrquez. 2024. Factual confidence of LLMs: on reliability and robustness of current estimators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). A… find
  17. James Mutinda, Waweru Mwangi, and George Okeyo. 2023. Sentiment Analysis of Text Reviews Using Lexicon-Enhanced Bert Embedding (LeBERT) Model with Convolutional Neural Network. Applied Sciences, 13(3):1445. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  19. Du Phan, Matthew D. Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A. Saurous. 2023. Training Chain-of-Thought via Latent-Variable Inference. Preprint, arXiv:2312.02179. — [arXiv:2312.02179](https://arxiv.org/abs/2312.02179) find
  20. Steve Rathje, Dan-Mircea Mirea, Ilia Sucholutsky, Raja Marjieh, Claire E. Robertson, and Jay J. Van Bavel. 2024. GPT is an effective tool for multilingual psychological text analysis. Proceedings of the National Academy of Sciences, 121(34):e2308950121. find
  21. Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Ruoxi Jia, and Ming Jin. 2024. Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models. Preprint, arXiv:2308.10379. find
  22. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Preprint, arXiv:2402.03300.
  23. Wangchunshu Zhou, Jinyi Hu, Hanlin Zhang, Xiaodan Liang, Maosong Sun, Chenyan Xiong, and Jian Tang. 2022b. Towards Interpretable Natural Language Understanding with Explanations as Latent Variables. Preprint, arXiv:2011.05268. — [arXiv:2011.05268](https://arxiv.org/abs/2011.05268) find
  24. Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. FineSurE: Fine-grained Summarization Evaluation using LLMs. Preprint, arXiv:2407.00908.
  25. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. Preprint, arXiv:2203.11171.
  26. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Preprint, arXiv:2305.10601. find
  27. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. 2024. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. Preprint, arXiv:2410.02736.
  28. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. 2024. Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking. Preprint, arXiv:2403.09629.
  29. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Preprint, arXiv:1904.09675. find
  30. Hongbo Zhao, Yixin Sheng, Yu Cai, Muyun Li, Tao Wu, and Yang Liu. 2024. On the reliability of large language models to misinformed and unconventional prompts. ACM Computing Surveys, 57. find
  31. Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. 2022a. Teaching Algorithmic Reasoning via In-context Learning. Preprint, arXiv:2211.09066.
  32. Junkai Zhou, Liang Pang, Huawei Shen, and Xueqi Cheng. 2024. Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3925–3951, Mexico City, Mexico. Association for Computat… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2026-cast-achieving-stable-llm-based/