Elamur Research⌕
☰
home / papers / 2026 / Can Large Language Models Replace Statistical Software?

Can Large Language Models Replace Statistical Software?

Paper · 2026 · Yves Staudt

Summary

The paper asks whether current LLMs could replace tools such as SPSS or R when people with limited statistical training need to run a hypothesis test. This matters most in medicine, where statistical literacy is often weak. The author built one benchmark task from survey data by Crottogini et al. (2026) on kick-scooter riding and knee pain among Swiss schoolchildren. The task requires a χ²-test of independence on two binary variables. Seven models (three proprietary, two open-weight, two compact) each received the same zero-shot chain-of-thought prompt, with the data embedded and a fixed four-step answer format. The author graded the answers by hand on eleven criteria using an ordinal A–G scale and compared them with output from conventional statistical software. All models chose the correct test, but none fully reproduced the reference analysis, mainly because none applied Yates' continuity correction. Gemini, Claude Opus and ChatGPT clearly led, while the smaller models struggled to reconstruct the data. The conclusion is that LLMs work as helpers for choosing and interpreting tests, but the calculations should stay in statistical software.

Key points

  • Models tested: The three proprietary models were Claude Opus 4.6, ChatGPT 5.4 Thinking and Gemini 3.1 Pro Preview, and the other four were Llama 4 Scout, Qwen 3.5 9b, Phi 4 and Apertus 8b. The open-weight and compact models ran locally through Ollama; the proprietary ones were used via their web interfaces.
  • Dataset: 134 complete responses from pupils in grades 3–6 (ages 8–12) in Chur, Switzerland, of whom 51.5% were female.
  • Grading: answers were scored on criteria including test choice, hypotheses, sample size, observed and expected frequencies, assumption checks, p-value, significance level and contextual interpretation. Grades ran from A (correct and correctly applied) to G (not addressed).
  • Main gap: no model applied Yates' correction for a 2 × 2 table with one degree of freedom. Gemini gave the most structured and transparent answers, and ChatGPT gave fewer calculation details.
  • Limitations: the study covers a single use case, a single prompt and a single evaluator. Planned future work includes t-tests, regression, tool-augmented prompting and studies of how LLMs affect human users' analyses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yves Staudt. 2026. Can Large Language Models Replace Statistical Software?. Proceedings of the 11th Edition of the Swiss Text Analytics Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{staudt2026can,
  title = {{Can Large Language Models Replace Statistical Software?}},
  author = {Yves Staudt},
  booktitle = {Proceedings of the 11th Edition of the Swiss Text Analytics Conference},
  year = {2026},
  url = {https://aclanthology.org/2026.swisstext-1.19/}
}

References

20 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… — Phi-4 Technical Report
  2. OpenAI. 2026. Introducing GPT-5.4. https://openai.com/index/introducing-gpt-5-4/. Accessed: 2026-05-04. ↗
  3. C. O’Dushlaine. 2019. Hypothesis testing and confidence intervals, pages 523–626. find
  4. Luigi De Angelis, Francesco Baglivo, Guglielmo Arzilli, Gaetano Pierpaolo Privitera, Paolo Ferragina, Alberto Eugenio Tozzi, and Caterina Rizzo. 2023. Chatgpt and the rise of large language models: the new ai-driven infodemic threat in public health. Frontiers in Public Health, 11. find
  5. Anthropic. 2026. Introducing Claude Opus 4.6. https://www.anthropic.com/news/claude-opus-4-6. Accessed: 2026-05-04. ↗
  6. Armando Crottogini, Bianca Schenk, and Yves Staudt. 2026. Rolling towards resilience: The impact of scooter riding on pediatric knee health. find
  7. M Fay and E Brittain. 2022. Statistical hypothesis testing in context. pages –. find
  8. Sharad Patel and Adam Green. 2025. Death by p-value: the overreliance on p-values in critical care research. Critical Care, 29:73. find
  9. Qwen Team. 2026. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id=qwen3. 5. Accessed: 2026-05-04. ↗
  10. Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2025. A systematic survey of prompt engineering in large language models: Techniques and applications.
  11. Swiss AI Initiative. 2025. Apertus: Democratizing open and compliant LLMs for global language environments. arXiv preprint arXiv:2509.14233. Developed by EPFL, ETH Zurich, and CSCS.
  12. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature Medicine, 29:1930–1940. find
  13. Larry Gonick and Woollcott Smith. 2005. The Cartoon Guide to Statistics. HarperCollins. find
  14. Google DeepMind. 2026. Gemini 3.1 Pro model card. https://deepmind.google/models/ Accessed: model-cards/gemini-3-1-pro/. 2026-05-04. ↗
  15. Mirjam Annina Jenny, Niklas Keller, and Gerd Gigerenzer. 2018. Assessing minimal medical statistical literacy using the quick risk test: a prospective observational study in germany. BMJ Open, 8:e020847. find
  16. Camille Lakhlifi, François-Xavier Lejeune, Marion Rouault, Mehdi Khamassi, and Benjamin Rohaut. 2023. Illusion of knowledge in statistics among clinicians: evaluating the alignment between objective accuracy and subjective confidence, an online survey. Cognitive Research: Principles and Implications… find
  17. Siyu Liu. 2025. Modern developments in hypothesis testing with emphasis on computational techniques. Theoretical and Natural Science, pages –. find
  18. D Mark, Kerry Lee, and F Harrell. 2016. Understanding the role of p values and hypothesis tests in clinical research. JAMA cardiology, 1 9:1048–1054. find
  19. D Turner, H Deng, and T Houle. 2020. Statistical hypothesis testing: Overview and application. Headache: The Journal of Head and Face Pain, 60:– . find
  20. Sirui Zhang. 2025. Analysis and optimization of the applicability of hypothesis testing methods. Advances in Operation Research and Production Management, pages –. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/staudt-2026-can-large-language-models-replace/