Elamur Research⌕
☰
home / papers / 2026 / Questionnaire Meets LLM: A Benchmark and Empirical Study of Structural Skills for Understanding Questions and Responses

Questionnaire Meets LLM: A Benchmark and Empirical Study of Structural Skills for Understanding Questions and Responses

Summary

Survey data is a grid of respondents by questions. It is hard for LLMs because cells mix types such as Likert scores, coded multiple-choice options and free text, and answer codes must be decoded through schema metadata. Practitioners also have no evidence on how to turn such data into text for a model. The authors present QASU (Questionnaire Analysis and Structural Understanding), an open-source benchmark of six micro-tasks covering retrieval and aggregation/reasoning skills, including answer lookup, reverse lookup, respondent counting and multi-hop inference. It crosses these tasks with six serialization formats and several prompt configurations. It also tests prompt-component ablations and self-augmented prompting, where the model first describes the data's structure before answering. They evaluate five LLMs. Formatting matters: the best format beats weaker ones by up to 9 points. Proprietary models clearly outperform open-weight ones. Self-augmentation helps only some models. The work gives practical, evidence-based guidance for using LLMs in medical, social-science and business survey analysis.

Key points

  • Formats: The six serializations compared were Turtle (TTL), plain text, Markdown, JSON, XML and HTML. On answer lookup, HTML scored up to 9 percentage points higher than TTL.
  • Models: GPT-5-mini, Gemini-2.5-Flash, Qwen3-32B, Llama3-70B and Amazon Nova Lite. Proprietary models led open-weight ones by 10–30 points. Some open-weight setups fell to about 45% accuracy on respondent counting.
  • Self-augmented prompting: Three variants were tested: format explanation, critical values and structural information. They gave proprietary models gains of up to +20 points on some datasets, but open-weight models did not benefit.
  • Prompt ablations: Five variants each changed one element of an XML-sectioned base prompt:
    • removing the role section,
    • removing the format explanation,
    • removing the one-shot example,
    • removing the partition tags,
    • moving the questionnaire section after the task.
  • Long questionnaires: Respondents are randomly sampled so the input fits the context window. The benchmark is released at github.com/ReML-AI/QASU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/438xkvmy2xd9

@inproceedings{nguyen2026questionnaire,
  title = {{Questionnaire Meets LLM: A Benchmark and Empirical Study of Structural Skills for Understanding Questions and Responses}},
  author = {Nguyen, Duc-Hai and Nanjappan, Vijayakumar and O'Sullivan, Barry and Nguyen, Hoang D.},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/438xkvmy2xd9},
  url = {https://doi.org/10.63317/438xkvmy2xd9}
}

References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nuria Bel, Sergio Espeja, and Montserrat Marimon. 2016. Abstracts in multiple languages: a multilingual and cross-lingual resource. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3815–3820. find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33, pages 1877–1901. find
  3. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Y. Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations. find our card
  4. Yuyan Chen, Qingpei Guo, Shuangjie You, and Zhixu Li. 2025. Medtranstab: Advancing medical cross-table tabular data generation. In Proceedings of the 18th ACM International Conference on Web Search and Data Mining (WSDM ’25), New York, NY, USA. ACM. find
  5. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural… find our card
  6. Mick P. Couper. 2000. Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64(4):464–494. find
  7. Don A. Dillman, Jolene D. Smyth, and Leah Melani Christian. 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 4th edition. John Wiley & Sons. find
  8. Josep Domingo-Ferrer and Vicenç Torra. 2001. Rank swapping for partial confidentiality. In Proceedings of the First International Conference on Privacy in Statistical Databases, pages 177–188. Springer. find
  9. Jörg Drechsler, Stefan Bender, and Susanne Rässler. 2011. Empirical evaluation of cross-sectional and longitudinal record linkage methods. Journal of Official Statistics, 27(3):403–421. find
  10. Floyd J. Fowler. 2013. Survey Research Methods, 5th edition. SAGE Publications. find
  11. Lynda T Goodfellow. 2023. An overview of survey research. Respiratory Care, 68(9):1309–1313. find
  12. Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786–2791. find
  13. MIMBCD-UI Group. 2022. Sus-uta7: Usability scores for wearable-technology studies. https://github.com/MIMBCD-UI/dataset-uta7-sus. ↗
  14. Robert M. Groves, Floyd J. Fowler, Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau. 2009. Survey Methodology, 2 edition. John Wiley and Sons. find
  15. Hyukjun Gweon and Matthias Schonlau. 2024. Automated classification for open-ended questions with bert. Journal of Survey Statistics and Methodology, 12(2):493–504. find
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations. find our card
  17. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333. ACL. find our card
  18. Nancy Ide and James Pustejovsky. 2017. Representing linguistic data: The text encoding initiative. In Nancy Ide and James Pustejovsky, editors, Handbook of Linguistic Annotation, pages 1289–1316. Springer. find
  19. HL7 International. 2019. Fhir structured data capture: Questionnaire resource. https://hl7.org/fhir/questionnaire.html. ↗
  20. Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin Zhao, and Ji-Rong Wen. 2023. Structgpt: A general framework for large language model to reason over structured data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9237– 9251, Singapore. Association… find
  21. Takeshi Kojima, Shixiang Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35:22199–22213. find our card
  22. Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022. Tapex: Table pre-training via learning a neural sql executor. In International Conference on Learning Representations. find our card
  23. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157–165, Dublin, Ireland. Association for Computational Linguistics. find
  24. Stack Overflow. 2022. Developer survey 2022. https://survey.stackoverflow.co/2022/. ↗
  25. Prasad Prasad. 2020. Healthcare dataset on kaggle. https://www.kaggle.com/datasets/prasad22/healthcare-dataset. ↗
  26. Lois Rink, Job Meijdam, and David Graus. 2024. Aspect-based sentiment analysis for open-ended hr survey responses. In Proceedings of the First Workshop on NLP for Human Resources, pages 16–26, Malta. Association for Computational Linguistics. find
  27. Damiano Scandolari, Gloria Re Calegari, Irene Celino, and Mario Scrocca. 2021. The survey ontology (v1.1). Technical report, Cefriel. Available at https://cefriel.github.io/survey-ontology/. ↗
  28. Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901. Association for Computational Linguis… find
  29. Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM ’24), pages 645–654, New Yo… find
  30. Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. 2021. Understanding unintended memorization in federated learning. In Proceedings of the Third Workshop on Privacy in Natural Language Processing, pages 1–10. Association for Computational Linguistics. find
  31. Stephanie Topp and Becky Thomas. 2015. Self-reported mental health survey. https://data.mendeley.com/datasets/bytb22nf7m/1. ↗
  32. Mary Vardigan, Pascal Heus, and Wendy Thomas. 2008. Data documentation initiative: Toward a standard for the social sciences. International Journal of Digital Curation, 3(1):107–134. find
  33. Carina Nina Vorisek, Sophie Anne Inès Klopfenstein, Matthias Löbe, Carsten Oliver Schmidt, Paula Josephine Mayer, Martin Golebiewski, and Sylvia Thun. 2024. Towards an interoperability landscape for a national research data infrastructure for personal health data. Scientific Data, 11(1):772. find
  34. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain-of-thought reasoning in language models. In International Conference on Learning Representations. find our card
  35. Stanley L. Warner. 1965. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association, 60(309):63–69. find
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35, pages 24824–24837. find our card
  37. Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. Tabert: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413–8426. ACL. find our card
  38. Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar-augmented pre-training for table semantic parsing. In International Conference on Learning Representations. find
  39. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql. In Proceedings of the 2018 Confe… find our card
  40. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. find
  41. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Representations. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2026-questionnaire-meets-llm-benchmark-empirical/