Elamur Research⌕
☰
home / papers / 2026 / Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant

Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant

Paper · 2026 · Joseph Matveyenko, James Liu, John David Parsons et al.

Summary

Manual qualitative coding is slow and hard to scale. Human coders tire and drift in how they interpret codes, and earlier computational tools could not match human nuance. The authors, from RAND Corporation, present Muse, an LLM-based assistant built with feedback from over 100 qualitative researchers. It is in production use, with 400+ users and 600+ projects.

Muse offers three tools. Generate codebook runs a multi-stage LLM pipeline that extracts themes, merges them, builds parent codes, validates them with embeddings and MMR, and refines them. Apply code labels excerpts against a single code. Distribute code assigns excerpts among several codes at once. Users can steer the system through code definitions and thumbs-up/down feedback, which becomes few-shot examples.

The authors benchmarked Muse on eleven public human-coded datasets, which they released, and tuned prompts, models and batching for accuracy versus cost and latency. For well-specified codes, Muse reached Cohen's κ ≈ 0.7 against human coders, a level usually considered substantial agreement. Error analysis showed that most disagreements come from ambiguous or inconsistently applied codebooks rather than model mistakes. The authors conclude that good codebook design matters as much as model choice.

Key points

  • Evaluation data: the curated collection of 11 datasets covers interviews, social media, surveys and domain corpora. It is published at github.com/jdmatv/muse-eval-dataset.
  • Codebook generation benchmark:
    • Uses the arXiv Categories dataset.
    • Each run samples 1,000 abstracts under three parent codes.
    • The resulting reference codebooks contain 22–52 codes.
  • Code application tuning:
    • Documents are split into ~80-word segments.
    • In a grid search with GPT-4o, the best setting reached κ = 0.545. It used batch size 5, 4 few-shot examples, a 1–10 discretized score and chain-of-thought.
  • Distribute code vs. apply code:
    • Distribute code beats apply code when there are fewer than about 6 sibling codes.
    • It needs roughly 55–65% of the time and 45–55% of the input tokens per excerpt.
    • Under researcher-specified constraints (e.g., one code per document), it gains about +0.18 κ on average.
  • Error analysis:
    • Covers the Reddit Topic Annotations and ATLAS.ti Sustainability datasets, comparing GPT-4.1, o3 and GPT-oss-120b with human coders.
    • Human deviations from the stated code definitions explain 66% of Reddit false positives, 74% of Reddit false negatives and 96% of Sustainability false positives.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

💡RAND CorporationSanta Monica · United States2 works · 38 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 worksEducation · 203 works

Cite

Cite as Joseph Matveyenko, James Liu, John David Parsons, Ryan Brown, Alina I. Palimaru, Vipul Gupta and Prateek Puri. 2026. Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.131

@inproceedings{matveyenko2026development,
  title = {{Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant}},
  author = {Joseph Matveyenko, James Liu, John David Parsons, Ryan Brown, Alina I. Palimaru, Vipul Gupta and Prateek Puri},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.131},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.131}
}

References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aly Abdelrazek, Yomna Eid, Eman Gawish, Walaa Medhat, and Ahmed Hassan. 2023. Topic modeling algorithms and applications: A survey. Information System, 112:306–4379. find
  2. ATLAS.ti. n.d. Sample projects. Mehmet Selman Baysan, Serkan Uysal, İrem İşlek, Çağla Çığ Karaman, and Tunga Güngör. 2025. Llm-as-a-judge: Automated evaluation of search query parsing using large language models. Frontiers in Big Data, 8:1611389. find
  3. Issam Bennis and Safwane Mouaffaq. 2025. Advancing ai-driven thematic analysis in qualitative research: A comparative study of nine generative models on cutaneous leishmaniasis data. BMC Medical Informatics and Decision Making, 25(1):124. find
  4. Sreyoshi Bhaduri, Satya Kapoor, Alex Gil, Anshul Mittal, and Rutu Mulkar. 2024. Reconciling methodological paradigms: Employing large language models as novice qualitative research assistants in talent management research. Preprint, arXiv:2408.11043. — [arXiv:2408.11043](https://arxiv.org/abs/2408.11043) find
  5. Nikhil Sanjay Borse, Ravishankar Chatta Subramaniam, and N. Sanjay Rebello. 2025. Investigation of the inter-rater reliability between large language models and human raters in qualitative analysis. Preprint, arXiv:2508.14764. — [arXiv:2508.14764](https://arxiv.org/abs/2508.14764) find
  6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  7. Rosanna Cole. 2024. Inter-rater reliability methods in qualitative case study research. Sociological Methods Research, 53(4):1944–1975. find
  8. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), volume 11, pages 512–515.
  9. Stefano De Paoli and Walter S. Mathis. 2025. Reflections on inductive thematic saturation as a potential metric for measuring the validity of an inductive thematic analysis with llms. Quality Quantity, 59:683– 709. find
  10. Norman K. Denzin and Yvonna S. Lincoln, editors. 2011. The SAGE Handbook of Qualitative Research, 4 edition. SAGE Publications, Inc., Thousand Oaks, CA. find
  11. Oliver Ferschke, Iryna Gurevych, and Yevgen Chebotar. 2012. Behind the article: Recognizing dialog acts in wikipedia talk pages. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 777–786, Avignon, France. Association for Computation…
  12. Uwe Flick. 2014. An Introduction to Qualitative Research, 5 edition. SAGE Publications Ltd, Thousand Oaks, CA. find
  13. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. find
  14. Andy Hickner. 2022. Data for “how do search systems impact systematic searching? a qualitative study”. QDR Main Collection. Version 1 (V1). find
  15. Ariba Khan, Stephen Casper, and Dylan Hadfield-Menell. 2025. Randomness, not representation: The unreliability of evaluating cultural alignment in llms. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 2151– 2165, New York, NY, USA. Association for Compu… find
  16. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. Preprint, arXiv:2309.06180.
  17. Joonatan Laato, Matti Mäntymäki, Bastian Kordyaka, and Samuli Laato. 2025. Automating qualitative data analysis with chain-of-thought reasoning models: A study with the gioia method. In AMCIS 2025: Proceedings of the 31st Americas Conference on Information Systems, Montréal, Canada. AIS Electronic L… find
  18. Michelle S. Lam, Janice Teoh, James A. Landay, Jeffrey Heer, and Michael S. Bernstein. 2024. Concept induction: Analyzing unstructured text with high-level concepts using lloom. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, page Article 766, New York, NY, USA. Asso… find
  19. David D. Lewis. 1987. Reuters-21578 text categorization collection. UCI Machine Learning Repository. find
  20. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: A comprehensive survey on llm-based evaluation methods. Preprint, arXiv:2412.05579.
  21. Sara Mannheimer. 2023. Interviews regarding data curation for qualitative data reuse and big social research. QDR Main Collection, Version V1. find
  22. David L. Morgan. 2023. Exploring the use of artificial intelligence for qualitative data analysis: The case of chatgpt. International Journal of Qualitative Methods, 22:16094069231211248. find
  23. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. Preprint, arXiv:2112.00…
  24. Jiaxing Qiu. 2025. Topic annotations on reddit posts from eating disorders and dieting forums by human and llms. University of Virginia Dataverse, Version V3. find
  25. Johnny Saldaña. 2012. The Coding Manual for Qualitative Researchers, 2 edition. SAGE Publications Ltd, London, UK. find
  26. Miriam Schirmer, Isaac Misael Olguín Nolasco, Edoardo Mosca, Shanshan Xu, and Jürgen Pfeffer. 2023. Uncovering trauma in genocide tribunals: An nlp approach using the genocide transcript corpus. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, pages 257–… find
  27. Tim Schopf, Alexander Blatzheim, Nektarios Machner, and Florian Matthes. 2024. Efficient few-shot learning for multi-label classification of scientific documents with many classes. In Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024), pages 186–1… find
  28. Tim Schopf, Daniel Braun, and Florian Matthes. 2023. Evaluating unsupervised text classification: Zero-shot and similarity-based approaches. In Proceedings of the 6th International Conference on Natural Language Processing and Information Retrieval (NLPIR), pages 6–15, New York, NY, USA. Association… find
  29. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2025. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM 2 ), pages 404–430, Vienna… find
  30. Qile Wang, Moath Erqsous, Kenneth E. Barner, and Matthew Louis Mauriello. 2025. Lata: A pilot study on llm-assisted thematic analysis of online social network data generation experiences. Association for Computing Machinery, 9(2):1–28. find
  31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS 2022), pages 24824–24837.
  32. Yiming Yang and Xin Liu. 1999. A re-examination of text categorization methods. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 42–49, New York, NY, USA. Association for Computing Machinery. find
  33. Elena Zotova, Rodrigo Agerri, Manuel Nuñez, and German Rigau. 2020. Multilingual stance detection in tweets: The catalonia independence corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1368–1375, Marseille, France. European Language Resources Association.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/matveyenko-2026-development-benchmarking-blended-human-ai/