Elamur Research⌕
☰
home / papers / 2026 / CentaurTA: A Self-Improving Human-Agents Collaboration Framework for Thematic Analysis

CentaurTA: A Self-Improving Human-Agents Collaboration Framework for Thematic Analysis

Paper · 2026 · Lei Wang, Min Huang and Eduard Dragut

Summary

Thematic analysis (TA) of qualitative data is slow and hard to reproduce. Existing LLM tools tend to fall into two camps. Fully automated pipelines sacrifice rigor. Human-in-the-loop systems use expert corrections only once, so experts end up fixing the same mistakes again and again. CentaurTA addresses this by treating expert feedback as an ongoing training signal. TA is split into open coding (code, quote, supporting sentence indices) and theme construction. An Actor agent generates outputs, a Critic agent judges them, and validated human labels and rationales drive a prompt optimizer that rewrites both agents' instructions. No model weights are updated. The authors also propose a rubric-based protocol that scores codes and themes against methodological constraints, offering finer signals than generic LLM-as-Judge scores. Across three datasets and several LLM families, agreement with experts on coding rose above 90%. Most of the gain came from relatively little feedback. This suggests that scalable, transparent LLM-assisted qualitative research is possible while humans keep interpretive authority.

Key points

  • Datasets:
    • USRS: 12 Chinese self-regulated-learning reflections, about 2,143 words each.
    • ASP: 15 English narratives about autistic job seekers.
    • Dreaddit: 214 short English stress posts, half of the test set.
  • Documents are handled in 10-sentence batches. Feedback comes in two stages: GPT-5.2 drafts provisional labels, then domain experts finalize them. Only the expert-validated labels supervise the optimizer.
  • Open-coding batch accuracy exceeded 90% after roughly 20 interaction rounds.
  • Rubric trajectories show that peak generalizable performance arrives with feedback on about 10% of a document. Rubric-based early stopping helps prevent overfitting.
  • The learned Actor/Critic prompts ("principles") transfer to other AI-assisted workflows. Code and data are available at github.com/Tom-Owl/CentaurTA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Temple UniversityPhiladelphia · United States10 works · 24,319 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 worksEducation · 203 works

Cite

Cite as Lei Wang, Min Huang and Eduard Dragut. 2026. CentaurTA: A Self-Improving Human-Agents Collaboration Framework for Thematic Analysis. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.778

@inproceedings{wang2026centaurta,
  title = {{CentaurTA: A Self-Improving Human-Agents Collaboration Framework for Thematic Analysis}},
  author = {Lei Wang, Min Huang and Eduard Dragut},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.778},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.778}
}

References

66 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdullah Aljebreen, Weiyi Meng, and Eduard C. Dragut. 2024. Analysis and detection of "pink slime" websites in social media posts. In Proceedings of the ACM Web Conference 2024, WWW ’24, page 2572–2581, New York, NY, USA. Association for Computing Machinery. find
  2. Akari Asai, Jiashu He, Rui Shao, and 1 others. 2026. Synthesizing scientific literature with retrieval-augmented language models. Nature, 650:857–863. find
  3. ATLAS.ti. 2025. Ai coding powered by OpenAI. https://atlasti.com/ ai-coding-powered-by-openai. Accessed: 2025-12-24. ↗
  4. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2):77–101. find
  5. Eduard Dragut, Yunyao Li, Lucian Popa, and Slobodan Vucetic. 2021. Data science with human in the loop. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, page 4123–4124, New York, NY, USA. Association for Computing Machinery. find
  6. Margaret Drouhard, Nan-Chen Chen, Jina Suh, Rafal Kocielnik, Vanessa Pena-Araya, Keting Cen, Xiangyi Zheng, and Cecilia R Aragon. 2017. Aeonium: Visual analytics to support collaborative qualitative coding. In 2017 IEEE Pacific Visualization Symposium (PacificVis), pages 220–229. IEEE. find
  7. Mingzhe Du, Luu Anh Tuan, Yue Liu, Yuhao Qing, Dong Huang, Xinyi He, Qian Liu, Zejun Ma, and See-kiong Ng. 2025. Afterburner: Reinforcement learning facilitates self-improving code efficiency optimization. arXiv preprint arXiv:2505.23387. — [arXiv:2505.23387](https://arxiv.org/abs/2505.23387) find
  8. Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. 2024a. Complex claim verification with evidence retrieved in the wild. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: L…
  9. Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui…
  10. Xiusi Chen, Hongzhi Wen, Sreyashi Nag, Chen Luo, Qingyu Yin, Ruirui Li, Zheng Li, and Wei Wang. 2024b. IterAlign: Iterative constitutional alignment of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  11. Yu Feng, Phu Mon Htut, Zheng Qi, Wei Xiao, Manuel Mager, Nikolaos Pappas, Kishaloy Halder, Yang Li, Yassine Benajiba, and Dan Roth. 2025a. Rethinking LLM uncertainty: A multi-agent approach to estimating black-box model uncertainty. In Findings of the Association for Computational Linguistics: EMNLP… find
  12. Zhijia Chen, Lihong He, Arjun Mukherjee, and Eduard Dragut. 2024c. Comquest: Large scale user comment crawling and integration. In Companion of the 2024 International Conference on Management of Data, SIGMOD ’24, page 432–435, New York, NY, USA. Association for Computing Machinery. find
  13. Zhijia Chen, Weiyi Meng, and Eduard Dragut. 2022. Web record extraction with invariants. Proc. VLDB Endow., 16(4):959–972. find
  14. Matheus de Morais Leça, Lucas Valença, Reydne Santos, and Ronnie de Souza Santos. 2025. Applications and implications of large language models in qualitative analysis: A new frontier for empirical software engineering. In Proceedings of the 2025 IEEE/ACM International Workshop on Methodological Issu… find
  15. Shijian Deng, Kai Wang, Tianyu Yang, Harsh Singh, and Yapeng Tian. 2025. Self-improvement in multimodal large language models: A survey. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 1987–2006, Suzhou, China. Association for Computational Linguistics.
  16. Yu Feng, Nathaniel Weir, Kaj Bostrom, Sam Bayless, Darion Cassel, Sapana Chaudhary, Benjamin Kiesl-Reiter, and Huzefa Rangwala. 2025b. Vericot: Neuro-symbolic chain-of-thought validation via logical consistency checks. Preprint, arXiv:2511.04662. — [arXiv:2511.04662](https://arxiv.org/abs/2511.04662) find
  17. Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Tagyoung Chung, Mohit Bansal, and Nanyun Peng. 2024. LLM self-correction with De-CRIM: Decompose, critique, and refine for enhanced following of instructions with multiple constraints.… find
  18. Jie Gao, Kenny Tsu Wei Choo, Junming Cao, Roy Ka-Wei Lee, and Simon Perrault. 2023. Coaicoder: Examining the effectiveness of ai-assisted human-to-human collaboration in qualitative analysis. ACM Trans. Comput.-Hum. Interact., 31(1). find
  19. Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Perrault. 2024. Collabcoder: A lower-barrier, rigorous workflow for inductive collaborative qualitative analysis with large language models. In Proceedings of the 2024 CHI Conference on Human Factors in… find
  20. Jie Gao, Zhiyao Shu, and Shun Yi Yeo. 2025. Mindcoder: Automated and controllable reasoning chain in qualitative analysis. Preprint, arXiv:2501.00775. — [arXiv:2501.00775](https://arxiv.org/abs/2501.00775) find
  21. Elizabeth Garrison, Stephen MacNeil, Donald A. Hantula, Michael West, Eduard Dragut, Matt Tincani, and Slobodan Vucetic. 2025. Exploring the challenges and assistive technology for autistic job seekers across employment pathways. Research in Developmental Disabilities, 167:105155. find
  22. Feijiang Han, Zelo ng Wang, Bowen Wang, Xinxin Liu, Skyler Cheung, Delip Rao, Chris Callison-Burch, and Lyle Ungar. 2026. Latex2layout: High-fidelity, scalable document layout annotation pipeline for layout detection. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37):30907–30915. find
  23. Min Huang. 2024. Beyond One-Dimension: A Study on Self-Regulated Learning and Its Typological Optimization of Chinese Undergraduates with National Scholarship. Doctoral dissertation, Huazhong University of Science and Technology. find
  24. Haydn Thomas Jones, Natalie Maus, Josh Magnus Ludan, Maggie Ziyu Huan, Jiaming Liang, Marcelo Der Torossian Torres, Jiatao Liang, Zachary Ives, Yoseph Barash, Cesar de la Fuente-Nunez, Jacob R. Gardner, and Mark Yatskar. 2025. A dataset for distilling knowledge priors from literature for therapeutic… find
  25. Fangping Lan, Abdullah Aljebreen, and Eduard Dragut. 2025. UniT: One document, many revisions, too many edit intention taxonomies. In Findings of the Association for Computational Linguistics: ACL 2025, pages 23005–23024, Vienna, Austria. Association for Computational Linguistics. find
  26. V Vien Lee, Stephanie CC van der Lubbe, Lay Hoon Goh, and Jose Maria Valderas. 2024. Harnessing chatgpt for thematic analysis: Are we ready? Journal of Medical Internet Research, 26:e54974. find
  27. Sunzhu Li, Jiale Zhao, Miteto Wei, Huimin Ren, Yang Zhou, Jingwen Yang, Shunyu Liu, Kaike Zhang, and Wei Chen. 2026. Rubrichub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. Preprint, arXiv:2601.08430.
  28. Wenbo Li, Longyin Wen, Xiao Bian, and Siwei Lyu. 2019. Evolvement constrained adversarial learning for video style transfer. In Computer Vision – ACCV 2018, pages 232–248, Cham. Springer International Publishing. find
  29. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. Preprint, arXiv:2406.04770.
  30. Chaitanya Malaviya, Subin Lee, Dan Roth, and Mark Yatskar. 2024. What if you said that differently?: How explanation formats affect human feedback efficacy and user perception. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  31. Megh Marathe and Kentaro Toyama. 2018. Semi-automated coding for qualitative research: A user-centered inquiry and initial prototypes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, page 1–12, New York, NY, USA. Association for Computing Machinery. find
  32. Sneha Mondal, Ashish Agrawal, Ritika, Preethi Jyothi, and Aravindan Raghuveer. 2024. DIMSIM: Distilled multilingual critics for Indic text simplification. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16093–16109, Bangkok, Thailand. Association for Computational Lingu… find
  33. David L Morgan. 2025. Query-based analysis: A strategy for analyzing qualitative data using chatgpt. Qualitative Health Research, page 10497323251321712. find
  34. Sid Nayak, Adelmo Morrison Orozco, Marina Have, Jackson Zhang, Vittal Thirumalai, Darren Chen, Aditya Kapoor, Eric Robinson, Karthik Gopalakrishnan, James Harrison, and 1 others. 2024. Long-horizon planning for multi-agent robots in partially observable environments. Advances in Neural Information P… find
  35. Lorelli S. Nowell, Jill M. Norris, Deborah E. White, and Nancy J. Moules. 2017. Thematic analysis: Striving to meet the trustworthiness criteria. International Journal of Qualitative Methods, 16(1):1–13. find
  36. Sudipta Paul, Niluthpol Chowdhury Mithun, and Amit K. Roy-Chowdhury. 2022. Text-based temporal localization of novel events. In Computer Vision – ECCV 2022, pages 567–587, Cham. Springer Nature Switzerland. find
  37. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. 2025. Generalizing verifiable instruction following. Preprint, arXiv:2507.02833.
  38. Tingrui Qiao, Caroline Walker, Chris Cunningham, and Yun Sing Koh. 2025. Thematic-lm: A llm-based multi-agent system for large-scale thematic analysis. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 649–658, New York, NY, USA. Association for Computing Machinery. find
  39. Delip Rao and Chris Callison-Burch. 2026a. Autorubric: Unifying rubric-based llm evaluation. Preprint, arXiv:2603.00077. — [arXiv:2603.00077](https://arxiv.org/abs/2603.00077) find
  40. Delip Rao and Chris Callison-Burch. 2026b. Bibtex citation hallucinations in scientific publishing agents: Evaluation and mitigation. Preprint, arXiv:2604.03159. — [arXiv:2604.03159](https://arxiv.org/abs/2604.03159) find
  41. Delip Rao, Eric Wong, and Chris Callison-Burch. 2026. Detecting and correcting reference hallucinations in commercial llms and deep research agents. Preprint, arXiv:2604.03173. — [arXiv:2604.03173](https://arxiv.org/abs/2604.03173) find
  42. Tim Rietz, Peyman Toreini, and Alexander Maedche. 2020. Cody: An interactive machine learning system for qualitative coding. In Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pages 90–92. find
  43. Hua Shen, Tiffany Knearem, Reshmi Ghosh, Kenan Alkiek, Kundan Krishna, Yachuan Liu, Ziqiao Ma, Savvas Petridis, Yi-Hao Peng, Li Qiwei, Sushrita Rakshit, Chenglei Si, Yutong Xie, Jeffrey P. Bigham, Frank Bentley, Joyce Chai, Zachary Lipton, Qiaozhu Mei, Rada Mihalcea, and 5 others. 2025. Towards bidi… find
  44. Melanie Subbiah, Akankshya Mishra, Grace Kim, Liyan Tang, Greg Durrett, and Kathleen McKeown. 2025. Is the top still spinning? evaluating subjectivity in narrative understanding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 185–203, Suzhou, China.… find
  45. Elsbeth Turcan and Kathy McKeown. 2019. Dreaddit: A Reddit dataset for stress analysis in social media. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), pages 97–107, Hong Kong. Association for Computational Linguistics. find
  46. Hanlin Wang, Chak Tou Leong, Jian Wang, and Wenjie Li. 2024. E 2 CL: Exploration-based error correction learning for embodied agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7626–7639, Miami, Florida, USA. Association for Computational Linguistics. find
  47. Lei Wang and Eduard Dragut. 2024. The overlooked repetitive lengthening form in sentiment analysis. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16225–16238, Miami, Florida, USA. Association for Computational Linguistics. find
  48. Lei Wang, Min Huang, and Eduard Dragut. 2026. Danceha: A multi-agent framework for document-level aspect-based sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 40(35):29714–29722. find
  49. Ximing Wen. 2025. Language model meets prototypes: Towards interpretable text classification models through prototypical networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 29307–29308. find
  50. Ximing Wen and Rezvaneh Rezapour. 2025. A transformer and prototype-based interpretable model for contextual sarcasm detection. arXiv e-prints, pages arXiv–2503. find
  51. Ximing Wen, Wenjuan Tan, and Rosina Weber. 2025. Gaprotonet: A multi-head graph attention-based prototypical network for interpretable text classification. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9891–9901. find
  52. M. West, M. Tincani, D. Hantula, and 1 others. 2025. Applying data science practices to identify characteristics of postsecondary autism support programs from their websites. Journal of Autism and Developmental Disorders. find
  53. Qiling Wu, Hyangeun Ji, Yuhyun Park, Sori Kim, Jiayu Yang, and Lei Wang. 2025a. East asian international doctoral students’ role identity development in the united states: A collaborative autoethnography. Journal of International Students, 15(1):61–86. find
  54. Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, and Sainbayar Sukhbaatar. 2025b. Meta-rewarding language models: Self-improving alignment with LLM-as-a-meta-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing…
  55. Huimin Xu, Seungjun Yi, Terence Lim, Jiawei Xu, Andrew Well, Carlos Mery, Aidong Zhang, Yuji Zhang, Heng Ji, Keshav Pingali, Yan Leng, and Ying Ding. 2025. Tama: A human-ai collaborative thematic analysis framework using multi-agent llms for clinical interviews. Preprint, arXiv:2503.20666. — [arXiv:2503.20666](https://arxiv.org/abs/2503.20666) find
  56. Tianyu Yang, Yuhan Liu, Sobin Alosious, Ethan A. Brown, Jason R. Rohr, Tengfei Luo, and Xiangliang Zhang. 2025a. Quest2DataAgent: Automating end-to-end scientific data collection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages… find
  57. Tianyu Yang, Yuhan Liu, Sobin Alosious, Ethan A. Brown, Jason R. Rohr, Tengfei Luo, and Xiangliang Zhang. 2025b. Quest2DataAgent: Automating end-to-end scientific data collection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages… find
  58. Seungjun Yi, Joakim Nguyen, Huimin Xu, Terence Lim, Andrew Well, Mia Markey, and Ying Ding. 2025. find
  59. Auto-ta: Towards scalable automated thematic analysis (ta) via multi-agent large language models with reinforcement learning. Preprint, arXiv:2506.23998. — [arXiv:2506.23998](https://arxiv.org/abs/2506.23998) find
  60. Li S. Yifei, Allen Chang, Chaitanya Malaviya, and Mark Yatskar. 2025. Researchqa: Evaluating scholarly question answering at scale across 75 fields with survey-mined questions and rubrics. Preprint, arXiv:2509.00496. find
  61. Qi Zhang, Zhijia Chen, Huitong Pan, Cornelia Caragea, Longin Jan Latecki, and Eduard Dragut. 2024. SciER: An entity and relation extraction dataset for datasets, methods, and tasks in scientific documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pag… find
  62. Qi Zhang, Huitong Pan, Zhijia Chen, Longin Jan Latecki, Cornelia Caragea, and Eduard Dragut. 2025a. DynClean: Training dynamics-based label cleaning for distantly-supervised named entity recognition. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2540–2556, Albuquerq… find
  63. Qingjie Zhang, Di Wang, Haoting Qian, Yiming Li, Tianwei Zhang, Minlie Huang, Ke Xu, Hewu Li, Liu Yan, and Han Qiu. 2025b. Understanding the dark side of LLMs’ intrinsic self-correction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers…
  64. Shanshan Zhang, Lihong He, Eduard Dragut, and Slobodan Vucetic. 2019. How to invest my time: Lessons from human-in-the-loop entity extraction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, page 2305–2313, New York, NY, USA. Association… find
  65. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  66. Andrew Zhu and Chris Callison-Burch. 2025. Overhearing llm agents: A survey, taxonomy, and roadmap. Preprint, arXiv:2509.16325. — [arXiv:2509.16325](https://arxiv.org/abs/2509.16325) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-centaurta-self-improving-human-agents/