Elamur Research⌕
☰
home / papers / 2026 / Investigating Reasoning with Hypotheses: The RIP2 Corpus

Investigating Reasoning with Hypotheses: The RIP2 Corpus

Summary

Little is known about how people form, keep, and drop hypotheses when they reason together. Annotated data on this has been scarce: before this work, RIP1 was the only corpus of group hypothesis-making. The authors introduce RIP2, built from recordings of 16 small groups of two or three strangers (42 people in total) trying to solve a murder mystery game within 90 minutes.

Data collection was part of a preregistered psychological experiment that also gathered background measures for each player. The game was simplified compared with RIP1. The code-breaking task was replaced by pre-decoded messages, and all evidence items were numbered.

The transcripts total roughly 177,000 words. Six three-person groups were annotated with Inference Anchoring Theory, using new guidelines for marking both hypotheses and conjectures. Half of these groups reached the correct solution. The annotated portion is nearly double the size of RIP1, which makes it the largest annotated hypothesising corpus so far.

Because the game has a known answer, RIP2 lets researchers compare successful and unsuccessful groups and approximate argument quality. It can also be used to train or fine-tune NLP models for argument mining.

Key points

  • Full corpus: 177,661 words from 16 transcripts. This covers 23.4 hours of audio, with sessions of 69–100 minutes (average 88).
  • Annotated subset: 80,000 words from six groups of three (three successful), containing 2666 relations annotated with IAT.
  • Participants were drawn from a pool of 189 people recruited via SONA Systems and local advertising. 19 groups were tested, and some were excluded, two of them because members already knew each other.
  • Design changes from RIP1: the code-breaking sub-task was removed, evidence was numbered and inventoried, pre- and post-game questionnaires were added, and every group had the same time limit.
  • Data and demographics are available at github.com/e-schad/RIP2, and the experiment preregistration is on OSF.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5crysfardw7h

@inproceedings{schad2026investigating,
  title = {{Investigating Reasoning with Hypotheses: The RIP2 Corpus}},
  author = {Schad, Ella and Seyfried, Clara and Reed, Chris},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5crysfardw7h},
  url = {https://doi.org/10.63317/5crysfardw7h}
}

References

38 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Heidi Fischer Bjelland and Johanne Yttri Dahl. 2017. Exploring criminal investigation practices the benefits of analysing police-generated investigation data. European Journal of Policing Studies. find
  2. Katarzyna Budzynska, Mathilde Janier, Juyeon Kang, Chris Reed, Patrick Saint-Dizier, Manfred Stede, and Olena Yaskorska. 2014a. Towards argument mining from dialogue. In Proceedings of the Fifth International Conference on Computational Models of Argument (COMMA 2014), pages 185–196. IOS Press. find
  3. Katarzyna Budzynska, Mathilde Janier, Juyeon Kang, Chris Reed, Patrick Saint-Dizier, Manfred Stede, and Olena Yaskorska. 2014b. Towards argument mining from dialogue. In Computational Models of Argument, pages 185–196. IOS Press. find
  4. Gonçalo Carnaz, Mário Antunes, and Vitor Beires Nogueira. 2021. An annotated corpus of crime-related portuguese documents for nlp and machine learning processing. Data, 6(7):71. find
  5. Mandeep K Dhami, Ian K Belton, and David R Mandel. 2019. The “analysis of competing hypotheses” in intelligence analysis. Applied Cognitive Psychology, 33(6):1080–1090. find
  6. Maria Di Maro, Martina Di Bratto, Sabrina Mennella, Antonio Origlia, and Francesco Cutugno. 2025. Argumentation in recommender dialogue agents (arda): An unexpected journey from pragmatics to conversational agents. Open Linguistics, 11(1):20250052. find
  7. Rory Duthie, John Lawrence, Katarzyna Budzynska, and Chris Reed. 2016. The cass technique for evaluating the performance of argument mining. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 40– 49. find
  8. Markus Eger. 2020. Murder mysteries: the white whale of narrative generation? In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pages 210–216. find
  9. Fina Fakhriyah, Ani Rusilowati, Endang Susilan-ingsih, et al. 2021. Argument-driven inquiry learning model: A systematic review. International Journal of Research in Education and Science, 7(3):767–784. find
  10. Baruch Fischhoff and Ruth Beyth-Marom. 1983. Hypothesis evaluation from a bayesian perspective. Psychological review, 90(3):239. find
  11. Arthur C. Graesser, Stephen M. Fiore, Samuel Greiff, Jessica Andrews-Todd, Peter W. Foltz, and Friedrich W. Hesse. 2018. Advancing the science of collaborative problem solving. Psychological Science in the Public Interest, 19:59–92. find
  12. Ulrike Hahn. 2020. Argument quality in real world argumentation. Trends in Cognitive Sciences, 24(5):363–374. find
  13. Raimo P Hämäläinen, Tuomas J Lahtinen, and Kai Virtanen. 2024. Generating policy alternatives for decision making: A process model, behavioural issues, and an experiment. EURO Journal on Decision Processes, 12:100050. find
  14. Annette Hautli-Janisz, Zlata Kikteva, Wassiliki Siskou, Kamila Gorska, Ray Becker, and Chris Reed. 2022a. QT30: A corpus of argument and conflict in broadcast debate. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3291–3300, Marseille, France. European Language… find
  15. Annette Hautli-Janisz, Ella Schad, and Chris Reed. 2022b. Disagreement space in argument analysis. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 1–9, Marseille, France. European Language Resources Association. find
  16. Richards J Heuer. 1999. Psychology of intelligence analysis. Center for the Study of Intelligence. find
  17. Michael HG Hoffmann. 2018. The elusive notion of “argument quality”. Argumentation, 32(2):213– 240. find
  18. M. Janier, J. Lawrence, and C Reed. 2014. OVA+: An argument analysis interface. In Proceedings of the Fifth International Conference on Computational Models of Argument (COMMA 2014), pages 463–464, Pitlochry. IOS Press. find
  19. Joshua Klayman and Young-won Ha. 1987. Confirmation, disconfirmation, and information in hypothesis testing. Psychological Review, 94(2):211–228. find
  20. Yevhen Kostiuk, Clara Seyfried, and Chris Reed. 2025. Automating alternative generation in decision-making. In Findings of the Association for Computational Linguistics: EMNLP 2025. find
  21. Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. 2023. Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay. arXiv preprint arXiv:2310.14985. find
  22. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159. find
  23. Ryota Nonomura and Hiroki Mori. 2025. Who speaks next? multi-party ai discussion leveraging the systematics of turn-taking in murder mystery games. Frontiers in Artificial Intelligence, 8:1582287. find
  24. Andreas Peldszus and Manfred Stede. 2013. From argument diagrams to argumentation mining in texts: A survey. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI), 7(1):1–31. find
  25. Fenna H Poletiek. 2013. Hypothesis-testing behaviour. Psychology Press. find
  26. Chris Reed and Katarzyna Budzynska. 2011. How dialogues create arguments. In Proceedings of the 7th Conference of the International Society for the Study of Argumentation (ISSA), pages 1633–1645. SicSat Amsterdam. find
  27. Ramon Ruiz-Dolz, John Lawrence, Ella Schad, and Chris Reed. 2024. Overview of dialam-2024: Argument mining in natural language dialogues. In 11th Workshop on Argument Mining, ArgMining 2024, pages 83–92. Association for Computational Linguistics (ACL). find
  28. Ella Schad, Kamila Górska, Eimear Maguire, Ramon Ruiz-Dolz, Melvin Abraham, John Lawrence, and Jacky Visser. 2024a. Annotating and mining hypotheses in argumentation. In The 10th International Conference on Computational Models of Argument, pages 253–264. IOS Press. find
  29. Ella Schad, Jacky Visser, and Chris Reed. 2024b. The rip corpus of collaborative hypothesis-making. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16047–16057. find
  30. Clara Seyfried, Chris Reed, and Yuki Kamide. 2024. Defining argumentative discourse units as clauses: Psycholinguistic evidence. In Proceedings of COMMA 2024, pages 277–288. IOS Press BV. find
  31. Nick Siegel, Blake Shepard, John Cabral, and Michael Witbrock. 2005. Hypothesis generation and evidence assembly for intelligence analysis: Cycorp’s noöscape application. In Proceedings of the 2005 International Conference on Intelligence Analysis (IA 2005), McLean, VA, USA. find
  32. Louisa M. Slowiaczek, Joshua Klayman, Steven J. Sherman, and Richard B. Skov. 1992. Information selection and use in hypothesis testing: What is a good question, and what is a good answer? Memory & Cognition, 20(4):392–405. find
  33. Robert Stalnaker. 2002. Common ground. Linguistics and philosophy, 25(5/6):701–721. find
  34. Chen Sun, Valerie J Shute, Angela Stewart, Jade Yonehiro, Nicholas Duran, and Sidney D’Mello. 2020. Towards a generalized competency model of collaborative problem solving. Computers & Education, 143:103672. find
  35. Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. 2017. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Comput… find our card
  36. Joi Phelps Walker, Victor Sampson, Jonathon Grooms, Brittany Anderson, and Carol O Zimmerman. 2012. Argument-driven inquiry in undergraduate chemistry labs: The impact on students’ conceptual understanding, argument skills, and attitudes toward science. Journal of college science teaching, 41(4):74–… find
  37. Si Zhang, Qianqian Gao, Mengyu Sun, Zhihui Cai, Honghui Li, Yanling Tang, and Qingtang Liu. 2022. Understanding student teachers’ collaborative problem solving: Insights from an epistemic network analysis (ena). Computers & Education, 183:104485. find
  38. Corinne Zimmerman. 2000. The development of scientific reasoning skills. Developmental Review, 20(1):99–149. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schad-2026-investigating-reasoning-hypotheses-rip2-corpus/