Elamur Research⌕
☰
home / papers / 2026 / Conjecture and Inquiry: Quantifying Software Performance Requirements via Interactive Retrieval-Augmented Preference Elicitation

Conjecture and Inquiry: Quantifying Software Performance Requirements via Interactive Retrieval-Augmented Preference Elicitation

Paper · 2026 · Shihai Wang and Tao Chen

Summary

Software performance requirements are usually written as vague natural-language statements, such as a minimum number of concurrent users. These statements leave open how strictly a threshold should be enforced and how values above or below it should be judged. The stakeholders themselves are often unsure of their exact preferences. The authors frame quantification as finding a piecewise satisfaction function, built from three linear patterns (P1–P3). Each function is characterised by its precision (how many patterns and thresholds it has) and its difficulty (the values of T, ∆ and satisfaction levels). The task is then cast as a state-transition search toward the stakeholder's true function. Their method, IRAP, has three stages:

  • It drafts an initial function using pattern classification together with LLM generation.
  • It adjusts that draft by analogy with retrieved past quantifications.
  • It refines the result through a tree-based question-answering dialogue with the stakeholder.

Compared with 10 state-of-the-art methods on four real-world datasets, IRAP performs best in every case. The distance between its output and the true function can shrink by as much as 40×, and it needs as few as five interaction rounds. This makes requirement quantification more practical for performance modeling, tuning and testing.

Key points

  • Requirements are modelled with three linear patterns: P1 (higher is better), P2 (lower is better) and P3 (an exact target value). By default, the tolerance ∆ is set to 10% × T. A sweep from 5% to 15% confirmed that 10% is generally the best choice.
  • Refinement uses three operations: ADD and REMOVE change the patterns (precision), and CHANGE adjusts T, ∆ or satisfaction values (difficulty). The objective is to reach the preferred function in as few steps as possible.
  • Baselines span RL-based alignment (e.g., DPO, WPO), RAG-based alignment and the rule-based LQPR. Experiments draw on real-world requirement data such as the PURE dataset, plus synthetic data generated with GPT-4. Open LLMs such as Qwen and GPT-2 were also used.
  • IRAP currently requires complex requirements to be split into single-pattern fragments. According to prior work, only about 22% of real-world performance requirements contain more than one pattern.
  • Code and data are released at github.com/ideas-labo/irap. The paper was published in ACL 2026 Findings by authors from UESTC and the University of Birmingham.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of BirminghamBirmingham · United Kingdom25 works · 62,564 citations🎓University of Electronic Science and Technology of ChinaChengdu · China31 works · 2,012 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksComputer Networks and Communications · 64 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Shihai Wang and Tao Chen. 2026. Conjecture and Inquiry: Quantifying Software Performance Requirements via Interactive Retrieval-Augmented Preference Elicitation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2042

@inproceedings{wang2026conjecture,
  title = {{Conjecture and Inquiry: Quantifying Software Performance Requirements via Interactive Retrieval-Augmented Preference Elicitation}},
  author = {Shihai Wang and Tao Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2042},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2042}
}

References

42 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2005. The promise repository of software engineering https://openscience.us/repo/ databases. requirements/requirements-other/nfr.html. Retrieved on Jan 01, 2025. ↗
  2. Waad Alhoshan, Alessio Ferrari, and Liping Zhao. 2025. How effective are generative large language models in performing requirements classification? CoRR, abs/2504.16768. — [arXiv:2504.16768](https://arxiv.org/abs/2504.16768) find
  3. Helmut Alt and Michael Godau. 1995. Computing the fréchet distance between two polygonal curves. Int. J. Comput. Geom. Appl., 5:75–91. find
  4. Luciano Baresi, Liliana Pasquale, and Paola Spoletini. 2010. Fuzzy goals for requirements-driven adaptation. In RE 2010, 18th IEEE International Requirements Engineering Conference, Sydney, New South Wales, Australia, September 27 - October 1, 2010, pages 125–134. IEEE Computer Society. find
  5. Manal Binkhonain and Reem Alfayaz. 2025. Are prompts all you need? evaluating prompt-based large language models (llm)s for software requirements classification. CoRR, abs/2509.13868. — [arXiv:2509.13868](https://arxiv.org/abs/2509.13868) find
  6. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cas… find
  7. Sebastian Bruch, Siyu Gai, and Amir Ingber. 2024. An analysis of fusion functions for hybrid retrieval. ACM Trans. Inf. Syst., 42(1):20:1–20:35. find
  8. Tianfeng Chai and Roland R Draxler. 2014. Root mean square error (rmse) or mean absolute error (mae)?– arguments against avoiding rmse in the literature. Geoscientific model development, 7(3):1247–1250. find
  9. Pengzhou Chen and Tao Chen. 2026. Promisetune: Unveiling causally promising and explainable configuration tuning. In 48th IEEE/ACM International Conference on Software Engineering (ICSE). ACM. find
  10. Pengzhou Chen, Tao Chen, and Miqing Li. 2024. MMO: meta multi-objectivization for software configuration tuning. IEEE Trans. Software Eng., 50(6):1478–1504. find
  11. Tao Chen, Ke Li, Rami Bahsoon, and Xin Yao. 2018. FEMOSAA: feature-guided and knee-driven multi-objective optimization for self-adaptive software. ACM Trans. Softw. Eng. Methodol., 27(2):5:1–5:50. find
  12. Tao Chen and Miqing Li. 2023. Do performance aspirations matter for guiding software configuration tuning? an empirical investigation under dual performance objectives. ACM Trans. Softw. Eng. Methodol., 32(3):68:1–68:41. find
  13. Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  14. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach… find
  15. Lawrence Chung, Brian A. Nixon, Eric Yu, and John Mylopoulos. 2000. Non-Functional Requirements in Software Engineering, volume 5 of International Series in Software Engineering. Springer. find
  16. Chengwen Du and Tao Chen. 2025. Causally perturbed fairness testing. ACM Transactions on Software Engineering and Methodology. find
  17. Jonas Eckhardt, Andreas Vogelsang, Henning Femmer, and Philipp Mager. 2016a. Challenging incompleteness of performance requirements by sentence patterns. In 24th IEEE International Requirements Engineering Conference, RE 2016, Beijing, China, September 12-16, 2016, pages 46–55. IEEE Computer Society… find
  18. Jonas Eckhardt, Andreas Vogelsang, and Daniel Méndez Fernández. 2016b. Are "non-functional" requirements really non-functional?: an investigation of non-functional requirements in practice. In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 1… find
  19. Carmine Ferrara, Francesco Casillo, Carmine Gravino, Andrea De Lucia, and Fabio Palomba. 2024. Re-fair: Toward a context-aware recommender for fairness requirements engineering. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 1… find
  20. Alessio Ferrari, Giorgio Oronzo Spagnolo, and Stefania Gnesi. 2017. PURE: A dataset of public requirements documents. In 25th IEEE International Requirements Engineering Conference, RE 2017, Lisbon, Portugal, September 4-8, 2017, pages 502–505. IEEE Computer Society. find
  21. Martin Glinz. 2007. On non-functional requirements. In 15th IEEE International Requirements Engineering Conference, RE 2007, October 15-19th, 2007, New Delhi, India, pages 21–26. IEEE Computer Society. find
  22. Jingzhi Gong, Tao Chen, and Rami Bahsoon. 2025. Dividable configuration performance learning. IEEE Trans. Software Eng., 51(1):106–134. find
  23. Tobias Hey, Jan Keim, Anne Koziolek, and Walter F. Tichy. 2020. Norbert: Transfer learning for requirements classification. In 28th IEEE International Requirements Engineering Conference, RE 2020, Zurich, Switzerland, August 31 - September 4, 2020, pages 169–179. IEEE. find
  24. Kristen Howell, Gwen Christian, Pavel Fomitchov, Gitit Kehat, Julianne Marzulla, Leanne Rolston, Jadin Tredup, Ilana Zimmerman, Ethan Selfridge, and Joseph Bradley. 2023. The economic trade-offs of large language models: A case study. CoRR, abs/2306.07402. find
  25. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open-Review.net.
  26. Harold W. Kuhn. 2010. The hungarian method for the assignment problem. In Michael Jünger, Thomas M. Liebling, Denis Naddef, George L. Nemhauser, William R. Pulleyblank, Gerhard Reinelt, Giovanni Rinaldi, and Laurence A. Wolsey, editors, 50 Years of Integer Programming 1958-2008 - From the Early Year… find
  27. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  28. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Proc…
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  30. Xianchang Luo, Yinxing Xue, Zhenchang Xing, and Jiamou Sun. 2022. PRCBERT: prompt learning for requirement classification using bert-based pre-trained language models. In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022, page… find
  31. Youpeng Ma, Tao Chen, and Ke Li. 2025. Faster configuration performance bug testing with neural dual-level prioritization. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025, pages 988–1000. IEEE. find
  32. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — Training language models to follow instructions with human feedback
  33. Murat Polat. 2020. Analysis of multiple-choice versus open-ended questions in language tests according to different cognitive domain levels. Novitas-ROYAL (Research on Youth and Language), 14(2):76–96. find
  34. Michael James David Powell. 1981. Approximation theory and methods. Cambridge university press. find
  35. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Sy…
  36. Mohammed Sayagh, Noureddine Kerzazi, Bram Adams, and Fábio Petrillo. 2020. Software configuration engineering in practice interviews, survey, and systematic literature review. IEEE Trans. Software Eng., 46(6):646–673. find
  37. Zain Shaukat Shaukat, Rashid Naseem, and Muhammad Zubair. 2018. A dataset for software requirements risk prediction. In 2018 IEEE International Conference on Computational Science and Engineering, CSE 2018, Bucharest, Romania, October 29-31, 2018, pages 112–118. IEEE Computer Society. find
  38. U.S. Department of Health and Human Services. 2016. HealthCare.gov: Case study of CMS management of the federal marketplace. Office of Inspector General Report. Accessed online. find
  39. Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.
  40. Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan Ö. Arik. 2025. Astute RAG: overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), AC…
  41. Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018. Joint embedding of words and labels for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Aust…
  42. To that end, we extract and prepare the anchors following the steps below: find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-conjecture-inquiry-quantifying-software-performance/