Elamur Research⌕
☰
home / papers / 2026 / What Question Did You Answer? Refining Contact Center Evaluation Plans via Backward Questions

What Question Did You Answer? Refining Contact Center Evaluation Plans via Backward Questions

Paper · 2026 · Prajwal Sood, Rushikesh Pawar, Digvijay Anil Ingle et al.

Summary

Contact center Quality Assurance (QA) scores agent conversations against company-specific questionnaires. Smaller, cheaper LMs handle this well only when given explicit evaluation plans. Large LMs tend to fill gaps with generic industry norms, and human QA experts find it hard to put their tacit judgment into words. The authors propose Backward Question-based Refinement (BQR) to close this gap.

In each BQR iteration, a small model (nova-lite) answers training items using the current plan. For each response, a large model (nova-pro) then infers the question the small model effectively answered, exposing how it misread the task. These backward questions are embedded, filtered for outliers and clustered. The best- and worst-performing clusters are condensed into representative questions, and the large model uses them to rewrite the plan. Validation-based early stopping selects the final plan.

On 12 QA questions drawn from proprietary banking, automotive and healthcare calls, BQR improved 8 questions. The approach turns expert intuition into auditable criteria and makes cheap models practical for QA at scale.

Key points

  • Dataset:
    • The authors sampled 300 conversations per question, giving about 3,600 pairs.
    • Five experts labeled each pair yes/no.
    • Only pairs with at least 4/5 agreement were kept, leaving 2,297 pairs (63.8%).
    • The data were split 50:25:25 with stratification and are not publicly released.
  • Pipeline:
    • all-mpnet-base-v2 embeddings for the backward questions.
    • Isolation Forest for outlier removal.
    • K-Means for clustering.
    • Greedy decoding for all LM calls.
  • Main result: BQR improved 8 of 12 questions, with a best absolute Macro F1 gain of 27.8% on the "Focus Resol." question. The refined plans added, removed or substituted criteria.
  • Gradient-descent analogy: The authors compare outlier removal to regularization and generation temperature to learning-rate control.
  • Cross-family advantage: Small LMs gained more when the refining large LM came from a different model family.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Prajwal Sood, Rushikesh Pawar, Digvijay Anil Ingle and Anup Pattnaik. 2026. What Question Did You Answer? Refining Contact Center Evaluation Plans via Backward Questions. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.113

@inproceedings{sood2026what,
  title = {{What Question Did You Answer? Refining Contact Center Evaluation Plans via Backward Questions}},
  author = {Prajwal Sood, Rushikesh Pawar, Digvijay Anil Ingle and Anup Pattnaik},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.113},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.113}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Justin Chih-Yao Chen, Zifeng Wang, Hamid Palangi, Rujun Han, Sayna Ebrahimi, Long T. Le, Vincent Perot, Swaroop Mishra, Mohit Bansal, Chen-Yu Lee, and Tomas Pfister. 2025. Reverse thinking makes llms stronger reasoners. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of… find
  2. Jaekeol Choi. 2025. Efficient prompt optimization for relevance evaluation via llm-based confusion matrix feedback. Applied Sciences, 15(9). find
  3. Md. Faisal Mahbub Chowdhury, Nandana Mihindukulasooriya, Niharika S. D’Souza, Horst Samulowitz, Neeru Gupta, Tomasz Hanusiak, and Michal Kapitonow. 2026. Automatic prompt engineering with no task cues and no tuning. CoRR, abs/2601.03130. — [arXiv:2601.03130](https://arxiv.org/abs/2601.03130) find
  4. Jaewon Chu, Seunghun Lee, and Hyunwoo J. Kim. 2025. PRESTO: preimage-informed instruction optimization for prompting black-box LLMs. CoRR, abs/2510.25808. — [arXiv:2510.25808](https://arxiv.org/abs/2510.25808) find
  5. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2024. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedin…
  6. Sophie Henning, William Beluch, Alexander Fraser, and Annemarie Friedrich. 2023. A survey of methods for addressing class imbalance in deep-learning based natural language processing. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL… find
  7. Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. 2023. Instruction induction: From few examples to natural language task descriptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 1935– 1952. find
  8. Digvijay Ingle, Aashraya Sachdeva, Surya Prakash Sahu, Mayank Sati, Cijo George, and Jithendra Vepa. 2024. Probing the depths of language models’ contact-center knowledge for quality assurance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: EMNLP 2024 - In… find
  9. Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T. Kwok. 2024. Forward-backward reasoning in large language models for mathematical verification. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11… find
  10. Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with "gradient descent" and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 79…
  11. Md. Abdur Rahman, Abdul Barek, ABM Kamrul Islam Riad, Md. Mostafizur Rahman, Md Bajlur Rashid, Md Raihan Mia, Hossain Shahriar, Guillermo Francia III, Fan Wu, Alfredo Cuzzocrea, and Sheikh Iqbal Ahamed. 2025. Embedding with large language models for classification of HIPAA safeguard compliance rules… find
  12. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, Ch…
  13. Shourya Roy, Ragunathan Mariappan, Sandipan Dandapat, Saurabh Srivastava, Sainyam Galhotra, and Balaji Peddamuthu. 2016. Qart : A system for real-time holistic quality assurance for contact center dialogues. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17,… find
  14. Prajwal Sood, Rajdeep Agrawal, Mayank Sati, Digvijay Ingle, and Cijo George. 2025. STREAQ: selective tiered routing for effective and affordable contact center quality assurance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025 - Industry Track, S… find
  15. Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. Isolation forest. In Proceedings of the 8th IEEE International Conference on Data Mining (ICDM 2008), December 15-19, 2008, Pisa, Italy, pages 413– 422. IEEE Computer Society. find
  16. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processi…
  17. J. B. MacQueen. 1967. Some methods for classification and analysis of multivariate observations. Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, 1(14):281–297. find
  18. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. CoRR, abs/2309.03409.
  19. Subash Neupane, Shaswata Mitra, Sudip Mittal, and Shahram Rahimi. 2025. Towards a HIPAA compliant agentic AI system in healthcare. CoRR, abs/2504.17669. — [arXiv:2504.17669](https://arxiv.org/abs/2504.17669) find
  20. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sood-2026-what-question-did-you-answer/