Elamur Research⌕
☰
home / papers / 2026 / Scaling Intent Understanding: A Framework for Classification with Clarification using Lightweight LLMs

Scaling Intent Understanding: A Framework for Classification with Clarification using Lightweight LLMs

Paper · 2026 · Subhadip Nandi, Tanishka Agarwal, Anshika Singh et al.

Summary

Most task-oriented dialogue systems assign a single intent even when a user's message could fit several. Proprietary LLMs can resolve this by asking follow-up questions, but they are too expensive at production scale. This Walmart paper shows how to train small open-source LLMs to classify intent and ask clarifying questions when needed.

Starting from ordinary intent-tagged data, a strong LLM creates ambiguous variants of clear queries. Two LLMs, one acting as assistant and one as user, then simulate multi-turn dialogues. Only dialogues that score well under an LLM-as-a-Judge evaluation are kept for supervised instruction-tuning of 3B–8B models. A label-dropout trick teaches the models to recognize out-of-domain requests.

The authors also propose a combined metric, SDRS. It rewards correct intents, asking questions at the right moments, and good question quality. Tested on a proprietary e-commerce dataset, SNIPS and ATIS, the tuned models roughly match GPT-4o and Gemini Flash-Lite at about one-eighth the cost. In live deployment they reduced misrouting.

Key points

  • Pipeline: the framework has three stages: synthetic conversation generation, instruction-tuning, and evaluation.
    • Ambiguous queries come from either editing clear queries or writing one query that fits two intents.
    • Label-dropout OOD training, inspired by class-dropout, removes the true label from some examples so the model learns to answer "out-of-domain."
  • SDRS metric: it blends a conversation-level Ambiguity Resolution Score (ARS) with response quality, weighted by α = 0.5.
    • Response quality covers relevance, conciseness, repetition avoidance, factuality and politeness.
  • Data:
    • Proprietary set: 14 intents plus an OOD class.
    • SNIPS and ATIS: only 1400 and 1700 training examples were used, since adding more brought little gain.
    • Baselines: CTRAN, CoBiC, untuned Llama-3.1-8B/70B and Qwen-2.5-7B, GPT-4o, and Gemini 2.0/2.5 Flash-Lite.
  • Serving cost:
    • Setup: TensorRT-LLM with Triton on one A100 80GB GPU, reaching 50–75 QPS with latency under one second.
    • Estimated monthly cost at 50 QPS: about $3.2K, compared with $26.2K for Gemini 2.0 Flash-Lite and $874.8K for GPT-4o.
  • Production A/B test (8B model):
    • Misrouting fell by 8% and end-to-end resolution time dropped by 19%.
    • The automation rate improved, reducing escalations to human agents.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Subhadip Nandi, Tanishka Agarwal, Anshika Singh and Priyanka Bhatt. 2026. Scaling Intent Understanding: A Framework for Classification with Clarification using Lightweight LLMs. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.14

@inproceedings{nandi2026scaling,
  title = {{Scaling Intent Understanding: A Framework for Classification with Clarification using Lightweight LLMs}},
  author = {Subhadip Nandi, Tanishka Agarwal, Anshika Singh and Priyanka Bhatt},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.14},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.14}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Neeraj Agrawal, Saurabh Kumar, Priyanka Bhatt, and Tanishka Agarwal. 2023. Hierarchical text classification using contrastive learning informed path guided hierarchy. In ECAI 2023, pages 19–26. IOS Press. find
  2. Andrea Alfieri, Ralf Wolter, and Seyyed Hadi Hashemi. 2022. Intent disambiguation for task-oriented dialogue systems. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 5079–5080. find
  3. Mohammad Aliannejadi, Leif Azzopardi, Hamed Zamani, Evangelos Kanoulas, Paul Thomas, and Nick Craswell. 2021. Analysing mixed initiatives and search strategies during conversational search. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 16–26. find
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, and Fei Huang. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Cen Chen, Chilin Fu, Xu Hu, Xiaolu Zhang, Jun Zhou, Xiaolong Li, and Forrest Sheng Bao. 2019. Reinforcement learning for user intent prediction in customer service bots. In Proceedings of the 42Nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1265– 12… find
  7. Maximillian Chen, Ruoxi Sun, Sercan Ö Arık, and Tomas Pfister. 2024. Learning to clarify: Multi-turn conversations with action-based contrastive self-training. arXiv preprint arXiv:2406.00222. — [arXiv:2406.00222](https://arxiv.org/abs/2406.00222) find
  8. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, and Evan Rosen. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprin…
  9. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, and Thibaut Lavril. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv prepr…
  10. Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. 2005. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19–67. find
  11. Kaustubh D Dhole. 2020. Resolving intent ambiguities by retrieving discriminative clarifying questions. arXiv preprint arXiv:2008.07559. — [arXiv:2008.07559](https://arxiv.org/abs/2008.07559) find
  12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  13. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, and Honghao Liu. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  14. Charles T Hemphill, John J Godfrey, and George R Doddington. 1990. The atis spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990. find
  15. Floris den Hengst, Ralf Wolter, Patrick Altmeyer, and Arda Kaygan. 2024. Conformal intent classification and clarification for fast and accurate intent recognition. arXiv preprint arXiv:2403.18973. find
  16. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  17. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, and Alec Radford. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  18. Bamba Kane, Fabio Rossi, Ophélie Guinaudeau, Valeria Chiesa, Ilhem Quénel, and Stéphane Chau. 2021. Joint intent detection and slot filling via cnn-lstm-crf. In 2020 6th IEEE Congress on Information Science and Technology (CiSt), pages 342–347. IEEE. find
  19. Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges. ACM Computing Surveys, 55(6):1– 40. find
  20. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022. Clam: Selective clarification for ambiguous questions with generative language models. arXiv preprint arXiv:2212.07769. find
  21. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  22. Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, Chenxue Wang, Shichao Liu, and Qing Wang. 2023. Clarifygpt: Empowering llm-based code generation with intention clarification. arXiv preprint arXiv:2310.10996. — [arXiv:2310.10996](https://arxiv.org/abs/2310.10996) find
  23. Subhadip Nandi, Neeraj Agrawal, Anshika Singh, and Priyanka Bhatt. 2024. Enhancing customer service chatbots with context-aware nlu through selective attention and multi-task learning. In Proceedings of the 8th International Conference on Data Science and Management of Data (12th ACM IKDD CODS and 3… find
  24. Workshop on Machine Learning and Programming Languages, pages 10–19. find
  25. Mickey van Zeelt, Floris den Hengst, and Seyyed Hadi Hashemi. 2020. Collecting high-quality dialogue user satisfaction ratings with third-party annotators. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, pages 363– 367. find
  26. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
  27. Kuan Xu, Chilin Fu, Xiaolu Zhang, Cen Chen, Ya-Lin Zhang, Wenge Rong, Zujie Wen, Jun Zhou, Xiaolong Li, and Yu Qiao. 2020. admscn: a novel perspective for user intent prediction in customer service bots. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, p… find
  28. Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck. 2020. Generating clarifying questions for information retrieval. In Proceedings of the web conference 2020, pages 418–428. find
  29. Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2003. On the means for clarification in dialogue. In Current and new directions in discourse and dialogue, pages 235–255. Springer. find
  30. Mehrdad Rafiepour and Javad Salimi Sartakhti. 2023. Ctran: Cnn-transformer-based network for natural language understanding. Engineering Applications of Artificial Intelligence, 126:107013. find
  31. Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2023. Gollie: Annotation guidelines improve zero-shot information-extraction. arXiv preprint arXiv:2310.03668.
  32. Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, and Tat-Seng Chua. 2024. Clamber: A benchmark of identifying and clarifying ambiguous information needs in large language models. arXiv preprint arXiv:2405.12063.
  33. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, and Eric Xing. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  34. A Clemencia Siro, Mohammad Aliannejadi, and Maarten de Rijke. 2022. Understanding user satisfaction with task-oriented dialogue systems. In Proceedings of the 45th International ACM SIGIR conference on research and development in information retrieval, pages 2018–2023. find
  35. Leila Tavakoli. 2020. Generating clarifying questions in conversational search systems. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 3253–3256. find
  36. Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Ablation Study

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nandi-2026-scaling-intent-understanding-framework-classification/