Elamur Research⌕
☰
home / papers / 2026 / A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews

A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews

Paper · 2026 · Aakash Trivedi, Aniket Upadhyay, Pratik Narang et al.

Summary

Customer reviews sometimes contain concrete requests for businesses, such as adding menu options or speeding up check-in. These requests are rare (13–18% of reviews) and appear alongside praise, complaints, and advice aimed at other customers. Earlier suggestion-mining work mostly flags sentences that contain a suggestion, and opinion summarization produces broad themes. Neither approach yields the precise operational fix a business needs.

The authors propose a two-stage hybrid pipeline. First, a RoBERTa-base classifier decides whether a review contains an actionable suggestion. It is trained with cross-entropy plus a differentiable precision–recall surrogate loss that pushes for high recall, since a missed suggestion cannot be recovered later. Second, a quantized, instruction-tuned Gemma-3 27B model running under Ollama handles the remaining work through few-shot prompts: it extracts and rewrites suggestions, assigns categories, clusters them, and summarizes each cluster.

On restaurant and ice-cream review data, the hybrid system beats prompt-only LLM, rule-based, and classifier-only baselines in extraction, categorization, clustering, and summarization. Industry experts also rated its outputs as clear and faithful. The study shows that classifier gating combined with tightly controlled LLM prompting can be deployed locally at industrial scale.

Key points

  • The classifier was trained on a proprietary set of 1,110 reviews (440 positive, 670 negative), and its learning curve levels off at about 70% of the data. The raw texts are not released, but the prompts, hyperparameters, and hardware specifications are provided.
  • On 150 held-out instances, category assignment accuracy was 0.90 for the hybrid pipeline, 0.78 for the prompt-only LLM, and 0.62 for the rule-based system.
  • Cluster summaries scored ROUGE-L 0.46 and BERTScore F1 0.91 for the hybrid pipeline, compared with 0.34/0.86 for prompt-only and 0.22/0.75 for rule-based. Three experts rated the outputs on a 1–5 Likert scale with κ = 0.74–0.85, giving clustering a score of 5.0.
  • Ablations showed the following effects:
    • Removing the PR loss lowered recall by 3.49%.
    • Removing clustering cut human-rated interpretability by 22%.
    • Dropping category assignment reduced AMI by 0.12.
    • Running without quantization required 2.4× more memory and 47% more latency, with ΔF1 < 0.01.
  • In cross-domain tests (real estate, healthcare, finance, automotive), recall stayed robust but precision fell where specialized terminology was common. The main remaining errors were sarcasm and occasional mis-clustering by the LLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Birla Institute of Technology and Science, PilaniPilāni · India16 works · 41,879 citations🏢VMwarePalo Alto · United States2 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Aakash Trivedi, Aniket Upadhyay, Pratik Narang, Dhruv Kumar and Praveen Kumar. 2026. A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.61

@inproceedings{trivedi2026hybrid,
  title = {{A Hybrid Supervised-LLM Pipeline for Actionable Suggestion Mining in Unstructured Customer Reviews}},
  author = {Aakash Trivedi, Aniket Upadhyay, Pratik Narang, Dhruv Kumar and Praveen Kumar},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.61},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.61}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stefanos Angelidis and Mirella Lapata. 2021. Summarizing opinions with gsum. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  2. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022.
  3. Algirdas Bražinskas, Róbert Busa-Fekete, and Daniel Preotiuc-Pietro. 2020. Learning to summarize product reviews by exploiting aspect-level ground truth. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877–…
  5. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL).
  6. Adji B. Dieng, Chong Wang, Jianfeng Gao, and John Paisley. 2020. Topic modeling in embedding spaces. In Transactions of the Association for Computational Linguistics, volume 8, pages 439–453.
  7. Li Dong, Qi Qian, and Lei Jiang. 2017. Attention-based neural networks for suggestion mining. In Proceedings of the 2017 Conference on Natural Language Processing (ACL). find
  8. Zhenzhong Ji, Richard Lee, Joseph Fries, and Roger Levy. 2023. A survey of hallucination in large language models. ACM Computing Surveys, 56(12):1– 42. find
  9. Xinyu Jiang, Min Chen, and Zhen Li. 2023. Mistral: Open-weight instruction-tuned language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  10. Mandar Joshi, Danqi Chen, and Yinhan Liu. 2020. Spanbert: Improving pre-training by representing and predicting spans. In Transactions of the Association for Computational Linguistics, volume 8, pages 64–77.
  11. Ryota Koto, Keisuke Yoshida, and Takuya Tanaka. 2022. Span-level inconsistencies in llm-based extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  13. Sai Mukku and Praneeth Mukku. 2024. Insightnet: A neural network for semantic theme extraction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  14. Mir Tafseer Nayeem and Davood Rafiei. 2024. LFO-Sum: Large-scale summarization of fine-grained opinions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). identified the top-performing models and further filtered them to select the best candidate for… find
  15. Model GPT-2 Small GPT-2 Medium ROBERTa-Large DeBERTa-Large BERT-Large XLNet-Large BART-Large ROBERTa-Base Sapna Negi and Paul Buitelaar. 2015. Towards the extraction of customer-to-customer suggestions from reviews. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proce… find
  16. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Journal of Machine Learning Research, 23:1–60.
  17. Muhammad Riaz, Ayesha Raza, and Hira Javed. 2024. TransLSTM: Transformer-enhanced LSTM for suggestion mining. Natural Language Engineering, 30(2):123–145. find
  18. Gemma Team. 2024. Gemma: Instruction-tuned large language model for structured extraction. https://gemma-model.org. ↗
  19. Budi Wicaksono and Sung-Hyon Myaeng. 2013. Mining product improvement suggestions from customer reviews. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL). find
  20. Yuxin Wu, Lili Wang, and Ming Chen. 2023. A survey of hybrid approaches for nlp tasks: Classifier-language model pipelines. ACM Computing Surveys, 56(7):1–36. find
  21. Peng Zhou and Wei Xu. 2018. Joint extraction of entities and relations based on a novel tagging scheme. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/trivedi-2026-hybrid-supervised-llm-pipeline-actionable/