Elamur Research⌕
☰
home / papers / 2025 / Rationale-Guided Distillation for E-Commerce Relevance Classification: Bridging Large Language Models and Lightweight Cross-Encoders

Rationale-Guided Distillation for E-Commerce Relevance Classification: Bridging Large Language Models and Lightweight Cross-Encoders

Summary

The authors, from Amazon, address the task of determining the relevance of query–product pairs in online stores. The model must process millions of such pairs every day, which makes large language models (LLMs) too slow and expensive for the job. The authors propose explanation-based distillation. Mixtral 8X7B Instruct generates textual rationales for why a pair is or is not relevant. A lightweight BERT cross-encoder (110M parameters) is trained in a cross-encoder-decoder architecture: the encoder performs binary classification, while the decoder reproduces the LLM's reasoning as an auxiliary task. Only the encoder is deployed to production, so inference costs do not increase. The method was evaluated on 9 multilingual e-commerce datasets, 3 ESCI datasets, and on GLUE and QADSM. Compared with a standard cross-encoder, ROC-AUC improves on average by 1.4%, 2.4% and 6%, respectively. The gap to a 7B-parameter LLM is no more than 1%, and the model is roughly 50 times faster per example. The work demonstrates a practical way to transfer the reasoning ability of LLMs into compact models for industrial search. The approach is especially useful when labeled data is scarce.

Key points

  • LLaMA2-7B and Mistral-7B-v0.3 serve as the upper bound on quality. Their weights are frozen, and only a linear layer on top of the last hidden state is trained. In zero-shot/few-shot modes, LLMs trail the fine-tuned models by more than 20 pp ROC-AUC.
  • With 10K training examples, the method yields +0.3–2.3 pp over BERT and outperforms the best fine-tuned LLM on 6 of 9 datasets. With 100K examples, it is better on only 1 of 9. On ESCI the gain is 1.3–3.9 pp, exceeding even the result of the 7B model.
  • On GLUE the method outperforms BERT but falls short of the LLMs (e.g., MRPC: 0.8811 vs. 0.7955 for BERT). On QADSM it reaches a ROC-AUC of 0.9123, whereas BERT achieves 0.7174 and the best LLM 0.8787.
  • Ablations were carried out on the English-language IN dataset. Decoder Warmup adds up to 2 pp compared with training on explanations alone. The cheaper Reasoning Embeddings variant nearly matches the full decoder. The method's largest gains come with small amounts of data.
  • Attention maps and Shapley values show that the model focuses on the meaningful tokens of the query and the product title more consistently than the baseline BERT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{agrawal2025rationale,
  title = {{Rationale-Guided Distillation for E-Commerce Relevance Classification: Bridging Large Language Models and Lightweight Cross-Encoders}},
  author = {Agrawal, Sanjay and Ahemad, Faizan and Sembium, Vivek Varadarajan},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics: Industry Track},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-industry.12/}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sanjay Agrawal, Srujana Merugu, and Vivek Sembium. 2023a. Enhancing e-commerce product search through reinforcement learning-powered query reformulation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4488–4494. find
  2. Sanjay Agrawal, Vivek Sembium, and MS Ankith. 2023b. Kd-boost: Boosting real-time semantic matching in e-commerce with knowledge distillation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 131–141. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  4. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216. find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  6. Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. 2017. Ensemble distillation for neural machine translation. arXiv preprint arXiv:1702.01802. — [arXiv:1702.01802](https://arxiv.org/abs/1702.01802) find
  7. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543. find our card
  8. Jiafeng Guo, Yinqiong Cai, Yixing Fan, Fei Sun, Ruqing Zhang, and Xueqi Cheng. 2022. Semantic models for the first-stage retrieval: A comprehensive review. ACM Transactions on Information Systems (TOIS), 40(4):1–42. find
  9. Geoffrey Hinton. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  10. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computati… find our card
  11. Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée La… find
  12. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Mi… find our card
  13. Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2645–2652. find
  14. Sourab Mangrulkar, Ankith MS, and Vivek Sembium. 2022. Be3r: Bert based early-exit using expert routing. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3504–3512. find
  15. Michinari Momma, Chaosheng Dong, and Yetian Chen. 2022. Multi-objective ranking with directions of preferences. find
  16. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. find
  17. Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. Zero-infinity: Breaking the GPU memory wall for extreme scale deep learning. CoRR, abs/2104.07857. find
  18. Chandan K. Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022. Shopping queries dataset: A large-scale ESCI benchmark for improving product search. find
  19. V Sanh. 2019. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  20. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint ar… find our card
  21. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984. find our card
  22. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria… find
  23. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… find our card
  24. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776–5788. find
  25. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agrawal-2025-rationale-guided-distillation-e-commerce/