Elamur Research⌕
☰
home / papers / 2024 / Teaching Small Language Models Reasoning through Counterfactual Distillation

Teaching Small Language Models Reasoning through Counterfactual Distillation

Paper · 2024 · Tao Feng, Yicheng Li, Chenglin Li et al.

Summary

The paper addresses a weakness of chain-of-thought (CoT) distillation, where a large teacher model's rationales are used to fine-tune small language models (SLMs). Students trained this way tend to latch onto spurious keyword patterns, generalize poorly to out-of-distribution (OOD) inputs, and see little reasoning diversity. The authors propose a counterfactual distillation framework with three steps. First, causally relevant phrases are located using topic words and syntactic analysis and then masked. Second, an LLM fills the masks so that the new question supports a different target answer, and it also screens the generated examples for quality. Third, a multi-view CoT supervises the student with two kinds of rationale: a positive view CoT (PVC) supporting the correct option, and a negative view CoT (NVC) explaining why each distractor is wrong. On four knowledge-intensive QA benchmarks, the approach beats standard CoT distillation in-distribution and holds up better on OOD data. This matters because it offers a cheap way to give compact models sturdier, elimination-style reasoning.

Key points

  • The average gain over standard CoT distillation on same-distribution data is 11.43%.
  • The improvements hold for student models from 120M to 770M parameters, in both decoder-only and encoder-decoder architectures.
  • Training on a mix of factual and counterfactual examples (CMix) outperforms a same-size factual-only sample (CSample) across models.
  • The method also works on non-multiple-choice tasks, after an LLM generates answer options for them:
    • SVAMP accuracy reaches 16.00%, versus 10.11% for FT-CoT and 8.67% for FT.
    • bAbI accuracy reaches 78.23%, versus 69.20% and 67.40%. The bAbI training set uses 50 examples from each of its 20 subtasks, 1000 in total.
  • The authors list several limitations:
    • Some generated counterfactuals did not match their intended answers and were discarded.
    • Experiments cover English only and single-task settings.
    • The teacher is a closed-source LLM, which adds cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🏢Ant GroupHangzhou · China42 works · 1,131 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Tao Feng, Yicheng Li, Chenglin Li, Hao Chen, Fei Yu and Yin Zhang. 2024. Teaching Small Language Models Reasoning through Counterfactual Distillation. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.333

@inproceedings{feng2024teaching,
  title = {{Teaching Small Language Models Reasoning through Counterfactual Distillation}},
  author = {Tao Feng, Yicheng Li, Chenglin Li, Hao Chen, Fei Yu and Yin Zhang},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.333},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.333}
}

References

36 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yonatan Bitton, Gabriel Stanovsky, Roy Schwartz, and Michael Elhadad. 2021. Automatic generation of contrast sets from scene graphs: Probing the compositional consistency of gqa. arXiv preprint arXiv:2103.09591. find
  2. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata, 58:2. find
  3. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071.
  4. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
  5. Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. 2022. In-context learning distillation: Transferring few-shot learning ability of pre-trained language models. arXiv preprint arXiv:2212.10670. — [arXiv:2212.10670](https://arxiv.org/abs/2212.10670) find
  6. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191.
  7. Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial distillation of closed-source large language model. arXiv preprint arXiv:2305.12870. — [arXiv:2305.12870](https://arxiv.org/abs/2305.12870) find
  8. Pouya Pezeshkpour, Hayate Iso, Thom Lake, Nikita Bhutani, and Estevam Hruschka. 2023. Distilling large language models using skill-occupation graph context for hr-related tasks. arXiv preprint arXiv:2311.06383. — [arXiv:2311.06383](https://arxiv.org/abs/2311.06383) find
  9. Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. arXiv preprint arXiv:1909.12434. find
  10. Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. 2020. More bang for your buck: Natural perturbation for robust question answering. arXiv preprint arXiv:2004.04849. find
  11. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082–8090. find
  12. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  13. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
  14. Jiaxuan Li, Lang Yu, and Allyson Ettinger. 2023a. Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios. arXiv preprint arXiv:2305.16572. find
  15. Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023b. Symbolic chain-of-thought distillation: Small models can also" think" step-by-step. arXiv preprint arXiv:2306.14050.
  16. Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023c. Unified demonstration retriever for in-context learning. arXiv preprint arXiv:2305.04320.
  17. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1:9. find
  18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.
  19. Alexis Ross, Tongshuang Wu, Hao Peng, Matthew E Peters, and Matt Gardner. 2021. Tailor: Generating and perturbing text with semantic controls. arXiv preprint arXiv:2107.07150.
  20. Rachneet Sachdeva, Martin Tutek, and Iryna Gurevych. 2023. Catfood: Counterfactual augmented training for improving out-of-domain performance and calibration. arXiv preprint arXiv:2309.07822. find
  21. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073.
  22. Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. 2023. Measuring inductive biases of in-context learning with underspecified demonstrations. arXiv preprint arXiv:2305.13299. find
  23. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019. Quarel: A dataset and models for answering questions about qualitative relationships. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7063– 7071. find
  24. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937.
  25. Xinwei Li, Li Lin, Shuai Wang, and Chen Qian. 2023d. Unlock the power: Competitive distillation for multi-modal large language models. arXiv preprint arXiv:2311.08213. — [arXiv:2311.08213](https://arxiv.org/abs/2311.08213) find
  26. Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023a. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint arXiv:2305.14160.
  27. Yongqi Li, Mayi Xu, Xin Miao, Shen Zhou, and Tieyun Qian. 2023e. Large language models as counterfactual generator: Strengths and weaknesses. arXiv preprint arXiv:2305.14791. — [arXiv:2305.14791](https://arxiv.org/abs/2305.14791) find
  28. Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023b. Scott: Self-consistent chain-of-thought distillation. arXiv preprint arXiv:2305.01879.
  29. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  30. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
  31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  32. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.
  33. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2021. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. arXiv preprint arXiv:2101.00288.
  34. Bohao Yang, Chen Tang, Kun Zhao, Chenghao Xiao, and Chenghua Lin. 2023. Effective distillation of table-based reasoning ability from llms. arXiv preprint arXiv:2309.13182. find
  35. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  36. Jiachen Zhao, Wenlong Zhao, Andrew Drozdov, Benjamin Rozonoyer, Md Arafat Sultan, Jay-Yoon Lee, Mohit Iyyer, and Andrew McCallum. 2023. Multi-stage collaborative knowledge distillation from large language models. arXiv preprint arXiv:2311.08640. — [arXiv:2311.08640](https://arxiv.org/abs/2311.08640) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/feng-2024-teaching-small-language-models-reasoning/