Elamur Research⌕
☰
home / papers / 2024 / C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness

C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness

Paper · 2024 · Yu Kang, Xianghui Sun, Liangyu Chen et al.

Summary

Chain-of-Thought (CoT) reasoning makes large language models more accurate, but the reasoning text is usually far longer than the answer. This raises decoding cost and latency, which is a problem for applications like search and recommendation that only need the final output. Earlier studies found that trimming reasoning steps tends to hurt accuracy, even when the essential content is kept. The authors propose C3oT (Conditioned Compressed Chain-of-Thought), which has three parts. First, GPT-4 condenses each original CoT into a minimal version that still carries the core logic. Second, a single model is fine-tuned on both the long and short versions, with each version marked by a different prefix placed before the instruction. Third, at inference time, the model is given the short-version prefix, so it writes brief reasoning while drawing on what it learned from the detailed chains. Tested on arithmetic (GSM8K, MathQA) and commonsense (ECQA, StrategyQA) benchmarks, the approach cuts generated CoT length by more than half while matching models trained only on full CoT. This makes CoT-trained models more usable in latency-sensitive settings.

Key points

  • Datasets: four benchmarks across two domains: GSM8K and MathQA (math), ECQA and StrategyQA (commonsense).
  • Main result: generated CoT is over 50% shorter, with accuracy comparable to a model trained only on long CoT. The prior compression method Implicit-CoT clearly underperforms explicit CoT.
  • Training idea: the conditioned training borrows the class-conditioned policy from OpenChat. Training only on GPT-4-compressed CoT, without the paired long versions, still degrades performance.
  • Mixed Conditions variant: uses six short-CoT prefixes, one for each training compression rate from 50% to 100%. It beats standard C3oT at every rate and outperforms Long CoT at the 50% rate. The level of compression can be chosen at inference.
  • C3oT Adapt variant: inspired by Orca 2, it assigns each training example the highest compression rate at which the model still answers correctly. This is determined with 5-way splits and 3 random seeds. The resulting model picks its own reasoning length and exceeds the GPT-4-compressed C3oT in both accuracy and compression.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Yu Kang, Xianghui Sun, Liangyu Chen and Wei Zou. 2024. C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness. arXiv:2412.11664.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.11664 · arXiv 2412.11664

@misc{kang2024c3ot,
  title = {{C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness}},
  author = {Yu Kang, Xianghui Sun, Liangyu Chen and Wei Zou},
  year = {2024},
  doi = {10.48550/arxiv.2412.11664},
  eprint = {2412.11664},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.11664}
}

References

40 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Aggarwal, S.; Mandowara, D.; Agrawal, V.; Khandelwal, D.; Singla, P.; and Garg, D. 2021. Explanations for commonsenseqa: New dataset and models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language…
  3. Amini, A.; Gabriel, S.; Lin, P.; Koncel-Kedziorski, R.; Choi, Y.; and Hajishirzi, H. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319.
  4. Bolya, D.; Fu, C.-Y.; Dai, X.; Zhang, P.; Feichtenhofer, C.; and Hoffman, J. 2022. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461. — [arXiv:2210.09461](https://arxiv.org/abs/2210.09461) find
  5. Chase, H. 2022. LangChain. find
  6. Chevalier, A.; Wettig, A.; Ajith, A.; and Chen, D. 2023. Adapting language models to compress contexts. arXiv preprint arXiv:2305.14788.
  7. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  8. Das, R. J.; Ma, L.; and Shen, Z. 2023. Beyond size: How gradients shape pruning decisions in large language models. arXiv preprint arXiv:2311.04902. — [arXiv:2311.04902](https://arxiv.org/abs/2311.04902) find
  9. Deng, Y.; Prasad, K.; Fernandez, R.; Smolensky, P.; Chaudhary, V.; and Shieber, S. 2023. Implicit Chain of Thought Reasoning via Knowledge Distillation. arXiv preprint arXiv:2311.01460. find
  10. Dettmers, T.; Lewis, M.; Belkada, Y.; and Zettlemoyer, L. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318–30332. find
  11. Frantar, E.; and Alistarh, D. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, 10323–10337. PMLR. find
  12. Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2022. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations. find
  13. Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720.
  14. Ge, T.; Hu, J.; Wang, X.; Chen, S.-Q.; and Wei, F. 2023. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945.
  15. Geva, M.; Khashabi, D.; Segal, E.; Khot, T.; Roth, D.; and Berant, J. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9: 346–361. find
  16. Jiang, H.; Wu, Q.; Lin, C.-Y.; Yang, Y.; and Qiu, L. 2023a. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736.
  17. Jiang, H.; Wu, Q.; Luo, X.; Li, D.; Lin, C.-Y.; Yang, Y.; and Qiu, L. 2023b. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839.
  18. Jin, M.; Yu, Q.; Zhao, H.; Hua, W.; Meng, Y.; Zhang, Y.; Du, M.; et al. 2024. The Impact of Reasoning Step Length on Large Language Models. arXiv preprint arXiv:2401.04925.
  19. Kim, S.; Shen, S.; Thorsley, D.; Gholami, A.; Kwon, W.; Hassoun, J.; and Keutzer, K. 2022. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 784– 794. find
  20. Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199– 22213.
  21. Lampinen, A. K.; Dasgupta, I.; Chan, S. C.; Matthewson, K.; Tessler, M. H.; Creswell, A.; McClelland, J. L.; Wang, J. X.; and Hill, F. 2022. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329.
  22. Li, Y.; Dong, B.; Lin, C.; and Guerin, F. 2023. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201.
  23. Ling, W.; Yogatama, D.; Dyer, C.; and Blunsom, P. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146. find
  24. Liu, T.; Chen, Z.; Liu, Z.; Tian, M.; and Luo, W. 2024. Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding. arXiv preprint arXiv:2409.08561. — [arXiv:2409.08561](https://arxiv.org/abs/2409.08561) find
  25. Maini, P.; Seto, S.; Bai, H.; Grangier, D.; Zhang, Y.; and Jaitly, N. 2024. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380. Marasović, A.; Beltagy, I.; Downey, D.; and Peters, M. E. 2021. Few-shot self-rationalization with natural lang…
  26. Merrill, W.; and Sabharwal, A. 2023. The Expresssive Power of Transformers with Chain of Thought. arXiv preprint arXiv:2310.07923. find
  27. Mitra, A.; Del Corro, L.; Mahajan, S.; Codas, A.; Simoes, C.; Agarwal, S.; Chen, X.; Razdaibiedina, A.; Jones, E.; Aggarwal, K.; et al. 2023. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11045.
  28. Modarressi, A.; Mohebbi, H.; and Pilehvar, M. T. 2022. Adapler: Speeding up inference by adaptive length reduction. arXiv preprint arXiv:2203.08991. find
  29. Mu, J.; Li, X.; and Goodman, N. 2024. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36.
  30. Mukherjee, S.; Mitra, A.; Jawahar, G.; Agarwal, S.; Palangi, H.; and Awadallah, A. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707.
  31. Nye, M.; Andreassen, A. J.; Gur-Ari, G.; Michalewski, H.; Austin, J.; Bieber, D.; Dohan, D.; Lewkowycz, A.; Bosma, M.; Luan, D.; et al. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114.
  32. Pan, Z.; Wu, Q.; Jiang, H.; Xia, M.; Luo, X.; Zhang, J.; Lin, Q.; Rühle, V.; Yang, Y.; Lin, C.-Y.; et al. 2024. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. arXiv preprint arXiv:2403.12968.
  33. Sun, M.; Liu, Z.; Bair, A.; and Kolter, J. Z. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695. find
  34. Suzgun, M.; Scales, N.; Schärli, N.; Gehrmann, S.; Tay, Y.; Chung, H. W.; Chowdhery, A.; Le, Q. V.; Chi, E. H.; Zhou, D.; et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
  35. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  36. Wang, G.; Cheng, S.; Zhan, X.; Li, X.; Song, S.; and Liu, Y. 2023. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235.
  37. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837.
  38. Wingate, D.; Shoeybi, M.; and Sorensen, T. 2022. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. arXiv preprint arXiv:2210.03162. find
  39. Xiao, G.; Lin, J.; Seznec, M.; Wu, H.; Demouth, J.; and Han, S. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, 38087–38099. PMLR. find
  40. Zhang, L.; Zhang, Y.; Ren, K.; Li, D.; and Yang, Y. 2023. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks. arXiv preprint arXiv:2304.14979.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kang-2024-c3ot-generating-shorter-chain-thought/