Elamur Research⌕
☰
home / papers / 2024 / Compressed Chain of Thought: Efficient Reasoning Through Dense Representations

Compressed Chain of Thought: Efficient Reasoning Through Dense Representations

Paper · 2024 · Jeffrey Cheng and Benjamin Van Durme

Summary

Chain-of-thought reasoning makes language models more accurate, but writing out long explicit reasoning traces slows generation considerably. Cheng and Van Durme propose Compressed Chain-of-Thought (CCoT), which replaces the full textual reasoning chain with a shorter, variable-length sequence of continuous "contemplation tokens." Each token is a dense vector meant to approximate a compressed version of a real reasoning trace. During training, the model learns, via teacher forcing, to reproduce a selected subset of the gold hidden states of full reasoning chains. Off-the-shelf decoder LLMs are adapted with LoRA, using only the roughly 9,000 GSM8K training instances. At inference, an intermediate-layer hidden state is fed back as the next input instead of a sampled discrete token. A compression ratio controls how many such tokens are produced, so users can trade accuracy against speed. On GSM8K, CCoT clearly beats both a no-reasoning baseline and fixed-length PAUSE tokens while adding little latency. Because the tokens are grounded in text, they may eventually allow the hidden reasoning to be decoded and inspected.

Key points

  • On GSM8K (exact match, no in-context examples, single Nvidia A100):
    • With no contemplation tokens, accuracy is 0.089 at 0.33 s.
    • CCoT at 10x compression reaches 0.179 at 0.78 s.
    • CCoT at 20x compression reaches 0.151 at 0.49 s.
    • Full reasoning chains reach 0.315 but take 8.10 s.
  • PAUSE tokens (Goyal et al., 2024) decode slightly faster but barely help: 0.092 at 20x and 0.099 at 10x. The authors attribute this to GSM8K needing sequential rather than parallel computation.
  • Taking hidden states from a middle layer (l ≈ L/2) worked best. Layers near the input or output could not be learned well. Accuracy stops improving at around r = 0.2.
  • Compared with COCONUT (Hao et al., 2024), CCoT adapts a 7B model rather than a 1.5B model. It uses about 9k training examples instead of about 400k augmented ones, and its tokens are grounded in text.
  • In a theoretical analysis, the authors argue that generating contemplation tokens autoregressively adds computational depth, whereas parallel filler or pause tokens add only width.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Jeffrey Cheng and Benjamin Van Durme. 2024. Compressed Chain of Thought: Efficient Reasoning Through Dense Representations. arXiv:2412.13171.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.13171 · arXiv 2412.13171

@misc{cheng2024compressed,
  title = {{Compressed Chain of Thought: Efficient Reasoning Through Dense Representations}},
  author = {Jeffrey Cheng and Benjamin {Van Durme}},
  year = {2024},
  doi = {10.48550/arxiv.2412.13171},
  eprint = {2412.13171},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.13171}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Burtsev, M. S., Kuratov, Y., Peganov, A., and Sapunov, G. V. Memory transformer, 2021. URL https://arxiv.org/abs/2006.11527. — [arXiv:2006.11527](https://arxiv.org/abs/2006.11527) ↗
  2. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. ↗
  3. Deng, Y., Prasad, K., Fernandez, R., Smolensky, P., Chaudhary, V., and Shieber, S. Implicit chain of thought reasoning via knowledge distillation, 2023. URL https://arxiv.org/abs/2311.01460. ↗
  4. Deng, Y., Choi, Y., and Shieber, S. From explicit cot to implicit cot: Learning to internalize cot step by step, 2024. URL https://arxiv.org/abs/2405.14838. ↗
  5. Ge, T., Hu, J., Wang, L., Wang, X., Chen, S.-Q., and Wei, F. In-context autoencoder for context compression in a large language model, 2024. URL https://arxiv.org/abs/2307.06945. ↗
  6. Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V. Think before you speak: Training language models with pause tokens, 2024. URL https://arxiv.org/abs/2310.02226. ↗
  7. Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., and Tian, Y. Training large language models to reason in a continuous latent space, 2024. URL https://arxiv.org/abs/2412.06769. ↗
  8. Herel, D. and Mikolov, T. Thinking tokens for language modeling, 2024. URL https://arxiv.org/abs/2405.08644. — [arXiv:2405.08644](https://arxiv.org/abs/2405.08644) ↗
  9. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685. ↗
  10. Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. Llmlingua: Compressing prompts for accelerated inference of large language models, 2023. URL https://arxiv.org/abs/2310.05736. ↗
  11. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners, 2023. URL https://arxiv.org/abs/2205.11916. ↗
  12. Kou, S., Hu, L., He, Z., Deng, Z., and Zhang, H. Cllms: Consistency large language models, 2024. URL https://arxiv.org/abs/2403.00835. — [arXiv:2403.00835](https://arxiv.org/abs/2403.00835) ↗
  13. Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., Nguyen, K., Cheng, N., Joseph, N., Schiefer, N., Rausch, O., Larson, R., McCandlish, S., Kundu, S., Kadavath, S., Yang, S., Henighan, T., Maxwell, T., Te… — Measuring Faithfulness in Chain-of-Thought Reasoning
  14. Liu, H., Sferrazza, C., and Abbeel, P. Chain of hindsight aligns language models with feedback, 2023. URL https://arxiv.org/abs/2302.02676. ↗
  15. Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., Maire, M., Hoffmann, H., Holtzman, A., and Jiang, J. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, ACM SI… — [doi:10.1145/3651890.3672274](https://doi.org/10.1145/3651890.3672274) find
  16. Ning, X., Lin, Z., Zhou, Z., Wang, Z., Yang, H., and Wang, Y. Skeleton-of-thought: Prompting llms for efficient parallel generation, 2024. URL https://arxiv.org/abs/2307.15337. ↗
  17. Pfau, J., Merrill, W., and Bowman, S. R. Let’s think dot by dot: Hidden computation in transformer language models, 2024. URL https://arxiv.org/abs/2404.15758. ↗
  18. Puerto, H., Chubakov, T., Zhu, X., Madabushi, H. T., and Gurevych, I. Fine-tuning with divergent chains of thought boosts reasoning through self-correction in language models, 2024. URL https://arxiv.org/abs/2407.03181. — [arXiv:2407.03181](https://arxiv.org/abs/2407.03181) ↗
  19. Qin, G., Rosset, C., Chau, E. C., Rao, N., and Durme, B. V. Dodo: Dynamic contextual compression for decoder-only lms, 2024. URL https://arxiv.org/abs/2310.02409. ↗
  20. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  21. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762. ↗
  22. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903. ↗
  23. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL https://arxiv.org/abs/2305.10601. ↗
  24. Zhang, H., Liu, Z., Zhao, Y., Zheng, J., Zhuang, C., Gu, J., and Chen, G. Fast chain-of-thought: A glance of future from parallel decoding leads to answers faster, 2024. URL https://arxiv.org/abs/2311.08263. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheng-2024-compressed-chain-thought-efficient-reasoning/