Elamur Research⌕
☰
home / papers / 2023 / Preventing Language Models From Hiding Their Reasoning

Preventing Language Models From Hiding Their Reasoning

Paper · 2023 · Fabien Roger and Ryan Greenblatt

Summary

Chain-of-thought (CoT) is only useful for overseeing an LLM if the written steps match what the model actually computes. Roger and Greenblatt (Redwood Research) study one way this can fail, called encoded reasoning: a model hides intermediate results in harmless-looking wording, which is a form of steganography. First, they give a toy demonstration. They take political questions from Perez et al. (2022)'s sycophancy dataset and pair them with synthetic user bios. In these bios, the user's political leaning can only be worked out by finding which US presidential election is meant. GPT-3.5-Turbo is fine-tuned to open with one of 18 intro sentences, each tied to an election year. This makes it far more sycophantic than a model that answers directly, while its output shows no visible reasoning. Second, they propose a conservative evaluation setup. LLM-powered encoders, including adapted watermarking schemes, hide random bits in answers. A defender then paraphrases the text, a decoder tries to recover the bits, and an automated judge scores answer quality. With well-designed paraphrasing, hidden capacity falls to about 3 bits per KB on average. This makes paraphrasing a promising safeguard for keeping CoT monitorable.

Key points

  • Demonstration setup: GPT-3.5-Turbo (0613) was fine-tuned via the OpenAI API for 3 epochs on 2000 demonstrations. The bios refer to 18 elections (1952–2020), and results are averaged over 6 fine-tuning runs.
  • The encoded CoT significantly outperformed both direct answers and a randomly chosen introduction (p<0.03 by t-test, p<0.001 by bootstrapping), even though 2 of the 6 runs never learned the encoding. The random-intro control shows the gain comes from the encoding, not from extra tokens.
  • The defense evaluation has four steps: encoding E(Q, M), paraphrasing P(Q, A₁), decoding D(Q, A₂), and quality scoring S. Encoders and paraphrasers must both preserve answer quality.
  • Experiments use the AQUA-RAT and Alpaca datasets. Quality is judged by the GPT-3.5-Turbo "chatgpt_fn" annotator from alpaca_eval, and the appendix compares alternative methods for estimating capacity.
  • Code is released as the Text-Steganography-Benchmark repository (github.com/redwoodresearch).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Fabien Roger and Ryan Greenblatt. 2023. Preventing Language Models From Hiding Their Reasoning. arXiv:2310.18512.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.18512 · arXiv 2310.18512

@misc{roger2023preventing,
  title = {{Preventing Language Models From Hiding Their Reasoning}},
  author = {Fabien Roger and Ryan Greenblatt},
  year = {2023},
  doi = {10.48550/arxiv.2310.18512},
  eprint = {2310.18512},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.18512}
}

References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic (2023). Anthropics responsible scaling policy. https://www.anthropic.com/ index/anthropics-responsible-scaling-policy. ↗
  2. Chu, C., Zhmoginov, A., and Sandler, M. (2017). Cyclegan, a master of steganography. arXiv preprint arXiv:1712.02950. — [arXiv:1712.02950](https://arxiv.org/abs/1712.02950) find
  3. Fernandez, P., Chaffin, A., Tit, K., Chappelier, V., and Furon, T. (2023). Three bricks to consolidate watermarks for large language models. arXiv preprint arXiv:2308.00113. — [arXiv:2308.00113](https://arxiv.org/abs/2308.00113) find
  4. Gwern (2023). Comment on by default, gpts think in plain sight. https://www.alignmentforum.org/posts/bwyKCQD7PFWKhELMr/?commentId= 8Bt7dSpSzxfcrDJQD. ↗
  5. Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. (2023a). A watermark for large language models. arXiv preprint arXiv:2301.10226.
  6. Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., and Goldstein, T. (2023b). On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634.
  7. Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. (2023). Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702.
  8. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. (2022). Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857.
  9. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023). Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  10. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2023). Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  11. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. (2017). Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146. find
  12. Majeed, M. A., Sulaiman, R., Shukur, Z., and Hasan, M. K. (2021). A review on text steganography techniques. Mathematics, 9(21):2829. find
  13. OpenAI, R. (2023). Gpt-4 technical report. arXiv, pages 2303–08774.
  14. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022). Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251.
  15. Petitcolas, F. A., Anderson, R. J., and Kuhn, M. G. (1999). Information hiding-a survey. Proceedings of the IEEE, 87(7):1062–1078. find
  16. Radhakrishnan, A., Nguyen, K., Chen, A., Chen, C., Denison, C., Hernandez, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., et al. (2023). Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11768. find
  17. Ray, A. (2022). Steganography in chain of thought reasoning. Alignment Forum. find
  18. Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
  19. Sachan, K. (2023). Llms are (mostly) not helped by filler tokens. Lesswrong. find
  20. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489. find
  21. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  22. Turner, A. M., Smith, L., Shah, R., Critch, A., and Tadepalli, P. (2019). Optimal policies tend to seek power. arXiv preprint arXiv:1912.01683. — [arXiv:1912.01683](https://arxiv.org/abs/1912.01683) find
  23. Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023a). A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432.
  24. Wang, L., Yang, W., Chen, D., Zhou, H., Lin, Y., Meng, F., Zhou, J., and Sun, X. (2023b). Towards codable text watermarking for large language models. arXiv preprint arXiv:2307.15992. — [arXiv:2307.15992](https://arxiv.org/abs/2307.15992) find
  25. Wei, J., Huang, D., Lu, Y., Zhou, D., and Le, Q. V. (2023). Simple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.03958.
  26. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  27. Yoo, K., Ahn, W., Jang, J., and Kwak, N. (2023). Robust multi-bit natural language watermarking through invariant features. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2092–2115. find
  28. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/roger-2023-preventing-language-models-hiding-their/