Elamur Research⌕
☰
home / papers / 2024 / TracrBench: Generating Interpretability Testbeds with Large Language Models

TracrBench: Generating Interpretability Testbeds with Large Language Models

Paper · 2024 · Hannes Thurnherr and Jérémy Scheurer

Summary

Testing new mechanistic interpretability methods is difficult because few transformers have internals that are fully known. Tracr can compile programs written in RASP into transformer weights whose functional roles are known by construction. However, writing RASP by hand is slow and unintuitive, and validating methods requires many such models. The authors propose using LLMs to write the RASP programs. They release TracrBench, a set of 121 sequence-to-sequence algorithms implemented in RASP and compiled with Tracr. Some programs were LLM-generated and others were written by hand when the models failed. All were checked by humans. Correctness is checked by a five-stage pipeline that covers compilation, output checks against a Python reference, the Tracr validator, weight compilation, and testing of the compiled transformer. Benchmarking frontier models shows that the task remains difficult. Successes cluster on short, simple programs, while the same models handle Python versions of the same algorithms almost perfectly. The resource gives the community ready-made ground-truth models for testing, though not developing, interpretability techniques.

Key points

  • Dataset: TracrBench contains 121 RASP programs with compiled weights, released as PyTorch models in TransformerLens.
    • The programs were split into 20 prompt examples and 101 test programs.
    • The largest compiled model has about 2.7M parameters.
    • Difficulty is measured by the number of RASP function calls, which range from about 3 to 43.
  • Construction: gpt-4-0125-preview initially produced 49 correct programs. The remaining ones were written manually.
  • Verification: Each program was tested on 1,000 input-output pairs, using inputs of length 1–10.
  • Best results: With a 20-shot chain-of-thought prompt and best-of-5 sampling, gpt-4-turbo-2024-04-09 solved 57 of 101 programs (about 56%). Its difficulty-weighted score was 0.29, while gpt-4o scored highest at 0.31.
  • Other models and caveats:
    • claude-3-opus passed 46% of programs and claude-3-5-sonnet passed 45%, with weighted scores of 0.23 and 0.27.
    • gpt-4-turbo reached 96% on the equivalent Python tasks.
    • Tracr weights are sparser than weights produced by training, which is why the benchmark is suited to testing rather than developing methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksSoftware · 150 works

🧭 Topics 1Research subjects and languages the work is about.Interpretability · 156 works

Cite

Cite as Hannes Thurnherr and Jérémy Scheurer. 2024. TracrBench: Generating Interpretability Testbeds with Large Language Models. arXiv:2409.13714.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.13714 · arXiv 2409.13714

@misc{thurnherr2024tracrbench,
  title = {{TracrBench: Generating Interpretability Testbeds with Large Language Models}},
  author = {Hannes Thurnherr and Jérémy Scheurer},
  year = {2024},
  doi = {10.48550/arxiv.2409.13714},
  eprint = {2409.13714},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.13714}
}

References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Anthropic. The claude 3 model family: Opus, sonnet, haiku. Preprint, 2024. URL https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf. ↗
  3. Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/ neuron-explainer/paper/index.html, 2023. ↗
  4. Black, S., Sharkey, L., Grinsztajn, L., Winsor, E., Braun, D., Merizian, J., Parker, K., Guevara, C. R., Millidge, B., Alfour, G., et al. Interpreting neural networks through the polytope lens. arXiv preprint arXiv:2211.12312, 2022. \#\# 7. Author Contributions Hannes Thurnherr executed the whole proj… — [arXiv:2211.12312](https://arxiv.org/abs/2211.12312) find
  5. Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, pp. 2, 2023. find
  6. Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., Filar, B., et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228, 2018. find
  7. Casper, S. Eis vii: A challenge for mechanists, 2020. URL https://www.lesswrong.com/posts/KSHqLzQscwJnv44T8/ eis-vii-a-challenge-for-mechanists. Accessed: 2024-05-23. ↗
  8. Casper, S., Bu, T., Li, Y., Li, J., Zhang, K., Hariharan, K., and Hadfield-Menell, D. Red teaming deep neural networks with feature synthesis tools. Advances in Neural Information Processing Systems, 36:80470–80516, 2023. find
  9. Casper, S., Yun, J., Baek, J., Jung, Y., Kim, M., Kwon, K., Park, S., Moore, H., Shriver, D., Connor, M., et al. The satml’24 cnn interpretability competition: New innovations for concept-level interpretability. arXiv preprint arXiv:2404.02949, 2024. — [arXiv:2404.02949](https://arxiv.org/abs/2404.02949) find
  10. Conmy, A., Mavor-Parker, A., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352, 2023. find
  11. Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., and Mordvintsev, A. The building blocks of interpretability. Distill, 3(3):e10, 2018. find
  12. Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. find
  13. Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022a.
  14. Huang, J., Wu, Z., Potts, C., Geva, M., and Geiger, A. Ravel: Evaluating interpretability methods on disentangling language model representations. arXiv preprint arXiv:2402.17700, 2024. find
  15. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022b.
  16. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566, 2024.
  17. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
  18. Imambi, S., Prakash, K. B., and Kanagachidambaresan, G. Pytorch. Programming with TensorFlow: Solution for Edge Computing Applications, pp. 87–104, 2021. find
  19. Scheurer, J., Balesni, M., and Hobbhahn, M. Technical report: Large language models can strategically deceive their users when put under pressure. arXiv preprint arXiv:2311.07590, 2023. — [arXiv:2311.07590](https://arxiv.org/abs/2311.07590) find
  20. Langosco, L., Baker, W., Alex, N., Quarel, D. J., Bradley, H., and Krueger, D. Towards meta-models for automated interpretability. Manuscript in preparation, 2024. find
  21. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324, 2023.
  22. Li, D. and Murr, L. Humaneval on latest gpt models–2024. arXiv preprint arXiv:2402.14852, 2024. — [arXiv:2402.14852](https://arxiv.org/abs/2402.14852) find
  23. Lindner, D., Kramár, J., Farquhar, S., Rahtz, M., McGrath, T., and Mikulik, V. Tracr: Compiled transformers as a laboratory for interpretability. Advances in Neural Information Processing Systems, 36, 2024. find
  24. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling mon… find
  25. Mazeika, M., Hendrycks, D., Li, H., Xu, X., Hough, S., Zou, A., Rajabi, A., Yao, Q., Wang, Z., Tian, J., et al. The trojan detection challenge. In NeurIPS 2022 Competition Track, pp. 279–291. find
  26. Michaud, E. J., Liao, I., Lad, V., Liu, Z., Mudide, A., Loughridge, C., Guo, Z. C., Kheirkhah, T. R., Vukelić, M., and Tegmark, M. Opening the ai black box: program synthesis via mechanistic interpretability. arXiv preprint arXiv:2402.05110, 2024. — [arXiv:2402.05110](https://arxiv.org/abs/2402.05110) find
  27. Thurnherr, H. and Riesen, K. Neural decompiling of tracr transformers. Manuscript in preparation, 2024. find
  28. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
  29. Nanda, N. and Bloom, J. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens, 2022. ↗
  30. Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.
  31. Weiss, G., Goldberg, Y., and Yahav, E. Thinking like transformers. In International Conference on Machine Learning, pp. 11080–11090. PMLR, 2021. find
  32. Ngo, R., Chan, L., and Mindermann, S. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thurnherr-2024-tracrbench-generating-interpretability-testbeds/