📄 Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibility

Summary

Code-LLMs perform well on benchmarks that check execution in one direction, but it is unclear whether their forward and backward reasoning about the same program agree. The authors present RoundTripCodeEval (RTCE), which treats code understanding as an inversion problem. It uses four lossless compression algorithms (RLE, LZW, Huffman and AE) with 250 real-world inputs per algorithm, written in Python. Models must carry out four related tasks:

• predict the encoder output from the input,
• predict the decoder output from the compressed data,
• invert the encoder,
• invert the decoder.

Scoring uses exact match without running code, and a sample counts as consistent only if the reconstruction equals the original input. The authors evaluate open-source Code-LLMs under three strategies: prompting without examples, fine-tuning on execution traces in a supervised way, and repeated rounds of self-reflection. None of these approaches closes the gap. Models often solve the forward and backward tasks separately yet fail the full round trip. Both fine-tuning and self-reflection plateau after one revision round, and errors appear even on simple bijections like RLE. This suggests current models lack the coherent internal model of program state needed for reliable bidirectional code reasoning.

Key points

• RTCE contains 1,000 exact-match instances (4 algorithms × 250 inputs). It is execution-free and limited to Python. Code and data are released at github.com/Nickil21/round-trip-code-compression.
• The four tasks are Output Prediction, Input Prediction, and their "with Inversion" variants. Together they form a round-trip check of whether dec(enc(x)) = x.
• SFT used LoRA on QwQ-32B, the strongest zero-shot model:
  • Setup: rank-8 adapters on all linear modules, 3 epochs, effective batch size 32, learning rate 1e-4 with a cosine schedule, and one adapter per algorithm.
  • Gains: forward-task Pass@5 rose to about 80% for RLE, 87.5% for LZW and 78.6% for AE.
• After fine-tuning, the gains were lopsided. Huffman output-side tasks stayed at 0% while its decoder side reached 35%, and the AE input-side scores fell to around 20–31%.
• Compared with earlier benchmarks such as IdentityChain, RTC, CRUXEval, CodeIO and CodeMind, RTCE is the only one that requires exact enc/dec bijection, including explicit inversion.