📄 Guided by the Plan: Enhancing Faithful Autoregressive Text-to-Audio Generation with Guided Decoding

Summary

Autoregressive (AR) text-to-audio generators produce temporally coherent sound. However, they often miss or misplace parts of complex prompts that describe several sound events. Diffusion models handle this better because they see the whole timeline at once. The authors probe the AR model Siren and find that its earliest tokens already predict global properties of the finished audio, such as how many events it contains and which object makes the sound. They call this implicit planning.

To exploit it, they train Plan-Critic, a small auxiliary network that estimates the final CLAP score from a partial token sequence. Its training objective borrows from Generalized Advantage Estimation (GAE). Only the last step receives the true CLAP score, and temporal consistency losses carry that signal back to earlier steps. At inference, many candidate prefixes are scored, weak ones are discarded, and compute is spent finishing the promising ones.

At the same token budget as best-of-N decoding, CLAP rises from 26.67 to 36.47, which the authors report as a new AR state of the art. The result shows that causal models can achieve better global alignment through guided decoding.

Key points

• Probing setup: MLP probes read penultimate-layer states of the first 32 tokens out of 288.
  • Event-count regression on the test set reaches Kendall 0.72, Spearman 0.86 and Pearson 0.85.
  • Classification over 10 sound-object categories reaches 81.41% test accuracy.
• Probe labels: event counts and category captions were produced with an MLLM/LLM (Comanici et al., 2025). The data came from 5,000 prompts for event count and 2,000 prompts per object.
• Main result: Plan-Critic-guided sampling lifts CLAP from 26.67 (AR baseline) to 36.47, compared with 31.59 for diffusion models. The cost is 4,608 tokens, the same as the baseline best-of-N.
• Ablations:
  • A GAE supervision interval of 32 is the chosen setting. An interval of 64 scores slightly higher (CLAP 37.34) but nearly doubles the cost.
  • Applying the critic early (Prefix-32) works best among the critic-timing options.
  • Sparse supervision and random pruning both perform far worse.
• User study: 30 domain experts made more than 18,000 pairwise A/B judgments against Siren, AudioX and GenAU.
• Code: to be released at github.com/wjc2830/Siren.