📄 Breaking the Illusion of Reasoning in Polish LLMs: Quality over Quantity of Thought

Summary

It remains unclear why explicit reasoning helps LLMs. One explanation is the content of the reasoning traces. Another is simply the extra tokens of computation. The authors separate these two factors in a Polish setting. They machine-translated about 32K samples of Mixture-of-Thoughts into Polish with DeepSeek-V3-0324 and filtered them, producing MoT-PL. They then built four versions that share identical questions and answers and differ only in the reasoning part:

• Detailed: the original traces.
• Summarized: compressed traces.
• BabyThink: simplistic, child-like traces of roughly the original length.
• Lengthy: traces at least doubled in size.

Each version was used to fine-tune Llama-PLLuM-8B-instruct. The models were tested on Belebele, Aya Collection, MoT-PL-eval and LightR1.

Only the Detailed model beat the non-reasoning baseline on average. Longer traces of similar quality did not help, and neither did weak traces of similar length. Within each model, correct answers tended to use fewer reasoning tokens. The study concludes that clear, relevant reasoning, including room for self-correction, matters more than verbosity. It also contributes a Polish reasoning dataset for building such models.

Key points

• Dataset: MoT-PL has 22,571 training examples (28% math, 17% code, 55% science) and a separate 900-example held-out set, MoT-PL-eval. Filtering used langdetect, BERTScore and blacklisted strings.
• Trace lengths: average reasoning length is about 2,523 tokens for Detailed, 444 for Summarized, 2,661 for BabyThink and 4,884 for Lengthy. Qwen3-235B and Llama-3.3-70B, used as LLM judges, confirmed that BabyThink has the lowest quality.
• Average scores: Detailed 0.540, baseline Llama-PLLuM-8B-instruct 0.494, BabyThink 0.392, Summarized 0.358, Lengthy 0.342. Improving quality at a similar length (Detailed vs. BabyThink) gives a 14.8 pp gain.
• Regressions of the Detailed model: it scores below the baseline on Polish Aya (0.584 vs. 0.656) and LightR1 (0.146 vs. 0.172). The authors attribute this to catastrophic forgetting from training only on MoT-PL.
• Code: on the MoT-PL-eval code subset, the baseline scores best. Fine-tuned models write real code but sometimes miss task constraints, and the judge accepted the baseline's prose descriptions of solutions.