Elamur Research⌕
☰
home / papers / 2024 / Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions

Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions

Paper · 2024 · Yu Zhao, Huifeng Yin, Bo Zeng et al.

Summary

Marco-o1, from Alibaba's MarcoPolo Team, is an early open attempt to reproduce the behavior of OpenAI's o1-style large reasoning models (LRM). It also asks whether such reasoning extends to tasks without clear correct answers or easily measured rewards. The authors fully fine-tune Qwen2-7B-Instruct on about 60K examples: a filtered Open-O1 CoT dataset, their own MCTS-generated CoT data, and instruction-following data. At inference they add Monte Carlo Tree Search. Each node is a partial reasoning state, and a path is scored by the model's own confidence: a softmax over the top-5 token log probabilities, averaged across the rollout. They compare whole reasoning steps with smaller 64- or 32-token "mini-steps" as search actions and add a self-reflection prompt. On the English and Chinese MGSM subsets, the MCTS variants outperform both the base model and the CoT-only model, though no single granularity wins in both languages. Qualitative examples suggest better handling of slang in Chinese–English translation. The work is useful as a transparent sketch of an o1-like pipeline, and the authors explicitly label it a work in progress.

Key points

  • Training data: 60,266 SFT samples in total.
    • 45,125 from the filtered Open-O1 CoT dataset.
    • 10,000 from the synthetic Marco-o1 CoT dataset, generated via MCTS.
    • 5,141 from the Marco Instruction Dataset.
  • Accuracy (Test@1):
    • MGSM-En: Qwen2-7B-Instruct scores 84.00%, Marco-o1-CoT 85.60%, and the best result is 90.40% (MCTS, step as action).
    • MGSM-Zh: Qwen2-7B-Instruct scores 76.80%. Marco-o1-CoT drops to 71.20%, which the authors attribute to English-only CoT data. The best result is 82.40% (MCTS, 32-token mini-step).
  • Test@8 / Test@32 results: with multiple attempts, all fine-tuned variants reach roughly 93–99.6%. MCTS gives its clearest advantage at Test@1.
  • Reflection prompt: appending a "rethink from scratch" style prompt reportedly fixes about half of the hard problems the model initially got wrong.
  • Limitations and future work: using confidence as the reward makes search results noticeably random. Planned work includes ORM/PRM reward models and RL. The authors also claim the first study of LRMs on machine translation, comparing against Google Translate in case studies only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China201 works · 17,173 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo and Kaifu Zhang. 2024. Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions. arXiv:2411.14405.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.14405 · arXiv 2411.14405

@misc{zhao2024marco,
  title = {{Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions}},
  author = {Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo and Kaifu Zhang},
  year = {2024},
  doi = {10.48550/arxiv.2411.14405},
  eprint = {2411.14405},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.14405}
}

References

7 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. ↗
  2. MGSM-En Test@1 Acc. Test@8 Acc. Test@32 Acc. 84.00% 85.60% 90.40% 88.40% 87.60% 89.60% 97.60% 99.20% 98.40% 98.80% 96.00% 99.20% 99.20% 99.60% 99.20% MGSM-Zh Test@1 Acc. Test@8 Acc. Test@32 Acc. 76.80% 71.20% 80.00% 80.40% 82.40% 80.80% 93.60% 93.60% 92.80% 93.20% 92.40% 96.40% 96.00% 95.20% 96.80%… find
  3. J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022.
  4. Z. Li, B. Peng, P. He, M. Galley, J. Gao, and X. Yan. Guiding large language models via directional stimulus prompting. Advances in Neural Information Processing Systems, 36, 2024. find
  5. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.
  6. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024. find
  7. OpenAI. Learning to reason with llms. learning-to-reason-with-llms/, 2024. [Accessed 19-09-2024]. https://openai.com/index/ ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2024-marco-o1-towards-open-reasoning/