Elamur Research⌕
☰
home / papers / 2024 / MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems

MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems

Paper · 2024 · Bin Lei, Yi Zhang, Shan Zuo et al.

Summary

Large language models such as GPT-4 still struggle with hard math problems that require long chains of deduction. Structured prompting schemes like Tree of Thought and Graph of Thought help, but they need prompts tailored to each task, so they generalize poorly. The authors propose MACM, a task-agnostic prompting framework that splits work among three GPT-based roles:

  • Thinker: restates the problem as a list of known conditions plus a goal, proposes new facts, and plans the solution.
  • Judge: verifies each proposed fact and decides whether the gathered facts are sufficient.
  • Executor: carries out the computations, using code.

The loop of mining and checking new conditions repeats until the goal is reachable or an iteration cap is hit. MACM was tested on the MATH benchmark, the 24-point game and number sorting. It clearly improved GPT-4 Turbo on MATH, most of all on the hardest Level 5 problems, and it also beat ToT and GoT on their own tasks. This matters because a single reusable prompt set gives large gains without any fine-tuning. The cost is many more model calls.

Key points

  • On MATH Level 5, GPT-4 Turbo accuracy rose from 54.68% to 76.73% with MACM. Overall MATH accuracy improved by 15.14 points, and MACM scored 7.8 points above SC-CoT.
  • Using GPT-4, MACM was 17 points more accurate than ToT on the 24-point game. On sorting 64 numbers with GPT-3.5, it reached 92% versus about 89% for GoT, where the GoT figure was estimated from a graph.
  • MACM improved all seven Level 5 categories. The biggest gains came in the weakest areas: Geometry rose from 18.18% by 31.82 points, and Intermediate Algebra (baseline 34.04%) also improved substantially.
  • The authors tested 200 MATH questions that GPT-4 Turbo originally got wrong. With a large query budget, MACM fixed nearly 90% of them, while I-O, CoT and SC-CoT plateaued around 20%.
  • Condition mining is capped at five rounds; if the goal is still unmet, the problem is treated as unsolvable. The main limitations are higher inference time and cost and continued weakness on geometry. Code is available at github.com/bin123apple/MACM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Bin Lei, Yi Zhang, Shan Zuo, Ali Payani and Caiwen Ding. 2024. MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems. arXiv:2404.04735.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.04735 · arXiv 2404.04735

@misc{lei2024macm,
  title = {{MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems}},
  author = {Bin Lei, Yi Zhang, Shan Zuo, Ali Payani and Caiwen Ding},
  year = {2024},
  doi = {10.48550/arxiv.2404.04735},
  eprint = {2404.04735},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.04735}
}

References

14 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
  2. Gpt-4 turbo. https://help.openai.com/en/ articles/8555510-gpt-4-turbo, 2024. ↗
  3. Abramski, K., Citraro, S., Lombardi, L., Rossetti, G., and Stella, M. Cognitive network science reveals bias in gpt-3, gpt-3.5 turbo, and gpt-4 mirroring math anxiety in high-school students. Big Data and Cognitive Computing, 7 (3):124, 2023. find
  4. Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gianinazzi, L., Gajda, J., Lehmann, T., Podstawski, M., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023. find
  5. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
  6. Lei, B., Liao, C., Ding, C., et al. Boosting logical reasoning in large language models through a new framework: The graph of thought. arXiv preprint arXiv:2308.08614, 2023. — [arXiv:2308.08614](https://arxiv.org/abs/2308.08614) find
  7. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023.
  8. Nori, H., King, N., McKinney, S. M., Carignan, D., and Horvitz, E. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023.
  9. OpenAI. Gpt-4. https://openai.com/research/ gpt-4, 2024. Accessed: 2024-02-01. ↗
  10. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
  11. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
  12. White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., and Schmidt, D. C. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382, 2023.
  13. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models, 2023. find
  14. A. Why is the generalizability of ToT and GoT limited This section demonstrates specific examples of using ToT and GoT to further illustrate why their generalizability is limited. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lei-2024-macm-utilizing-multi-agent-system/