Elamur Research⌕
☰
home / papers / 2023 / Expression Syntax Information Bottleneck for Math Word Problems

Expression Syntax Information Bottleneck for Math Word Problems

Paper · 2023 · Jing Xiong, Chengming Li, Min Yang et al.

Summary

Neural solvers for math word problems (MWP) often score well by exploiting shallow links between wording and answer templates. For example, they may map any story about travelling between two places to the same speed formula instead of reasoning about what is actually asked. Rather than adding extra features, ESIB (SIGIR '22) removes them. It applies a variational information bottleneck (VIB) to the latent space so that mainly information useful for building the expression syntax tree is kept. Two networks are trained together through mutual learning. Each receives a different representation of the same problem, and both must output the same tree. This pushes them toward shared, syntax-relevant content and away from model-specific redundancy. A self-distillation loss also makes the decoder depend more on latent syntax information, which yields more varied valid solutions. On two large benchmarks, including Math23K, the authors report state-of-the-art accuracy, greater solution diversity and robustness on adversarial and out-of-distribution data. The work matters because it tackles spurious correlation directly, a known weakness of MWP solvers.

Key points

  • Reported accuracy is 85.9% on Math23K and 89.3% in cross-dataset transfer to MAWPS.
  • Solution diversity reaches 13.5%, compared with 12.2% for the NS-Solver baseline.
  • The best compression strength on Math23K is λ = 0.01. Weaker compression keeps spurious features, while stronger compression discards useful semantics.
  • Mutual learning raises training time to about 1.8× and memory to about 1.5×. Inference uses a single network, and the VIB module adds under 2% compute.
  • Code is released at github.com/menik1126/math_ESIB. Acknowledged limitations are the Gaussian priors and the cost of training two networks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations🔬Shenzhen Institutes of Advanced TechnologyShenzhen · China17 works · 2,599 citations🎓Lanzhou UniversityLanzhou · China5 works · 24 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputational Theory and Mathematics · 252 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksSyntax & grammar formalisms · 214 works

Cite

Cite as Jing Xiong, Chengming Li, Min Yang, Xiping Hu and Bin Hu. 2023. Expression Syntax Information Bottleneck for Math Word Problems. arXiv:2310.15664.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.15664 · arXiv 2310.15664

@misc{xiong2023expression,
  title = {{Expression Syntax Information Bottleneck for Math Word Problems}},
  author = {Jing Xiong, Chengming Li, Min Yang, Xiping Hu and Bin Hu},
  year = {2023},
  doi = {10.48550/arxiv.2310.15664},
  eprint = {2310.15664},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.15664}
}

References

40 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alessandro Achille and Stefano Soatto. 2018. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence 40, 12 (2018), 2897–2905. find
  2. Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. 2016. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 (2016). — [arXiv:1612.00410](https://arxiv.org/abs/1612.00410) find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).
  4. Daniel G Bobrow. 1964. Natural language input for a computer problem solving system. (1964). find
  5. Ting-Rui Chiang and Yun-Nung Chen. 2018. Semantically-aligned equation generation for solving and reasoning math word problems. arXiv preprint arXiv:1811.00720 (2018). find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  7. Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. 2020. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017 (2020). — [arXiv:2002.07017](https://arxiv.org/abs/2002.07017) find
  8. Ian Fischer. 2020. The conditional entropy bottleneck. Entropy 22, 9 (2020), 999. find
  9. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).
  10. Zhenya Huang, Qi Liu, Weibo Gao, Jinze Wu, Yu Yin, Hao Wang, and Enhong Chen. 2020. Neural mathematical solver with enhanced formula structure. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1729–1732. find
  11. Artemy Kolchinsky, Brendan D Tracey, and Steven Van Kuyk. 2019. Nonlinear information bottleneck. In Entropy, Vol. 21. 1181. find
  12. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1152–1157. find
  13. Vivek Kumar, Rishabh Maheshwary, and Vikram Pudi. 2021. Adversarial Examples for Evaluating Math Word Problem Solvers. arXiv preprint arXiv:2109.05925 (2021). find
  14. Xiang Lisa Li and Jason Eisner. 2019. Specializing word embeddings (for parsing) by information bottleneck. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. 2744–2754. find
  15. Zhenwen Liang and Xiangliang Zhang. 2021. Solving Math Word Problems with Teacher Supervision. IJCAI (2021). find
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
  17. Ilya Loshchilov and Frank Hutter. 2018. Fixing weight decay regularization in adam. (2018). find
  18. Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2021. Variational information bottleneck for effective low-resource fine-tuning. In International Conference on Learning Representations. find
  19. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 975–984. find
  20. Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan. 2020. CCMI: Classifier based conditional mutual information estimation. In Uncertainty in artificial intelligence. PMLR, 1083–1093. find
  21. Yu Pan, Zeyong Su, Ao Liu, Jingquan Wang, Nannan Li, and Zenglin Xu. 2022. A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks. In ICML (Proceedings of Machine Learning Research, Vol. 162). PMLR, 17238–17257. find
  22. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP Models really able to Solve Simple Math Word Problems? arXiv preprint arXiv:2103.07191 (2021).
  23. Jinghui Qin, Xiaodan Liang, Yining Hong, Jianheng Tang, and Liang Lin. 2021. Neural-symbolic solver for math word problems with auxiliary tasks. arXiv preprint arXiv:2107.01431 (2021). find
  24. Jinghui Qin, Lihui Lin, Xiaodan Liang, Ruimin Zhang, and Liang Lin. 2020. Semantically-aligned universal tree-structured solver for math word problems. arXiv preprint arXiv:2010.06823 (2020). find
  25. Yibin Shen and Cheqing Jin. 2020. Solving math word problems with multi-encoders and multi-decoders. In Proceedings of the 28th International Conference on Computational Linguistics. 2924–2934. find
  26. Naftali Tishby, Fernando C Pereira, and William Bialek. 2000. The information bottleneck method. arXiv preprint physics/0004057 (2000). find
  27. Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. 2018. Translating a math word problem to an expression tree. arXiv preprint arXiv:1811.05632 (2018). find
  28. Lei Wang, Dongxiang Zhang, Lianli Gao, Jingkuan Song, Long Guo, and Heng Tao Shen. 2018. Mathdqn: Solving arithmetic word problems via deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32. find
  29. Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. 2019. Template-based math word problem solvers with recursive neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 7144–7151. find
  30. Maolin Wang, Yu Pan, Xiangli Yang, Guangxi Li, and Zenglin Xu. 2023. Tensor Networks Meet Neural Networks: A Survey. CoRR abs/2302.09019 (2023). — [arXiv:2302.09019](https://arxiv.org/abs/2302.09019) find
  31. Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 845–854. find
  32. Qinzhuo Wu, Qi Zhang, Jinlan Fu, and Xuan-Jing Huang. 2020. A knowledge-aware sequence-to-tree network for math word problem solving. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 7137–7146. find
  33. Qinzhuo Wu, Qi Zhang, Zhongyu Wei, and Xuan-Jing Huang. 2021. Math word problem solving with explicit numerical values. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… find
  34. Zhipeng Xie and Shichao Sun. 2019. A Goal-Driven Tree-Structured Neural Model for Math Word Problems. In IJCAI. 5299–5305. find
  35. Jiong Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yin, Enze Xie, Zhicheng Yang, Qingxing Cao, Haiming Wang, Xiongwei Han, et al. 2023. DQ-LORE: DUAL QUERIES WITH LOW RANK APPROX-IMATION RE-RANKING FOR IN-CONTEXT LEARNING. arXiv preprint arXiv:2310.02954 (2023).
  36. Richard Zanibbi, Behrooz Mansouri, Anurag Agarwal, and Douglas W Oard. 2021. ARQMath: a new benchmark for math-aware CQA and math formula retrieval. In ACM SIGIR Forum, Vol. 54. ACM New York, NY, USA, 1–9. find
  37. Jipeng Zhang, Ka Wei LEE, Ee-Peng Lim, Wei Qin, Lei Wang, Jie Shao, Qianru Sun, et al. 2020. Teacher-student networks with multiple decoders for solving math word problem. (2020). find
  38. Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020. Graph-to-tree learning for solving math word problems. Association for Computational Linguistics. find
  39. Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4320–4328. find
  40. Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape210k: A large-scale and template-rich dataset of math word problems. arXiv preprint arXiv:2009.11506 (2020). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiong-2023-expression-syntax-information-bottleneck-math/