Elamur Research⌕
☰
home / papers / 2019 / Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

Paper · 2019 · Daniel Andor, Luheng He, Kenton Lee et al.

Summary

Neural reading comprehension systems are good at extracting answer spans but weak at arithmetic. On DROP, many answers must be computed instead of copied from the passage. The authors address this by letting a BERT-based reader choose a small executable program rather than generate the number itself. The program space includes:

  • literals (yes/no/unknown, digits 0–9);
  • text spans;
  • arithmetic over numbers found in the passage (Sum, Diff, Diff100, and later Mul and Div);
  • shallow two-step compositions (Sum3, Merge).

Each candidate is scored from BERT token representations, and an external calculator executes the best one. Training marginalizes over all oracle derivations that yield the gold answer, and pruning keeps the composed operations tractable. The approach gains over 33 absolute points over the previous best DROP system, NAQANet. It also learns new operations from few examples on the Illinois math word problems. The work matters because it moves computation out of the network and makes predictions partly interpretable. The operation inventory can also grow without redesigning the architecture.

Key points

  • DROP results: NAQANet reaches 47.77 test F1. The model pre-trained on CoQA and fine-tuned on DROP reaches 80.47 test F1, and a 6-model ensemble reaches 81.78. The oracle ceiling with all operations is 93.01 dev F1.
  • Ablations: Diff100 adds about 9 points and Sum3 about 1.1. Merge helps multi-span answers, but they stay under-predicted because single spans outnumber them about 11:1.
  • CoQA and forgetting: Fine-tuning on DROP alone drops CoQA dev F1 to 52.2. Joint training keeps both: 82.2 F1 on CoQA and 81.1 F1 on DROP.
  • Illinois math word problems: Accuracy is 48.6 for the basic model, 74.0 after adding Mul/Div, and 83.2 when DROP data is also used. This exceeds prior systems at 73.3–80.1.
  • Training details: 36% of answerable examples have several valid derivations, and the model learns to resolve this ambiguity during training. Pruning keeps the top 128 Span and Sum candidates before composing Merge and Sum3.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 487 works

Cite

Cite as Daniel Andor, Luheng He, Kenton Lee and Emily Pitler. 2019. Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1609 · arXiv 1909.00109

@inproceedings{andor2019giving,
  title = {{Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension}},
  author = {Daniel Andor, Luheng He, Kenton Lee and Emily Pitler},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1609},
  eprint = {1909.00109},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1609}
}

References

13 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
  2. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL.
  3. Kenton Lee, Tom Kwiatkowski, Ankur P. Parikh, and Dipanjan Das. 2017. Learning recurrent span representations for extractive question answering. CoRR, abs/1611.01436. — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) find
  4. Chao-Chun Liang, Kuang-Yi Hsu, Chien-Tsung Huang, Chung-Min Li, Shen-Yu Miao, and Keh-Yih Su. 2016. A tag-based statistical english math word problem solver with understanding, reasoning and explanation. In IJCAI. find
  5. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you dont know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789.
  6. Abhilasha Ravichander, Aakanksha Naik, Carolyn Penstein Rosé, and Eduard H. Hovy. 2019. Equate: A benchmark evaluation framework for quantitative reasoning in natural language inference. CoRR, abs/1901.03735.
  7. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. CoQA: A conversational question answering challenge. CoRR, abs/1808.07042.
  8. Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In EMNLP.
  9. Subhro Roy, Tim Vieira, and Dan Roth. 2015. Reasoning about quantities in natural language. Transactions of the Association for Computational Linguistics, 3:1–13. find
  10. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. CoRR, abs/1904.01557.
  11. Lei Wang, Dongxiang Zhang, Lianli Gao, Jingkuan Song, Long Guo, and Heng Tao Shen. 2018. Math-DQN: Solving arithmetic word problems via deep reinforcement learning. In AAAI. find
  12. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. Proceedings of ICLR.
  13. Dongxiang Zhang, Lei Wang, Nuo Xu, Bing Tian Dai, and Heng Tao Shen. 2018. The gap of semantic parsing: A survey on automatic math word problem solvers. IEEE transactions on pattern analysis and machine intelligence. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/andor-2019-giving-bert-calculator-finding-operations/