Elamur Research⌕
☰
home / papers / 2022 / Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval

Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval

Paper · 2022 · Wei Zhong, Jheng-Hong Yang, Yuqing Xie et al.

Summary

Math Information Retrieval (MIR) is difficult because formulas are highly structured. The same concept can appear in very different notation, and formulas interact with the text around them. The strongest systems still rely on hand-engineered structure matching. This paper asks whether bi-encoder dense retrievers can fill in what structure search misses. The authors adapt two representative models to the math domain through pretraining and fine-tuning: DPR, which scores whole passages, and ColBERT, which scores individual tokens. They evaluate both on NTCIR-12 Wiki-Formula and ARQMath-2. They then merge each dense run with the Approach0 structure search engine, mainly through linear score interpolation. On their own, the dense models trail cross encoders. Combined with structure search, however, they set new state-of-the-art results. The two signals help each other in different ways. Dense retrieval finds relevant documents that share no formula substructure with the query. Structure search suppresses dense hits that look similar but are mathematically different. The released checkpoints and pipeline give a useful baseline for a domain short on resources.

Key points

  • The authors describe this as the first evaluation of a DPR model for MIR. Their fine-tuned ColBERT clearly beats earlier ColBERT-style retrievers reported in this domain.
  • Benchmarks: NTCIR-12 Wiki-Formula, with 20 concrete queries over roughly 591,000 indexed formulas, and ARQMath-2.
  • On ARQMath-2, the Approach0 + ColBERT linear fusion reaches NDCG' 0.449, MAP' 0.217, P'@10 0.279 and BPref 0.204. Borda Count, CombSUM, ISR, log-ISR and RRF (k = 60) do no significantly better than this, as computed with the polyfuse tool.
  • Per-topic analysis shows large gains from fusion:
    • Topic A.287: P@10 rises from 0 to 0.8 compared with Approach0 alone.
    • Topic A.219: P@10 rises from 0.1 to 0.5 compared with ColBERT alone.
  • The stated limitations are the engineering overhead of maintaining two retrieval stacks, the lack of match highlighting for DPR, and ColBERT's heavy storage and GPU demands. Code and models are available at github.com/approach0/math-dense-retrievers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada85 works · 15,188 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 worksComputational Theory and Mathematics · 496 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 worksInformation retrieval & search · 561 works

Cite

Cite as Wei Zhong, Jheng-Hong Yang, Yuqing Xie and Jimmy Lin. 2022. Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval. Findings of the Association for Computational Linguistics: EMNLP 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-emnlp.78 · arXiv 2203.11163

@inproceedings{zhong2022evaluating,
  title = {{Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval}},
  author = {Wei Zhong, Jheng-Hong Yang, Yuqing Xie and Jimmy Lin},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.78},
  eprint = {2203.11163},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.78}
}

References

49 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Sci-BERT: A pretrained language model for scientific text. arXiv:1903.10676.
  2. Pankaj Dadure, Partha Pakray, and Sivaji Bandyopadhyay. 2021. BERT-based embedding model for formula retrieval. In CLEF. find
  3. Kenny Davila and Richard Zanibbi. 2017. Layout and semantics: Combining representations for mathematical formula search. In SIGIR. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805.
  5. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv:2109.10086.
  6. Dallas Fraser, Andrew Kane, and Frank Tompa. 2018. Choosing math features for BM25 ranking with Tangent-L. In DocEng. find
  7. Liangcai Gao, Zhuoren Jiang, Yue Yin, Ke Yuan, Zuoyu Yan, and Zhi Tang. 2017. Preliminary exploration of formula embedding for mathematical information retrieval: Can mathematical formulae be embedded like a natural language? arXiv:1707.05154. — [arXiv:1707.05154](https://arxiv.org/abs/1707.05154) find
  8. Luyu Gao and Jamie Callan. 2021. Condenser: A pre-training architecture for dense retrieval. arXiv:2104.08253.
  9. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv:2004.10964. find
  10. Michael Kohlhase, Bogdan A Matican, and Corneliu-Claudiu Prodescu. 2012. MathWebSearch 0.5: Scaling an open formula search engine. In CICM. find
  11. Giovanni Yoko Kristianto, Goran Topic, and Akiko Aizawa. 2016. MCAT math retrieval system for NTCIR-12 MathIR task. In NTCIR. find
  12. Xiaoyan Lin, Liangcai Gao, Xuan Hu, Zhi Tang, Ying-nan Xiao, and Xiaozhong Liu. 2014. A mathematics retrieval system for formulae in layout presentations. In SIGIR. find
  13. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv:1711.05101. find
  14. Behrooz Mansouri, Douglas W Oard, and Richard Zanibbi. 2020. DPRL systems in the CLEF 2020 ARQMath lab. In CLEF. find
  15. Behrooz Mansouri, Douglas W. Oard, and Richard Zanibbi. 2021a. DPRL systems in the CLEF 2021 ARQMath lab: Sentence-BERT for answer retrieval, learning-to-rank for formula retrieval. In CLEF. find
  16. Behrooz Mansouri, Shaurya Rohatgi, Douglas W. Oard, Jian Wu, C Lee Giles, and Richard Zanibbi. 2019. Tangent-CFT: An embedding model for mathematical formulas. In SIGIR. find
  17. Behrooz Mansouri, Richard Zanibbi, Douglas W. Oard, and Anurag Agarwal. 2021b. Overview of ARQMath-2 (2021): Second CLEF lab on answer retrieval for questions on math (working notes version). In CLEF. find
  18. Bruce R. Miller and Abdou Youssef. 2003. Technical aspects of the digital library of mathematical functions. In AMAI. find
  19. Yin Ki Ng. 2021. Dowsing for math answers: Exploring MathCQA with a math-aware search engine. Master’s thesis, University of Waterloo. find
  20. Yoshinori Hijikata, Hideki Hashimoto, and Shogo Nishida. 2009. Search mathematical formulas by mathematical formulas. In SHI (Symposium on Human Interface). find
  21. Yin Ki Ng, Dallas Fraser, Besat Kassaie, and Frank Tompa. 2021. Dowsing for answers to math questions: Ongoing viability of traditional MathIR. In CLEF. find
  22. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In SIGIR.
  23. Yin Ki Ng, Dallas J. Fraser, Besat Kassaie, George Labahn, Mirette S Marzouk, Frank Tompa, and Kevin Wang. 2020. Dowsing for math answers with Tangent-L. In CLEF. find
  24. Hervé Jégou, Romain Tavenard, Matthijs Douze, and Laurent Amsaleg. 2011. Searching in one billion vectors: Re-rank with source coding. In IEEE ICASSP. find
  25. Vít Novotnỳ, Michal Štefánik, Dávid Lupták, Martin Geletka, Petr Zelina, and Petr Sojka. 2021. Ensembling ten math information retrieval systems. In CLEF. find
  26. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv:2004.04906.
  27. Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In SIGIR.
  28. Iadh Ounis, Gianni Amati, Vassilis Plachouras, Ben He, Craig Macdonald, and Douglas Johnson. 2005. Terrier information retrieval platform. In ECIR. find
  29. Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. 2021. MathBERT: A pre-trained model for mathematical formula understanding. arXiv:2105.00377. — [arXiv:2105.00377](https://arxiv.org/abs/2105.00377) find
  30. Anja Reusch, Maik Thiele, and Wolfgang Lehner. 2021a. An ALBERT-based similarity measure for mathematical answer retrieval. In SIGIR. find
  31. Wei Zhong and Richard Zanibbi. 2019. Structural similarity search for formulas using leaf-root paths in operator subtrees. In ECIR. find
  32. Anja Reusch, Maik Thiele, and Wolfgang Lehner. 2021b. TU_DBS in the ARQMath lab 2021, CLEF. In CLEF. find
  33. Wei Zhong, Xinyu Zhang, Ji Xin, Jimmy Lin, and Richard Zanibbi. 2021. Approach Zero and Anserini at the CLEF-2021 ARQMath track: Applying substructure search and BM25 on operator tree path tokens. In CLEF. find
  34. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2021. Col-BERTv2: Effective and efficient retrieval via lightweight late interaction. arXiv:2112.01488.
  35. Thomas Schellenberg, Bo Yuan, and Richard Zanibbi. 2012. Layout-based substitution tree indexing and retrieval for mathematical expressions. In DRR. find
  36. Qi Zhu, Yuxian Gu, Lingxiao Luo, Bing Li, Cheng Li, Wei Peng, Minlie Huang, and Xiaoyan Zhu. 2021. When does further pre-training MLM help? An empirical study on task-oriented dialog pre-training. In EMNLP Insights Workshop. find
  37. Petr Sojka and Martin Líška. 2011. The art of mathematics retrieval. In DocEng. find
  38. Yujin Song and Xiaoyu Chen. 2021. Searching for mathematical formulas based on graph representation learning. In CICM. find
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
  40. Keisuke Yokoi and Akiko Aizawa. 2009. An approach to similarity search for mathematical expressions using MathML. In DML (Digital Mathematics Library). find
  41. Abdou Youssef. 2005. Search of mathematical contents: Issues and methods. In IASSE. find
  42. Richard Zanibbi, Akiko Aizawa, Michael Kohlhase, Iadh Ounis, Goran Topic, and Kenny Davila. 2016a. NTCIR-12 MathIR task overview. In NTCIR.
  43. Richard Zanibbi and Dorothea Blostein. 2012. Recognition and retrieval of mathematical expressions. In IJDAR. find
  44. Richard Zanibbi, Kenny Davila, Andrew Kane, and Frank Tompa. 2015. The Tangent search engine: Improved similarity metrics and scalability for math formula search. arXiv:1507.06235. — [arXiv:1507.06235](https://arxiv.org/abs/1507.06235) find
  45. Richard Zanibbi, Kenny Davila, Andrew Kane, and Frank Tompa. 2016b. Multi-stage math formula search: Using appearance-based similarity metrics at scale. In SIGIR. find
  46. Wei Zhong. 2015. A novel similarity-search method for mathematical content in LaTeX markup and its implementation. University of Delaware. find
  47. Wei Zhong and Hui Fang. 2016. OPMES: A similarity search engine for mathematical content. In ECIR. find
  48. Wei Zhong and Jimmy Lin. 2021. PyA0: A Python toolkit for accessible math-aware search. In SIGIR. find
  49. Wei Zhong, Shaurya Rohatgi, Jian Wu, Lee Giles, and Richard Zanibbi. 2020. Accelerating substructure similarity search for formula retrieval. In ECIR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhong-2022-evaluating-token-level-passage-level/