Elamur Research⌕
☰
home / papers / 2016 / NTCIR-12 MathIR Task Overview

NTCIR-12 MathIR Task Overview

Paper · 2016 · Richard Zanibbi, Akiko Aizawa, Michael Kohlhase et al.

Summary

Most search engines cannot handle mathematical formulae, though formulae are central to scientific communication. This overview describes the third NTCIR math retrieval evaluation, NTCIR-12 MathIR. It offers a shared benchmark for systems that search documents using keywords, formulae, or both. The organizers reused the NTCIR-11 arXiv collection, split into paragraph-level units. They also built a new English Wikipedia collection that targets non-expert users, such as students. Four subtasks were defined:

  • arXiv-main (keywords plus formulae)
  • arXiv-simto, an experimental task testing a new "similar-to" region operator inside formula queries
  • Wiki-main
  • Wiki-formula, which uses isolated formula queries and graded relevance instead of recall of known targets

Student assessors judged pooled results in the SEPIA interface. The strongest systems, MCAT and ICST, combined text retrieval with formula retrieval and used surrounding context. Still, every run fell well short of the ideal ranking derived from the pools. The authors conclude that combining formula appearance with formula semantics looks more promising than choosing one encoding over the other.

Key points

  • The arXiv corpus contains 105,120 articles converted to HTML+MathML by KWARC. It is split into 8,301,578 paragraph units holding roughly 60 million formulae.
  • The new Wikipedia corpus has 319,689 articles and over 590,000 formulae in LaTeX, Presentation MathML and Content MathML. About 10% of the articles contain math markup, and the remaining 90% are text-only distractors.
  • There were 107 topics in total: arXiv-main 29, arXiv-simto 8, Wiki-main 30 and Wiki-formula 40. Half of the Wiki-formula queries contain wildcards.
  • Teams could submit up to four runs per task, with 1,000 results per query. Inter-assessor agreement (Fleiss' κ) was higher for arXiv than for Wikipedia and lowest for Wiki-formula.
  • In Wiki-formula, MCAT slightly beat RITUW but was slower. Unification-based re-ranking helped ranking quality but cost speed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksInformation Systems · 1,888 worksComputational Theory and Mathematics · 496 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Richard Zanibbi, Akiko Aizawa, Michael Kohlhase, Iadh Ounis, Goran Topić and Kenny Davila. 2016. NTCIR-12 MathIR Task Overview. NTCIR-12.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zanibbi2016ntcir,
  title = {{NTCIR-12 MathIR Task Overview}},
  author = {Richard Zanibbi, Akiko Aizawa, Michael Kohlhase, Iadh Ounis, Goran Topić and Kenny Davila},
  booktitle = {NTCIR-12},
  year = {2016},
  url = {https://ir.webis.de/anthology/2016.ntcir_conference-2016.52/}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Aizawa, M. Kohlhase, and I. Ounis. NTCIR-10 Math pilot task overview. In N. Kando and K. Kishida, editors, NTCIR Workshop 10 Meeting, pages 1–8, Tokyo, Japan, 2013.
  2. A. Aizawa, M. Kohlhase, I. Ounis, and M. Schubotz. NTCIR-11 math-2 task overview. In NTCIR. National Institute of Informatics (NII), 2014.
  3. K. Davila, R. Zanibbi, A. Kane, and F. Tompa. Tangent-3 at the NTCIR-12 MathIR task. In Proc. NTCIR-12, 2016. find
  4. L. Gao, K. Yuan, Y. Wang, Z. Jiang, and Z. Tang. The math retrieval system of ICST for NTCIR-12 MathIR task. In Proc. NTCIR-12, 2016. find
  5. F. Guidi and C. S. Coen. A survey on retrieval of mathematical knowledge. In M. K. et al., editor, Proc. CICM, volume 9150 of LNAI, pages 296–315, 2015. find
  6. M. Kohlhase. Formats for topics and submissions for the MathIR task at NTCIR-12. Technical report, NTCIR, 2015. find
  7. G. Y. Kristianto, G. Topić, and A. Aizawa. MCAT math retrieval system for NTCIR-12 MathIR task. In Proc. NTCIR-12, 2016. find
  8. M. Lı́ška, P. Sojka, and M. Rûžička. Combining text and formula queries in math information retrieval: Evaluation of query results merging strategies. In Proc. Int’l Work. Novel Web Search Interfaces and Systems, pages 7–9, New York, 2015. find
  9. L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab, November 1999.
  10. M. Rûžička, P. Sojka, and M. Lı́ška. Math indexer and searcher under the hood: Fine-tuning query expansion and unification strategies. In Proc. NTCIR-12, 2016. find
  11. M. Schubotz, M. Leich, N. Meuschke, and B. Gipp. Exploring the one-brain barrier: a manual contribution to the NTCIR-12 Math task. In Proc. NTCIR-12, 2016. find
  12. M. Schubotz, A. Youssef, V. Markl, and H. S. Cohl. Challenges of mathematical information retrieval in the NTCIR-11 Math Wikipedia Task. In Proc. ACM SIGIR, pages 951–954, 2015. find
  13. sepia: Standard evaluation package for information access systems. https://code.google.com/p/sepia/. ↗
  14. P. Sojka and M. Lı́ška. The Art of Mathematics Retrieval. In Proc. ACM DocEng, pages 57–60, Mountain View, CA, Sept. 2011. find
  15. A. Thanda, A. Agarwal, K. Singla, A. Prakash, and A. Gupta. A document retrieval system for math queries. In Proc. NTCIR-12, 2016. find
  16. Common evaluation measures. In E. M. Voorhees and L. P. Buckland, editors, The Sixteenth Text REtrieval Conference (TREC 2007) Proceedings, number SP 500-274 in NIST Special Publication, 2007. find
  17. R. Zanibbi and D. Blostein. Recognition and retrieval of mathematical expressions. Int. J. Doc. Analysis and Recognition, 15(4):331–357, 2012. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zanibbi-2016-ntcir-12-mathir-task-overview/