NTCIR-11 Math-2 Task Overview
Paper · 2014 · Akiko Aizawa, Michael Kohlhase, Iadh Ounis et al.
Summary
Mainstream search engines generally cannot retrieve mathematical formulae. Research on math information retrieval (MIR) has been held back by a lack of large structured datasets and agreed evaluation procedures. Building on the NTCIR-10 Math Pilot, the NTCIR-11 Math-2 Task provides a shared ad-hoc retrieval benchmark that combines formula and keyword queries.
The organizers converted about 105,120 arXiv preprints from LaTeX to HTML+MathML using LaTeXML. They split each document into paragraph-sized retrieval units, which makes relevance judgment manageable. Each of the 50 topics combines keywords with formulae, which may include wildcard query variables, encoded in presentation MathML, content MathML and LaTeX. Submitted runs were pooled and judged by humans in the SEPIA tool, with MAP, P@5, P@10 and Bpref as metrics.
An optional free subtask used English Wikipedia with automatic scoring. Eight teams took part, and their approaches differed in how they handled formula structure, query variables and search engines. The judged collection is a first comprehensive MIR test set and was scheduled for release by NTCIR in 2015.
Key points
- Corpus: 8,301,578 paragraph units containing about 60 million formulae. The arXiv categories include math, cs, stat, hep-th and others. Total size is about 174GB, stored as HTML5 and XHTML5.
- Topics: 50 formula/keyword queries, with 0–8 query variables each (
<mws:qvar>, written as ?x). - Participation: 8 groups, including 4 new ones, submitted 20 runs. The groups were ICST, IFISB, FSE, KWARC, MCAT, MIRMU, RIT and TUW-IMP.
- Assessment: 50 pooled units per topic gave 2,500 judgments in total. Each unit was assessed by at least two evaluators.
- The Wikipedia subtask used 100 automatically generated queries and immediate automatic evaluation. Four teams joined it.
Fields, topics and concepts
📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksInformation Systems · 1,888 worksComputational Theory and Mathematics · 496 works
🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,948 worksInformation retrieval & search · 561 works
Cite
Cite as Akiko Aizawa, Michael Kohlhase, Iadh Ounis and Moritz Schubotz. 2014. NTCIR-11 Math-2 Task Overview. NTCIR-11.
@inproceedings{aizawa2014ntcir,
title = {{NTCIR-11 Math-2 Task Overview}},
author = {Akiko Aizawa, Michael Kohlhase, Iadh Ounis and Moritz Schubotz},
booktitle = {NTCIR-11},
year = {2014},
url = {https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings11/NTCIR/toc_ntcir.html}
}References
- Proceedings of the 11th NTCIR Conference, Tokyo, Japan, 2014. find
- A. Aizawa, M. Kohlhase, and I. Ounis. NTCIR-10 Math pilot task overview. In N. Kando and K. Kishida, editors, NTCIR Workshop 10 Meeting, pages 18, Tokyo, Japan, 2013.
- J. M. Gonzalez Pinto, S. Barthel, and W.-T. Balke. QUALIBETA at the NTCIR-11 Math 2 task: An attempt to query math collections. [1]. find
- M. Kohlhase. Formats for topics and submissions for the Math2 task at NTCIR-11. Technical report, NTCIR, 2014. find
- M. Kohlhase, R. Hambasan, and C.-C. Prodescu. MathWebSearch at NTCIR-11. [1]. find
- G. Y. Kristianto, G. Topi¢, F. Ho, and A. Aizawa. The MCAT math retrieval system for NTCIR-11 Math track. [1]. find
- J. R. Landis and G. G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33:159174, 1977. find
- L. H. Liangcai Gao, Yuehan Wang and Z. Tang. Icst math retrieval system for ntcir-11 math-2 task. [1]. find
- A. Lipani, L. Andersson, F. Piroi, M. Lupu, and find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/aizawa-2014-ntcir-11-math-2-task/