Elamur Research⌕
☰
home / papers / 2013 / NTCIR-10 Math Pilot Task Overview

NTCIR-10 Math Pilot Task Overview

Paper · 2013 · Akiko Aizawa, Michael Kohlhase and Iadh Ounis

Summary

Most search systems ignore mathematical formulae in documents. This is partly because there were no large structured formula collections, well-defined tasks or agreed evaluation methods. The NTCIR-10 Math Pilot Task was the first shared evaluation dedicated to math information access.

The organizers built a test collection of about 100,000 arXiv papers from mathematics, physics and computer science, converted by the arXMLiv project. In it, each formula is stored as presentation MathML, content MathML and LaTeX. Two subtasks were offered:

  • Math Retrieval: formula search with wildcard query variables, combined keyword-and-formula search, and open free-text queries.
  • Math Understanding: extracting textual descriptions of formulae.

Submitted runs were pooled per query and judged on a per-formula basis by mathematically trained assessors in the SEPIA system. Six of sixteen registered teams took part. Open-query runs could not be judged because they lacked formula identifiers. The only Math Understanding system reached high precision but modest recall.

The task created reusable topics, relevance judgments and tools. It also helped bring together a cross-disciplinary math IR community.

Key points

  • Dataset is roughly 63 GiB, with about 35.5 million formulae and 297 million subformulae (the units targeted by formula search).
  • Mathematicians supplied 56 topics: 22 formula search (FS), 15 full-text (FT) and 19 open (OMIR).
    • FS-17 was dropped as ill-formed.
    • No OMIR topics were evaluated because runs reported only document URIs.
  • Pooling collected at least 100 formulae per query from all runs.
    • Assessors from Zentralblatt Math and Jacobs University students labelled hits as relevant, partially relevant or not relevant.
    • 2,128 formulae were judged for FS topics alone.
  • Participants were BRKLY, FSE, KWARC, MCAT, MIRMU and NAK, with up to four runs per team. All six teams entered Math Retrieval.
  • In Math Understanding, MCAT paired noun phrases with formulae and classified the pairs with an SVM.
    • Best strict-matching F1 was 50.77 (short descriptions).
    • Best soft-matching F1 was 62.18, with precision 87.25 and recall 48.30.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬National Institute of InformaticsTokyo · Japan40 works · 3,317 citations🎓University of GlasgowGlasgow · United Kingdom15 works · 804 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksInformation Systems · 1,888 worksComputational Theory and Mathematics · 496 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Akiko Aizawa, Michael Kohlhase and Iadh Ounis. 2013. NTCIR-10 Math Pilot Task Overview. NTCIR-10.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{aizawa2013ntcir,
  title = {{NTCIR-10 Math Pilot Task Overview}},
  author = {Akiko Aizawa, Michael Kohlhase and Iadh Ounis},
  booktitle = {NTCIR-10},
  year = {2013},
  url = {https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings10/NTCIR/toc_ntcir.html}
}

References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. arxiv.org e-Print archive. web page at http://www.arxiv.org.seen November 2012. ↗
  2. arXMLiv: Translating the Math Archives to XML+MathML. web page at http://trac.kwarc.info/arXMLiv/. seen April 2010. ↗
  3. Hiroya Hagino and Hiroaki Saito. Partial-match retrieval with structure-reflected indices at the ntcir-10 math task. In NTCIR-10 [NTC13]. this volume. find
  4. Michael Kohlhase. Topics for the ntcir-10 math task; math retrieval subtask. Technical report, NTCIR, 2012. find
  5. Michael Kohlhase and Corneliu-Claudiu Prodescu. Mathwebsearch at ntcir-10. In NTCIR-10 [NTC13]. this volume. find
  6. Ray R. Larson, Chloe Reynolds, and Fredric Gey. The abject failure of keyword ir for mathematics search: Berkeley at ntcir-10 math. In NTCIR-10 [NTC13]. this volume. find
  7. Martin Lśka, Petr Sojka, and MichalRůžička. Mirmu at the ntcir-10 math task: Similarity search for mathematics. In NTCIR-10 [NTC13]. this volume. find
  8. NTCIR Workshop 10 Meeting, Tokyo, Japan, 2013. this volume. find
  9. NTICR Pilot Task: Math Task. http://ntcir-math.nii.ac.jp/, 2013. ↗
  10. sepia: Standard evaluation package for information access systems. https://code.google.com/p/sepia/. ↗
  11. Heinrich Stamerjohanns, Michael Kohlhase, Deyan Ginev, Catalin David, and Bruce Miller. Transforming large collections of scientific publications to XML. Mathematics in Computer Science, 3(3):299–307, 2010.
  12. Moritz Schubotz, Marcus Leich, and Volker Markl. Querying large collections of mathematical publications. In NTCIR-10 find
  13. Goran Topic, Giovanni Yoko Kristianto, Minh-Quoc Nghiem, and Akiko Aizawa. The mcat math retrieval system for ntcir-10 math track. In NTCIR-10 [NTC13]. this volume. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aizawa-2013-ntcir-10-math-pilot-task/