Elamur Research⌕
☰
home / papers / 2010 / Transforming Large Collections of Scientific Publications to XML

Transforming Large Collections of Scientific Publications to XML

Paper · 2010 · Heinrich Stamerjohanns, Michael Kohlhase, Deyan Ginev et al.

Summary

Most mathematical and scientific papers are written in LaTeX. LaTeX mixes content with layout and lets authors define their own macros, so machines struggle to recover the meaning of formulas from it. The arXMLiv project tackles this at scale by converting the arχiv e-print collection into XHTML+MathML with LaTeXML. LaTeXML re-implements much of the TeX engine and keeps document structure and math semantics as intact as it can. The paper's main contribution is a distributed build system. It schedules conversion jobs across several hosts and parses LaTeXML's logs. It stores per-document outcomes in an SQL database, including errors, missing packages and unsupported macros. These statistics show developers which package bindings and tool fixes matter most, so the system works both as a production pipeline and as a test harness. After the whole pre-2007 arχiv was processed, more than 90% of documents could be converted. Around 61% converted cleanly. The result is a large real-world MathML corpus for semantic analysis, math search and accessibility.

Key points

  • The corpus covers the complete arχiv through 2006: more than 400,000 documents, about 150 GB, shared over NFS. One full run takes roughly a processor-year of computation.
  • Outcomes:
    • About 61% of documents convert with no or only minor warnings.
    • Another ~30% produce XHTML+MathML of uncertain quality.
    • Together these add over 240,000 usable documents.
  • The build system consists of a PHP build manager, a queue manager, an SQL work queue and statistics database, and Makefiles that call latexml and latexmlpost. Each conversion has a 180-second timeout, and results can be browsed at arxmliv.kwarc.info.
  • The team wrote more than two thousand LaTeXML binding files for arχiv packages. Abbreviative macros use DefMacro, and semantic macros use DefConstructor. Some packages, such as pstricks, are only partly supported.
  • Planned next steps:
    • Better OpenMath/Content-MathML generation and notation disambiguation.
    • Conversion of further corpora; early tests on Zentralblatt Math show nearly perfect conversion rates.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Constructor UniversityBremen · Germany8 works · 30,344 citations🔬National Institute of Standards and TechnologyGaithersburg · United States5 works · 492 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArts and Humanities · 17,959 worksInformation Systems · 1,888 worksComputational Theory and Mathematics · 496 worksLiterature and Literary Theory · 542 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Heinrich Stamerjohanns, Michael Kohlhase, Deyan Ginev, Catalin David and Bruce R. Miller. 2010. Transforming Large Collections of Scientific Publications to XML. Mathematics in Computer Science 3(3).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{stamerjohanns2010transforming,
  title = {{Transforming Large Collections of Scientific Publications to XML}},
  author = {Heinrich Stamerjohanns, Michael Kohlhase, Deyan Ginev, Catalin David and Bruce R. Miller},
  booktitle = {Mathematics in Computer Science 3(3)},
  year = {2010},
  doi = {10.1007/s11786-010-0024-7},
  url = {https://doi.org/10.1007/s11786-010-0024-7}
}

References

15 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Ausbrooks, Bert Bos, Olga Caprotti, David Carlisle, Giorgi Chavchanidze, Ananth Coorg, Stéphane Dalmas, Stan Devitt, Sam Dooley, Margaret Hinchcliffe, Patrick Ion, Michael Kohlhase, Azzeddine Lazrek, Dennis Leas, Paul Libbrecht, Manolis Mavrikis, Bruce Miller, Robert Miner, Murray Sargent, Kyle… find
  2. Romeo Anghelache. Hermes - a semantic xml+mathml+unicode e-publishing/self-archiving tool for latex authored scientific articles. web page at http://hermes.roua.org/, 2007. ↗
  3. arXiv.org e-Print archive, seen December2007. web page at http://www.arxiv.org. ↗
  4. Stephen Buswell, Olga Caprotti, David P. Carlisle, Michael C. Dewar, Marc Gaetano, and Michael Kohlhase. The Open Math standard, version 2.0. Technical report, The Open Math Society, 2004. find
  5. Deyan Ginev, Constantin Jucovschi, Stefan Anca, Mihai Grigore, Catalin David, and Michael Kohlhase. An architecture for linguistic and semantic analysis on the arXMLiv corpus. In Applications of Semantic Technologies (AST) Workshop at Informatik 2009, 2009. in press. find
  6. Michael Kohlhase. OMDoc – An open markup format for mathematical documents [Version 1.2]. Number 4180 in LNAI. Springer Verlag, 2006. find
  7. Michael Kohlhase. Using L A TEX as a semantic markup format. Mathematics in Computer Science, 2008. find
  8. Michael Kohlhase and Ioan Şucan. A search engine for mathematical formulae. In Tetsuo Ida, Jacques Calmet, and Dongming Wang, editors, Proceedings of Artificial Intelligence and Symbolic Computation, AISC’2006, number 4120 in LNAI, pages 241–253. Springer Verlag, 2006. find
  9. Mathplayer: Speech instructions and examples. web page at http://www.dessci.com/en/products/mathplayer/tech/accessibility.htm. ↗
  10. Math Web Search. http://kwarc.info/projects/mws/, seen Dec. 2008. ↗
  11. Bruce Miller. LaTeXML: A L A TEX to xml converter. Web Manual at http://dlmf.nist.gov/LaTeXML/, seen September2007. + ↗
  12. Heinrich Stamerjohanns, Deyan Ginev, Catalin David, Dimitar Misev, Vladimir Zamdzhiev, and Michael Kohlhase. Mathml-aware article conversion from L A TEX, a comparison study. In Petr Sojka, editor, Towards Digital Mathematics Library, DML 2009 workshop. Masaryk University, Brno, 2009. find
  13. TeX4ht: LaTeX and TeX for hypertext. web page at http://www.tug.org/applications/tex4ht/mn.html. ↗
  14. Mark van den Brand and Jürgen Stuber. Extracting mathematical semantics from latex documents. In Proc. Intl. Workshop on Principles and Practice of Semantic Web Reasoning (PPSWR 2003), number 2901 in LNCS, pages 160–173, Mumbai, India, 2003. Springer. find
  15. Zentralblatt MATH, seen December2007. web page at http://www.zentralblatt-math.org. H. Stamerjohanns Computer Science, Jacobs University Bremen e-mail: h.stamerjohanns@jacobs-university.de M. Kohlhase Computer Science, Jacobs University Bremen e-mail: m.kohlhase@jacobs-university.de D. Ginev Comput… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stamerjohanns-2010-transforming-large-collections-scientific-publications/