Elamur Research⌕
☰
home / papers / 2026 / Sanskrit Travelogue: A Large-Scale Unified and Annotated Corpus of Sanskrit Texts

Sanskrit Travelogue: A Large-Scale Unified and Annotated Corpus of Sanskrit Texts

Summary

Computational work on Sanskrit is hampered by orthographic sandhi, heavy compounding and rich inflection. Digital texts are also scattered across libraries with inconsistent transliteration, markup and metadata. The authors build Sanskrit Travelogue, an open corpus that merges eight public collections, including GRETIL, SARIT, DSBC, DCS and DharmaNexus. A reproducible pipeline converts everything to IAST, reconciles metadata, keeps structural information such as verse markers, chapter hierarchy and apparatus, and removes duplicates conservatively so that genuine variant readings survive. Each segment is then annotated automatically by two tools: ByT5-Sanskrit divides sandhi and compounds, while the process-sanskrit library strips inflectional endings and assigns morphological tags. The output includes several aligned views (cleaned, split, stemmed, diacritic-free, tagged), and ambiguities are kept rather than resolved. Accuracy is high, and a manual check on the Yoga Sūtra finds that combining both systems beats either one alone. The resource supports search across sandhi and inflection, and it supplies training data for segmentation and lemmatization. It also enables philological and diachronic research.

Key points

  • Scale: after de-duplication the corpus has 12,394 texts, about 73.1M tokens and more than 9M segments. Before de-duplication it had about 84M tokens, and 693 texts were removed.
  • Segmentation and root extraction: ByT5-Sanskrit reaches 84.61% sentence-level exact match. Root extraction for compounds is 92.37% correct, and the Yoga Sūtra score is 95.94%.
  • Combined systems: manual evaluation on the Yoga Sūtra gives 98% correct root extraction when both systems are used together.
  • Release: TSV shards, a database dump, a query normalizer and acquisition and annotation code are published on GitHub (SanskritVoyager/SanskritTravelogue), with Hugging Face distribution planned. The license is Creative Commons non-commercial according to the abstract, although other parts of the paper call it "permissive." Two IAST converters for rare transliteration schemes (Kyoto Sangyo University and Tsukuba) are also included.
  • Limitations: the annotation is fully automatic and has not been validated corpus-wide. Some duplicates likely remain, and GRETIL markup normalization covers only a subset of its texts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Treebanks

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5ibfaw5j9br6

@inproceedings{luca2026sanskrit,
  title = {{Sanskrit Travelogue: A Large-Scale Unified and Annotated Corpus of Sanskrit Texts}},
  author = {Luca, Giacomo De and Croce, Danilo and Basili, Roberto},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5ibfaw5j9br6},
  url = {https://doi.org/10.63317/5ibfaw5j9br6}
}

References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. George R Cardona. 1988. Pān.ini : his work and its traditions. Motilal Banarsidass, Delhi, India. find
  2. Cologne University. 2024. Cologne digital Sanskrit dictionaries. Online resource. Accessed on February 19, 2025. find
  3. Giacomo De Luca. 2025. Accessible Sanskrit: A cascading system for text analysis and dictionary access. In Proceedings of the Second Workshop on Ancient Language Processing, pages 38–46, The Albuquerque Convention Center, Laguna. Association for Computational Linguistics. find
  4. Brendan S Gillon. 2007. Pān.ini’s" as.t.ādhyāyı̄" and linguistic theory. Journal of Indian philosophy, 35(5/6):445–468. find
  5. Oliver Hellwig and Erica Biagetti. 2025. The sanskrit sembank. Language Resources and Evaluation. Advance online publication (2025-07-16). find
  6. Oliver Hellwig and Sebastian Nehrdich. 2018. Sanskrit word segmentation using character-level recurrent and convolutional neural networks. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2754–2763. find
  7. Gérard Huet. 2009. Sanskrit segmentation. In Proceedings of the South Asian Languages Analysis Roundtable XXVIII, Denton, Ohio. find
  8. Stephanie W Jamison and Joel P Brereton. 2014. The Rigveda: 3-Volume Set. Oxford University Press. find
  9. Amrith Krishna, Pavan Kumar Satuluri, and Pawan Goyal. 2017. A dataset for Sanskrit word segmentation. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 105–114, Vancouver, Canada. Association for Computa… find
  10. Sriram Krishnan, Amba Kulkarni, and Gérard Huet. 2020. Validation and normalization of dcs corpus using sanskrit heritage tools to build a tagged gold corpus. find
  11. Sebastian Nehrdich, Oliver Hellwig, and Kurt Keutzer. 2024. One model is all you need: ByT5-Sanskrit, a unified model for Sanskrit NLP tasks. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13742–13751, Miami, Florida, USA. Association for Computational Linguistics. find
  12. Jivnesh Sandhan, Rathin Singha, Narein Rao, Suvendu Samanta, Laxmidhar Behera, and Pawan Goyal. 2022. Translist: A transformer-based linguistically informed sanskrit tokenizer. arXiv preprint arXiv:2210.11753. 8. find
  13. Language Resource References Grünendahl, Reinhold. 2001. GRETIL: Göttingen Register of Electronic Texts in Indian Languages. Niedersächsische Staats- und Universitätsbibliothek Göttingen. PID https://gretil.sub.uni-goettingen.de/. Comprehensive repository of machine-readable texts in Sanskrit and ot… ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/luca-2026-sanskrit-travelogue-large-scale-unified/