Elamur Research⌕
☰
home / papers / 2026 / ParsCORE: The Persian Corpus of Online Registers

ParsCORE: The Persian Corpus of Online Registers

Paper · 2026 · Alireza Razzaghi, Erik Henriksson and Veronika Laipalla

Summary

Automatic web register (genre) classifiers are now reliable enough to tag web-scale datasets and guide data selection for LLM training. However, their training data come mostly from European languages, and Persian was represented in Multilingual CORE by only 69 documents. To fill this gap, the authors built ParsCORE v0.1, a manually labeled corpus of Persian online registers drawn from a random 2,000-document sample of the Persian part of HPLT 3.0.

Labels follow the hierarchical, multi-label CORE scheme, so hybrid texts can receive several registers. Incoherent or junk pages were filtered out. Rare classes were then supplemented with harder examples, chosen using HPLT's own automatic register confidence scores. The paper also reports the register distribution, studies distinctive vocabulary with text dispersion keyness, and trains register classifiers under three settings.

Classification reaches a micro F1 of about 0.76, roughly on par with results for well-resourced languages. The resource gives Persian a basis for register-aware web data filtering. It also supports research into how Persian online language varies across registers and cultural contexts.

Key points

  • Source and scheme: The data come from the Persian section of HPLT 3.0 (about 124M near-deduplicated documents). Annotation uses the CORE taxonomy of 9 main and 14 subregister classes.
  • Filtering: 700 documents were rejected during annotation, for example as junk (JNK), as texts mixing more than three main registers (MLPT), or as text in another language (OTHL).
  • Language-mixing tags: Accepted texts could be tagged for code-switching (OTHLI) or Latin-script Persian (FINGLISH). Code-switching is more common in How-to and Informational Persuasion and rarer in Narrative.
  • Balancing classes: Underrepresented classes were raised to at least 145 instances. The added documents had HPLT automatic confidence scores between 0.2 and 0.4.
  • Results and limitations: Overall micro F1 is about 0.76, and the multilingual and zero-shot setups score near 0.75. No formal inter-annotator agreement was computed, and only the first 512 or 1024 tokens of each document were used. Data and code are released on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TurkuTurku · Finland17 works · 4,890 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Alireza Razzaghi, Erik Henriksson and Veronika Laipalla. 2026. ParsCORE: The Persian Corpus of Online Registers. The Proceedings of the First Workshop on NLP and LLMs for the Iranian Language Family.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.silkroadnlp-1.7

@inproceedings{razzaghi2026parscore,
  title = {{ParsCORE: The Persian Corpus of Online Registers}},
  author = {Alireza Razzaghi, Erik Henriksson and Veronika Laipalla},
  booktitle = {The Proceedings of the First Workshop on NLP and LLMs for the Iranian Language Family},
  year = {2026},
  doi = {10.18653/v1/2026.silkroadnlp-1.7},
  url = {https://doi.org/10.18653/v1/2026.silkroadnlp-1.7}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Culpeper and J. Demmen. 2015. Keywords, pages 90–105. find
  2. Douglas Biber. 1988. Variation across Speech and Writing. Cambridge University Press, Cambridge. find
  3. Jesse Egbert and Doug Biber. 2018. Incorporating text dispersion into keyword analyses. Corpora, 14(1):77–104. Publisher Copyright: © Edinburgh University Press. find
  4. Douglas Biber. 1995. Dimensions of Register Variation: A Cross-Linguistic Comparison. Cambridge University Press, Cambridge. find
  5. Selcen Erten. 2023. Exploring register variation in turkish web corpus. 14–15 September 2023, University of Mannheim, Germany, page 60. find
  6. Douglas Biber and Susan Conrad. 2019. Register, Genre, and Style. Cambridge University Press, Cambridge. find
  7. Selcen Erten-Johansson, Valtteri Skantsi, Sampo Pyysalo, and Veronika Laippala. 2024. Linguistic variation beyond the Indo-European Web: Analyzing Turkish Web registers in TurCORE. Register Studies. find
  8. Douglas Biber and Jesse Egbert. 2018. Register Variation Online. Cambridge University Press. find
  9. Douglas Biber and Jesse Egbert. 2023. What is a register? accounting for linguistic and situational variation within – and outside of – textual varieties. Register Studies, 5(1):1–22. find
  10. Stefan Gries. 2021. A new approach to (key) keywords analysis: Using frequency, and now also dispersion. Research in Corpus Linguistics, 9:1–33. find
  11. Douglas Biber, Jesse Egbert, and Mark Davies. 2015. Exploring the composition of the searchable web: A corpus-based taxonomy of web registers. Corpora, 10(1):11–45. find
  12. Saara Hellström, Valtteri Skantsi, Anna Salmela, and Veronika Laippala. 2025. From keywords to key embeddings – contrasting french and swedish web registers using multilingual deep learning. Corpus Linguistics and Linguistic Theory. find
  13. Douglas Biber, Jesse Egbert, and Daniel Keller. 2020. Reconceptualizing register in a continuous situational space. Corpus Linguistics and Linguistic Theory, 16(3):581–616. find
  14. Erik Henriksson, Amanda Myntti, Saara Hellström, Anni Eskelinen, Selcen Erten-Johansson, and Veronika Laippala. 2024. Automatic register identification for the open web using multilingual deep learning. arXiv preprint arXiv:2406.19892. find
  15. Marina Bondi. 2010. Perspectives on keywords and keyness: An introduction, pages 1–18. John Benjamins Publishing Company. find
  16. Taja Kuzman and Nikola Ljubešić. 2023. Automatic genre identification: A survey. Language Resources and Evaluation. find
  17. Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Mal… — An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
  18. Taja Kuzman, Igor Mozetič, and Nikola Ljubešić. 2023. Automatic genre identification for robust enrichment of massive text collections: Investigation of classification methods in the era of large language models. Machine Learning and Knowledge Extraction, 5(3):1149–1175. find
  19. Taja Kuzman, Peter Rupnik, and Nikola Ljubešić. 2022. The GINCO training dataset for web genre identification of documents out in the wild. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1584–1594, Marseille, France. European Language Resources Association. find
  20. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.
  21. Mikhail Lepekhin and Serge Sharoff. 2021. Experiments with adversarial attacks on text genres. CoRR, abs/2107.02246. — [arXiv:2107.02246](https://arxiv.org/abs/2107.02246) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/razzaghi-2026-parscore-persian-corpus-online-registers/