Elamur Research⌕
☰
home / papers / 2026 / Śmigiel Dataset: Laying Foundations for Investigating Machine-Generated Text Detection in Polish

Śmigiel Dataset: Laying Foundations for Investigating Machine-Generated Text Detection in Polish

Summary

Research on detecting machine-generated text (MGT) has focused mainly on English, even though current LLMs write convincingly in many other languages, including Polish. Before this work, Polish had only one preliminary study, limited to e-mails. The authors built Śmigiel, an openly released resource for training and benchmarking detectors in Polish. They gathered recent, permissively licensed human-written passages from six domains. Sentence-based prefixes taken from these passages were used to prompt eight LLMs to continue each text in its original style. The result is a large raw corpus of generated texts and a cleaned version balanced by source and domain. Baseline experiments with text classifiers show that detection difficulty depends on three factors: the domain, the generating model, and whether similar data appeared in training. General-purpose classifiers transferred well to LLMs they had not seen but performed worse on genres missing from the training data. Śmigiel is the first multi-domain, multi-model MGT benchmark for Polish and was already used in a PolEval 2025 shared task.

Key points

  • The raw corpus contains 462 360 Polish fragments generated by eight LLMs of different sizes. The balanced, cleaned dataset has 64 538 human-written and machine-generated fragments.
  • Human-written sources include PLSC, open-coursebooks-pl, Wikiźródła, PolEmo, Allegro Reviews, Filmweb, Filmweb+, TwitterEmo, BAN-PL, Polish Wikipedia, Polish Wikinews and ParlaMint.
  • Four domains (literature, reviews, social, wikipedia) appear in both training and test data. The news and parlamint domains are held out for testing only, to measure cross-genre generalisation.
  • Prefixes consist of whole sentences segmented with the LAMBO tokeniser. Their minimum length depends on genre, ranging from 26 characters for social media to 50 characters for encyclopedic entries.
  • The data is available on Zenodo (doi:10.5281/zenodo.18919631) and served as the basis for Task 1 of PolEval 2025.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3p7ghe9pfm8v

@inproceedings{strebeyko2026smigiel,
  title = {{Śmigiel Dataset: Laying Foundations for Investigating Machine-Generated Text Detection in Polish}},
  author = {Strebeyko, Jakub and Wróblewska, Alina and Przybyła, Piotr},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3p7ghe9pfm8v},
  url = {https://doi.org/10.63317/3p7ghe9pfm8v}
}

References

76 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Forough Amirjalili, Masoud Neysani, and Ahmadreza Nikbakht. 2024. Exploring the boundaries of authorship: a comparative analysis of AI-generated text and human academic writing in English literature. Frontiers in Education, Volume 9 -. find
  2. Tony Berber Sardinha. 2024. AI-generated vs human-authored texts: A multidimensional comparison. Applied Corpus Linguistics, 4(1):100083. find
  3. Daria Beresneva. 2016. Computer-Generated Text Detection Using Machine Learning: A Systematic Review. In Natural Language Processing and Information Systems, pages 421–426, Cham. Springer International Publishing. find
  4. Janek Bevendorff, Daryna Dementieva, Maik Fröbe, Bela Gipp, André Greiner-Petter, Jussi Karlgren, Maximilian Mayerl, Preslav Nakov, Alexander Panchenko, Martin Potthast, Artem Shelmanov, Efstathios Stamatatos, Benno Stein, Yuxia Wang, Matti Wiegmann, and Eva Zangerle. 2025. Overview of PAN 2025: Voi… find
  5. Stanisław Bogdanowicz, Hanna Cwynar, Aleksandra Zwierzchowska, Cezary Klamra, Witold Kieraś, and Łukasz Kobyliński. 2023. Twitter-emo: Annotating emotions and sentiment in polish twitter. In Computational Science – ICCS 2023, pages 212–220, Cham. Springer Nature Switzerland. find
  6. Savvas Chamezopoulos, Drahomira Herrmannova, Anita De Waard, Drahomira Herrmannova, Domenic Rosati, and Yury Kashnitsky. 2024. Overview of the DagPap24 shared task on detecting automatically generated scientific paper. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024)… find
  7. Shammur Absar Chowdhury, Hind Almerekhi, Mucahid Kutlu, Kaan Efe Keleş, Fatema Ahmad, Tasnim Mohiuddin, George Mikros, and Firoj Alam. 2025. GenAI content detection task 2: AI vs. human – academic essay authenticity challenge. In Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect… find
  8. Consortium PLLuM. 2025. Pllum: A family of polish large language models. find
  9. Roberto Corizzo and Sebastian Leal-Arenas. 2023. A Deep Fusion Model for Human $vs$. find
  10. Machine-Generated Essay Classification. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1–10. find
  11. Simon Corston-Oliver, Michael Gamon, and Chris Brockett. 2001. A Machine Learning Approach to the Automatic Evaluation of Machine Translation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 148–155, Toulouse, France. Association for Computational Li… find
  12. Evan N. Crothers, Nathalie Japkowicz, and Herna L. Viktor. 2023. Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods. IEEE Access, 11:70977–71002. find
  13. Sławomir Dadas, Michał Perełkiewicz, and Rafał Poświata. 2020. Pre-training polish transformer-based language models at scale. In Artificial Intelligence and Soft Computing, pages 301–314. Springer International Publishing. find
  14. Sławomir Dadas. 2025. Polish gpt-2 models. find
  15. DeepSeek-AI. 2024. Deepseek-v3 technical report. find
  16. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Fine-tuning of Quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. Curran Associates, Inc. find
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  18. Liam Dugan, Andrew Zhu, Firoj Alam, Preslav Nakov, Marianna Apidianaki, and Chris Callison-Burch. 2025. GenAI content detection task 3: Cross-domain machine generated text detection challenge. In Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect), pages 377–388, Abu Dhabi, UAE.… find
  19. Pieter Fivez, Walter Daelemans, Tim Van de Cruys, Yury Kashnitsky, Savvas Chamezopoulos, Hadi Mohammadi, Anastasia Giachanou, Ayoub Bagheri, Wessel Poelman, Juraj Vladika, Esther Ploeger, Johannes Bjerva, Florian Matthes, and Hans van Halteren. 2024. The clin33 shared task on the detection of text g… find
  20. Christina Frohock. 2025. Ghosts at the Gate: A Call for Vigilance Against AI-Generated Case Hallucinations. Penn State Law Review, 130(1). find
  21. Matthias Gallé, Jos Rozen, Germán Kruszewski, and Hady Elsahar. 2021. Unsupervised and Distributional Detection of Machine-Generated Text. find
  22. Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical Detection and Visualization of Generated Text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy. Association for Computat… find our card
  23. Gemma Team and Google DeepMind. 2024. Gemma: Open Models Based on Gemini Research and Technology. Technical report, Google DeepMind. find
  24. Gemma Team, Morgane Riviere, Shreya Pathak, and Pier Giuseppe Sessa et al. 2024. Gemma 2: Improving open language models at a practical size. find
  25. Gilad Gressel, Rahul Pankajakshan, and Yisroel Mirsky. 2024. Discussion Paper: Exploiting LLMs for Scam Automation: A Looming Threat. In Proceedings of the 3rd ACM Workshop on the Security Implications of Deepfakes and Cheapfakes, WDC ’24, pages 20–24, New York, NY, USA. Association for Computing Ma… find
  26. Paweł Gryka, Kacper Gradoń, Marek Kozłowski, Miłosz Kutyła, and Artur Janicki. 2024. Detection of AI-Generated Emails - A Case Study. In Proceedings of the 19th International Conference on Availability, Reliability and Security, ARES ’24, New York, NY, USA. Association for Computing Machinery. find
  27. Ahmed Abdeen Hamed and Xindong Wu. 2024. Detection of ChatGPT fake science with the xFakeSci learning algorithm. Scientific Reports, 14(1):16231. find
  28. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. find
  29. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  30. Yekyung Kim, Jenna Russell, Marzena Karpinska, and Mohit Iyyer. 2025. One ruler to measure them all: Benchmarking multilingual long-context language models. find
  31. Zae Myung Kim, Kwang Lee, Preston Zhu, Vipul Raheja, and Dongyeop Kang. 2024. Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5449–5474, Bangkok,… find
  32. Jan Kocoń, Piotr Miłkowski, and Monika Zaśko-Zielińska. 2019. Multi-level sentiment analysis of PolEmo 2.0: Extended corpus of multi-domain consumer reviews. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 980–991, Hong Kong, China. Association for Com… find
  33. Anna Kolos, Inez Okulska, Kinga Głąbińska, Agnieszka Karlinska, Emilia Wisnios, Paweł Ellerik, and Andrzej Prałat. 2024. BAN-PL: A Polish dataset of banned harmful and offensive content from wykop.pl web service. In Proceedings of the 2024 Joint International Conference on Computational Linguistics,… find
  34. Cyril Labbé and Dominique Labbé. 2013. Duplicate and fake publications in the scientific literature: How many SCIgen papers in computer science? Scientometrics, 94(1):379–396. find
  35. Thomas Lavergne, Tanguy Urvoy, and François Yvon. 2008. Detecting fake content with relative entropy scoring. In Proceedings of the 2008 International Conference on Uncovering Plagiarism, Authorship and Social Software Misuse - Volume 377, PAN’08, pages 27–31, Aachen, DEU. CEUR-WS.org. find
  36. Martin Májovský, Martin Černý, Matěj Kasal, Martin Komarc, and David Netuka. 2023. Artificial Intelligence Can Generate Fraudulent but Authentic-Looking Scientific Medical Articles: Pandora’s Box Has Been Opened. J Med Internet Res, 25(1):e46924. find
  37. Andrii Maslo and Silvia Gargova. 2025. BuST: A Siamese Transformer Model for AI Text Detection in Bulgarian. In Proceedings of Interdisciplinary Workshop on Observations of Misunderstood, Misguided and Malicious Use of Language Models, pages 45–52, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  38. Meta Team, Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri et al. 2024. The Llama 3 Herd of Models. find
  39. Dan Milmo. 2023. Mushroom pickers urged to avoid foraging books on Amazon that appear to be written by AI. The Guardian. find
  40. Lorenz Mindner, Tim Schlippe, and Kristina Schaaff. 2023. Classification of Human- and AI-Generated Texts: Investigating Features for Chat-GPT , pages 152–170. Springer Nature Singapore. find
  41. Mistral AI team. 2024. Mistral NeMo. find
  42. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Res… find our card
  43. Robert Mroczkowski, Piotr Rybak, Alina Wr ’oblewska, and Ireneusz Gawlik. 2021. Her-BERT: Efficiently pretrained transformer-based language model for Polish. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 1–10, Kiyv, Ukraine. Association for Computational Lingu… find
  44. Hoang-Quoc Nguyen-Son, Minh-Son Dao, and Koji Zettsu. 2024. SimLLM: Detecting Sentences Generated by Large Language Models Using Similarity between the Generation and its Regeneration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22340–22352, Miami… find
  45. Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Krzysztof Wróbel, and Adrian Gwoździej. 2025. Bielik v3 small: Technical report. find
  46. Piotr Przybyła. 2022. LAMBO: Layered Approach to Multi-level BOundary identification. find
  47. Piotr Przybyła, Nicolau Duran-Silva, and Santiago Egea-Gómez. 2023. I’ve Seen Things You Machines Wouldn’t Believe: Measuring Content Predictability to Identify Automatically-Generated Text. In Proceedings of the 5th Workshop on Iberian Languages Evaluation Forum (IberLEF 2023), Jaén, Spain. CEUR Wo… find
  48. Piotr Przybyła, Alexander Shvets, and Horacio Saggion. 2024. Verifying the robustness of automatic credibility assessment. Natural Language Processing, 31(5):1134 – 1162. find
  49. Piotr Przybyła, Jakub Strebeyko, and Alina Wróblewska. 2025. find
  50. PolEval 2025 task 1 śmigiel: Spotting machine-generated text from LLMs for Polish. In Proceedings of the PolEval 2025 Workshop, pages 5–15, Warsaw. Institute of Computer Science PAS and Association for Computational Linguistics. find
  51. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language Models are Unsupervised Multitask Learners. Technical report, OpenAI. find
  52. Juan Diego Rodriguez, Todd Hay, David Gros, Zain Shamsi, and Ravi Srinivasan. 2022. Cross-Domain Detection of GPT-2-Generated Technical Text. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1213–… find
  53. Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and Ireneusz Gawlik. 2020. KLEJ: Comprehensive benchmark for polish language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1191–1201, Online. Association for Computational Linguistics. find
  54. Areg Mikael Sarvazyan, José Ángel González, Marc Franco Salvador, Francisco Rangel, Berta Chulvi, and Paolo Rosso. 2023. Overview of the AuTexTification 2023 Shared Task: Detection and Attribution of Machine-Generated Text in Multiple Domains. In Procesamiento del Lenguaje Natural, Jaén, Spain. find
  55. Areg Mikael Sarvazyan, José Ángel González, Francisco Rangel, Paolo Rosso, and Marc Franco-Salvador. 2024. Overview of IberAuTex-Tification at IberLEF 2024: Detection and Attribution of Machine-Generated Text on Languages of the Iberian Peninsula. Procesamiento del Lenguaje Natural, Revista, (73):42… find
  56. Aditya Shah, Prateek Ranka, Urmi Dedhia, Shruti Prasad, Siddhi Muni, and Kiran Bhowmick. 2023. Detecting and Unmasking AI-Generated Texts through Explainable Artificial Intelligence using Stylistic Features. International Journal of Advanced Computer Science and Applications, 14(10). find
  57. Tatiana Shamardina, Vladislav Mikhailov, Daniil Chernianskii, Alena Fenogenova, Marat Saidov, Anastasiya Valeeva, Tatiana Shavrina, Ivan Smurov, Elena Tutubalina, and Ekaterina Artemova. 2022. Findings of the the ruatd shared task 2022 on artificial text detection in russian. In Computational Lingui… find
  58. Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. 2023. Intrinsic dimension estimation for robust detection of AI-generated texts. In Proceedings of the 37th International Conference on Neural… find
  59. Christoforos Vasilatos, Manaar Alam, Talal Rahwan, Yasir Zaki, and Michail Maniatakos. 2025. HowkGPT: Investigating the Detection of ChatGPT-generated University Student Homework through Context-Aware Perplexity Analysis. find
  60. Saranya Venkatraman, Adaku Uchendu, and Dongwon Lee. 2024. GPT-who: An Information Density-based Machine-Generated Text Detector. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 103–115, Mexico City, Mexico. Association for Computational Linguistics. find
  61. Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2024. Ghostbuster: Detecting Text Ghostwritten by Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)… find
  62. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, and Thomas Arnold. 2024. SemEval-2024 task 8: Multidomain, multimodel and multilingual machine-generated text detection. In Proceedings of the 18th Internatio… find
  63. Yuxia Wang, Artem Shelmanov, Jonibek Mansurov, Akim Tsvigun, Vladislav Mikhailov, Rui Xing, Zhuohan Xie, Jiahui Geng, Giovanni Puccetti, Ekaterina Artemova, Jinyan Su, Minh Ngoc Ta, Mervat Abassy, Kareem Ashraf Elozeiri, Saad El Dine Ahmed El Etter, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar,… find
  64. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003–4012, Mar… find our card
  65. Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. 2025. A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions. Computational Linguistics, 51(1):275–338. find
  66. Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2023. LLMDet: A Third Party Large Language Models Generated Text Detection Tool. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2113–2133, Singapore. Association for Computational Linguistics. find
  67. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… find our card
  68. Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023. Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA… find
  69. Biru Zhu, Lifan Yuan, Ganqu Cui, Yangyi Chen, Chong Fu, Bingxiang He, Yangdong Deng, Zhiyuan Liu, Maosong Sun, and Ming Gu. 2023. Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection via Querying ChatGPT. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language P… find
  70. Narolski, Paweł. 2020. Filmweb+. find
  71. Poświata, Rafał. 2024. Open Coursebooks PL. find
  72. Rafał Poświata, Sławomir Dadas, and Michał Perełkiewicz. 2024. Pl-mteb: Polish massive text embedding benchmark. arXiv preprint arXiv:2405.10138. find
  73. Przybyła, Kamil. 2024. dataset. find
  74. Polish movie reviews Wikinews, contributors. 2025. Wikinews. find
  75. Wikipedia, contributors. 2025. Wikipedia, The Free Encyclopedia. find
  76. Wikiźródła, contributors. 2025. Wikiźródła — wolna biblioteka. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/strebeyko-2026-smigiel-dataset-laying-foundations-investigating/