Elamur Research⌕
☰
home / papers / 2026 / DeepPavlov Strikes Back: A Toolkit for Improving LLM Reliability and Trustworthiness

DeepPavlov Strikes Back: A Toolkit for Improving LLM Reliability and Trustworthiness

Paper · 2026 · Evgenii Nikolaev, Timur Ionov, Anna Korzanova et al.

Summary

DeepPavlov 1.1 adapts the long-running open-source DeepPavlov framework to LLM-centric pipelines. LLM systems with retrieval can produce unfaithful, unnecessary or harmful output, and established toolkits such as spaCy, Stanza and Flair offer no components for checking this. The release keeps the classical linguistic and BERT-era modules and adds three multilingual encoder-based safeguards:

  • a span-level detector that flags answer fragments unsupported by the retrieved context;
  • a classifier that recognizes "evergreen" questions, whose answers do not change over time and so may skip RAG retrieval;
  • a toxicity classifier that acts as a fallback when model alignment fails.

The models are trained on PsiloQA, EverGreenQA and TextDetox, and all are built through DeepPavlov's configuration and chainer system. The authors compare them with Qwen2.5 and gpt-oss LLMs used as few-shot judges. The encoders reach accuracy close to or above these LLM baselines while running about two orders of magnitude faster. This makes cheap reliability checks practical in production pipelines. Ukrainian (UK) is among the languages covered by the toxicity model.

Key points

  • Hallucination detection: framed as token/span labeling over passage, question and answer. It is evaluated on PsiloQA (14 languages) with character-level IoU. The baselines are lettuce-detect-base, ModernBERT-base, Qwen2.5 (7B/32B/72B, 3-shot), gpt-oss-120B and FActScore (GPT-4o). The multilingual mmBERT-base checkpoint is reused from the PsiloQA authors. Its parameter count is inconsistent in the paper: 307M in the text, 110M in the table.
  • Toxicity detection: a binary BERT-base classifier trained on TextDetox, which has 5,000 examples for each of 15 languages. The authors split the data 70/10/20 themselves. The model ranks second overall behind the much larger XLM-RoBERTa-large, with the best English F1 (97.00).
  • Speed: all experiments ran on a single Nvidia H200, with LLMs served via vLLM and encoders run with batch size 256. The encoders are roughly 100× faster than the LLMs.
  • Framework: PyTorch backend with HuggingFace AutoModel support. It offers CLI, REST API and Python interfaces, installs via pip install deeppavlov, and is released under Apache 2.0.
  • Limitations stated by the authors:
    • The hallucination detector's span-level quality is still low, especially for summarization.
    • TextDetox labels are noisy.
    • Experiments used no multiple seeds or ablations.
    • There is no native LLM-orchestration integration.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓ITMO UniversitySt Petersburg · Russia4 works · 24 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Evgenii Nikolaev, Timur Ionov, Anna Korzanova, Vasily Konovalov and Maksim Savkin. 2026. DeepPavlov Strikes Back: A Toolkit for Improving LLM Reliability and Trustworthiness. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-demo.2

@inproceedings{nikolaev2026deeppavlov,
  title = {{DeepPavlov Strikes Back: A Toolkit for Improving LLM Reliability and Trustworthiness}},
  author = {Evgenii Nikolaev, Timur Ionov, Anna Korzanova, Vasily Konovalov and Maksim Savkin},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-demo.2},
  url = {https://doi.org/10.18653/v1/2026.eacl-demo.2}
}

References

35 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, and 21… — TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
  2. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: an easy-to-use framework for state-of-the-art NLP. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo…
  3. Mikhail Burtsev, Alexander Seliverstov, Rafael Airapetyan, Mikhail Arkhipov, Dilyara Baymurzina, Nickolay Bushkov, Olga Gureenkova, Taras Khakhulin, Yuri Kuratov, Denis Kuznetsov, Alexey Litinsky, Varvara Logacheva, Alexey Lymar, Valentin Malykh, Maxim Petrov, Vadim Polulyakh, Leonid Pugachev, Alexe… find
  4. Daryna Dementieva, Daniil Moskovskiy, Nikolay Babakov, Abinew Ali Ayele, Naquee Rizwan, Fro-lian Schneider, Xintog Wang, Seid Muhie Yimam, Dmitry Ustalov, Elisei Stakovskii, Alisa Smirnova, Ashraf Elnagar, Animesh Mukherjee, and Alexander Panchenko. 2024. Overview of the multilingual text detoxifica… find
  5. Dmitry Karpov and Vasily Konovalov. 2023. Knowledge transfer between tasks and languages in the multi-task encoder-agnostic transformer-based models. In Computational Linguistics and Intellectual Technologies, volume 2023. find
  6. Ádám Kovács and Gábor Recski. 2025. Lettucedetect: A hallucination detection framework for RAG applications. CoRR, abs/2502.17125. find
  7. Daryna Dementieva, Vitaly Protasov, Nikolay Babakov, Naquee Rizwan, Ilseyar Alimova, Caroline Brun, Vasily Konovalov, Arianna Muti, Chaya Liebeskind, Marina Litvak, Debora Nozza, Shehryaar Shah Khan, Sotaro Takeshita, Natalia Vanetik, Abinew Ali Ayele, Florian Schneider, Xintong Wang, Seid Muhie Yim… find
  8. Nikita Krayko, Ivan Sidorov, Fedor Laputin, Alexander Panchenko, Daria Galimzianova, and Vasily Konovalov. 2025. Rurage: Robust universal rag evaluator for fast and affordable qa performance testing. In Advances in Information Retrieval, pages 135–145, Cham. Springer Nature Switzerland. find
  9. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Proc…
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  11. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  12. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn J. Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. CoRR, abs/2509.06888.
  13. Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. 2024. Enhancing noise robustness of retrieval-augmented language models with adaptive adversarial training. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
  14. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natur…
  15. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1– 6, Melbourne,…
  16. Daniil Moskovskiy, Sergey Pletenev, and Alexander Panchenko. 2024. LLMs to replace crowdsourcing for parallel data creation? the case of text detoxification. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14361– 14373, Miami, Florida, USA. Association for Computation… find
  17. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  18. Viktor Moskvoretskii, Maria Marina, Mikhail Salnikov, Nikolay Ivanov, Sergey Pletenev, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Irina Matthew Honnibal. 2017. spacy 2: Natural language understanding with bloom embeddings, convolu-Elisei Rykov, Valerii Olisov, Maksim Savkin, Artem Vazhents… find
  19. Nikishina, and Alexander Panchenko. 2025. Adaptive retrieval without self-knowledge? bringing uncertainty back home. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6355– 6384, Vienna, Austria. Association for Computational Li…
  20. Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
  21. Elisei Rykov, Kseniia Petrushina, Maksim Savkin, Valerii Olisov, Artem Vazhentsev, Kseniia Titova, Alexander Panchenko, Vasily Konovalov, and Julia Belikova. 2025b. When models lie, we learn: Multilingual span-level hallucination detection with PsiloQA. In Findings of the Association for Computation… find
  22. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. http://openai.com/blog/ chatgpt. ↗
  23. OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
  24. Maksim Savkin and Vasily Konovalov. 2024. Tuning-free discriminative nearest neighbor few-shot intent detection via consecutive knowledge transfer. In Recent Trends in Analysis of Images, Social Networks and Texts, pages 96–110, Cham. Springer Nature Switzerland. find
  25. CoRR, Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 ot… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  26. Maksim Savkin, Anastasia Voznyuk, Fedor Ignatov, Anna Korzanova, Dmitry Karpov, Alexander Popov, and Vasily Konovalov. 2024. DeepPavlov 1.0: Your gateway to advanced NLP models backed by transformers and transfer learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Languag… find
  27. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  28. Jason Phang, Phil Yeres, Jesse Swanson, Haokun Liu, Ian F. Tenney, Phu Mon Htut, Clara Vania, Alex Wang, and Samuel R. Bowman. 2020. jiant 2.0: A software toolkit for research on general-purpose text understanding models. http://jiant.info/. ↗
  29. Sergey Pletenev, Maria Marina, Nikolay Ivanov, Daria Galimzianova, Nikita Krayko, Mikhail Salnikov, Vasily Konovalov, Alexander Panchenko, and Viktor Moskvoretskii. 2025. Will it still be true tomorrow? multilingual evergreen question classification to improve trustworthy QA. In Proceedings of the 2… find
  30. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  31. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July…
  32. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. FreshLLMs: Refreshing large language models with search engine augmentation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1369…
  33. B Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural…
  34. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast,…
  35. Latency Model Params PsiloQA EvergreenQA TextDetox 7B 32B 72B 120B 990 2756 3000 2164 36 130 190 539 74 196 225 675 560M 395M 395M 355M 110M – 8.7 8.8 – – 6.3 5.9 – – 0.4 – – – 6.3 1.4 Transformer encoder models EG-E5-large ModernBERT-base (our) mmBERT-base (our) RoBERTa-large BERT-base (our) Value…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nikolaev-2026-deeppavlov-strikes-back-toolkit-improving/