Elamur Research⌕
☰
home / papers / 2026 / Active Learning for Corpus Refinement: Cost-Effective Preprocessing to Improve Validity of Applied Quantitative Text Analysis

Active Learning for Corpus Refinement: Cost-Effective Preprocessing to Improve Validity of Applied Quantitative Text Analysis

Paper · 2026 · Jakob Steglich and Stephan Poppe

Summary

Social scientists often build text corpora by keyword search. This pulls in documents where the term carries an unrelated meaning, such as "depression" in an economic sense or "schizophrenia" used metaphorically, and that weakens the validity of later measurements. Steglich and Poppe propose integrating corpus cleaning into the analysis pipeline through uncertainty-based active learning with a transformer classifier. A human expert labels the snippets the model is least sure about. These borderline cases also push the researcher to state explicit inclusion and exclusion rules.

They tested the approach on German newspaper snippets about depression and schizophrenia, keeping only medical uses. As a downstream check, they computed embedding-based stigma scores following Best and Arseniev-Koehler (2023). The relevance classifiers reached an F1 of about 0.8 after 100–150 labeled snippets and beat both random sampling and a weakly supervised baseline. Filtering barely changed the depression results but raised measured stigma for schizophrenia. Because the impact of filtering cannot be known in advance, systematic refinement is a worthwhile safeguard.

Key points

  • The classifiers achieved F1 ≈ 0.8 with only 100–150 labeled snippets. Hyperparameter tuning added modest further gains.

  • Active learning outperformed random sampling and cosine-similarity bootstrapping across all iterations, on both F1-minority and F1-macro.

  • Stigma was measured as the cosine similarity between disease embeddings and a stigma dimension. That dimension was built from anchor terms, such as negative versus positive character traits.

  • After filtering, schizophrenia stigma scores increased (d = 0.23), while depression scores stayed largely stable. The depression–schizophrenia stigma gap widened, with its effect size growing by 0.20.

  • The authors list several limitations:

    • test sets of only 100 snippets per condition
    • no confidence intervals
    • a single annotator
    • one downstream task

    The code is published at github.com/jakobstgl/active-learning-corpus-refinement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Jakob Steglich and Stephan Poppe. 2026. Active Learning for Corpus Refinement: Cost-Effective Preprocessing to Improve Validity of Applied Quantitative Text Analysis. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.70

@inproceedings{steglich2026active,
  title = {{Active Learning for Corpus Refinement: Cost-Effective Preprocessing to Improve Validity of Applied Quantitative Text Analysis}},
  author = {Jakob Steglich and Stephan Poppe},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.70},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.70}
}

References

54 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent Anti-Muslim Bias in Large Language Models. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298–306.
  2. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, pages 2623–2631, New York, NY, USA. Ass…
  3. Alina Arseniev-Koehler and Jacob G. Foster. 2022. Machine Learning as a Model for Cultural Learning: Teaching an Algorithm What it Means to be Fat. Sociological Methods & Research, 51(4):1484–1539. find
  4. Satanjeev Banerjee and Ted Pedersen. 2002. An Adapted Lesk Algorithm for Word Sense Disambiguation Using WordNet. In Computational Linguistics and Intelligent Text Processing, pages 136–145, Berlin, Heidelberg. Springer. find
  5. Rachel Kahn Best and Alina Arseniev-Koehler. 2023. The Stigma of Diseases: Unequal Burden, Uneven Decline. American Sociological Review, 88(5):938– 969. find
  6. Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: a survey. In Proceedings of the thirtieth international joint conference on artificial intelligence, IJCAI-21, pages 4330–4338. International Joint Conferences on Artificia… find
  7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135–146. Place: Cambridge, MA Publisher: MIT Press.
  8. Andrei Boutyline and Alina Arseniev-Koehler. 2025. Meaning in hyperspace: Word embeddings as tools for cultural measurement. Annual Review of Sociology, 51. find
  9. Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, and Sanjeev Arora. 2018. A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page… find
  10. Patrick W. Corrigan, Fred E. Markowitz, and Amy C. Watson. 2004. Structural Levels of Mental Illness Stigma and Discrimination. Schizophrenia Bulletin, 30(3):481–491. find
  11. Carolin Kilian, Jakob Manthey, Sinclair Carr, Franz Hanschmidt, Jürgen Rehm, Sven Speerforck, and Georg Schomerus. 2021. Stigmatization of people with alcohol use disorders: An updated systematic review of population studies. Alcoholism, Clinical and Experimental Research, 45(5):899–911. find
  12. Xun Deng, Wenjie Wang, Fuli Feng, Hanwang Zhang, Xiangnan He, and Yong Liao. 2023. Counterfactual Active Learning for Out-of-Distribution Generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11362–11377, Toronto, Ca… find
  13. Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. 2020. Captum: A unified and generic model interpretability library for PyTorch. arXiv preprint. ArXiv:2009.0… — [arXiv:2009.07896](https://arxiv.org/abs/2009.07896) find
  14. Paul DiMaggio, Manish Nag, and David Blei. 2013. Exploiting affinities between topic modeling and the sociological perspective on culture: Application to newspaper coverage of U.S. government arts funding. Poetics, 41(6):570–606. find
  15. Michalis Korakakis, Andreas Vlachos, and Adrian Weller. 2024. ALVIN: Active Learning Via INterpolation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22715–22728, Miami, Florida, USA. Association for Computational Linguistics. find
  16. Satish Gojare, Rahul Joshi, and Dhanashree Gaigaware. 2015. Analysis and Design of Selenium WebDriver Automation Testing Framework. Procedia Computer Science, 50:341–346. find
  17. Justin Grimmer, Margaret E. Roberts, and Brandon M. Stewart. 2022. Text as data: a new framework for machine learning and the social sciences. Princeton University Press, Princeton. OCLC: on1295105650. find
  18. Uri Hanani, Bracha Shapira, and Peretz Shoval. 2001. Information filtering: Overview of issues, research and systems. User modeling and user-adapted interaction, 11(3):203–259. find
  19. David Hanny, Sebastian Schmidt, and Bernd Resch. 2024. Active Learning for Identifying Disaster-Related Tweets: A Comparison with Keyword Filtering and Generic Fine-Tuning. In Intelligent Systems and Applications, pages 126–142, Cham. Springer Nature Switzerland. find
  20. Pieter Floris Jacobs, Gideon Maillette de Buy Wenniger, Marco Wiering, and Lambert Schomaker. 2022. Active Learning for Reducing Labeling Effort in Text Classification Tasks. In Artificial Intelligence and Machine Learning, pages 3–29, Cham. Springer International Publishing. find
  21. Andrey Kapitanov, Ilona Kapitanova, Vladimir Troy-anovskiy, Vladimir Ilyushechkin, and Ekaterina Dorogova. 2019. Clustering of Word Contexts as a Method of Eliminating Polysemy of Words. In 2019 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), pages 1… find
  22. Austin C. Kozlowski, Matt Taddy, and James A. Evans. 2019. The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings. American Sociological Review, 84(5):905–949. Publisher: SAGE Publications Inc. find
  23. Punit Kumar and Atul Gupta. 2020. Active Learning Query Strategies for Classification, Regression, and Clustering: A Survey. Journal of Computer Science and Technology, 35(4):913–945. find
  24. Marc Kupietz, Cyril Belica, Holger Keibel, and Andreas Witt. 2010. The German Reference Corpus DeReKo: A Primordial Sample for Linguistic Research. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC‘10), Valletta, Malta. European Language Resources Asso… find
  25. David D. Lewis and William A. Gale. 1994. A Sequential Algorithm for Training Text Classifiers. In SIGIR ’94, pages 3–12, London. Springer. find
  26. Bruce G. Link and Jo Phelan. 2014. Stigma power. find
  27. Social Science & Medicine (1982), 103:24–32. find
  28. Bruce G. Link and Jo C. Phelan. 2001. Conceptualizing stigma. Annual Review of Sociology, 27:363–385. Place: US Publisher: Annual Reviews. find
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint. ArXiv:1907.11692 [cs].
  30. Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. 2021. Active Learning by Acquiring Contrastive Examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 650–663, Online and Punta Cana, Dominican Republic. Association for Com… find
  31. Blake Miller, Fridolin Linder, and Walter R. Mebane. 2020. Active Learning Approaches for Labeling Text: Review and Assessment of the Performance of Active Learning Approaches. Political Analysis, 28(4):532– 551. find
  32. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having Beer after Prayer? Measuring Cultural Bias in Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Associatio…
  33. Laura K. Nelson. 2021. Leveraging the alignment between machine learning and intersectionality: Using word embeddings to measure intersectional experiences of the nineteenth century U.S. South. Poetics, 88:101539. find
  34. Bernice A. Pescosolido, Andrew Halpern-Manners, Liying Luo, and Brea Perry. 2021. Trends in Public Stigma of Mental Illness in the US, 1996-2018. JAMA network open, 4(12):e2140202. find
  35. Jo C. Phelan, Bruce G. Link, and John F. Dovidio. 2008. Stigma and prejudice: One animal or two? Social Science & Medicine, 67(3):358–367. find
  36. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  37. Pedro L. Rodriguez, Arthur Spirling, and Brandon Stewart. 2023a. conText: ’a la Carte’ on Text (ConText) Embedding Regression. R package version 1.4.3. find
  38. Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart. 2023b. Embedding Regression: Models for Context-Specific Description and Inference. American Political Science Review, 117(4):1255–1274. find
  39. Nicholas Roy and Andrew McCallum. 2001. Toward Optimal Active Learning through Sampling Estimation of Error Reduction. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pages 441–448, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  40. Georg Schomerus, Stephanie Schindler, Christian Sander, Eva Baumann, and Matthias C. Angermeyer. 2022. Changes in mental illness stigma over 30 years - Improvement, persistence, or deterioration? European Psychiatry: The Journal of the Association of European Psychiatrists, 65(1):e78. find
  41. Christopher Schröder, Lydia Müller, Andreas Niekler, and Martin Potthast. 2023. Small-Text: Active Learning for Text Classification in Python. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 84–95. ArXiv:210… find
  42. Christopher Schröder, Andreas Niekler, and Martin Potthast. 2022. Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2194–2203, Dublin, Ireland. Association for Computational Linguistics. find
  43. M. Sittner, T. Rechenberg, S. Speerforck, M. C. Angermeyer, and G. Schomerus. 2024. ‘Broken souls’ vs. ‘mad ax man’ – changes in the portrayal of depression and schizophrenia in the German media over 10 years. Epidemiology and Psychiatric Sciences, 33:e37. find
  44. Dustin S. Stoltz and Marshall A. Taylor. 2021. Cultural cartography with word embeddings. Poetics, 88:101567. find
  45. Dustin S. Stoltz and Marshall A. Taylor. 2024. Mapping texts: computational text analysis for the social sciences. Computational social science. Oxford University Press, New York, NY. find
  46. Dustin S. Stoltz, Marshall A. Taylor, and Jennifer S. K. Dudley. 2024. A Tool Kit for Relation Induction in Text Analysis. Sociological Methods & Research, page 00491241241233242. Publisher: SAGE Publications Inc. find
  47. Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient Few-Shot Learning Without Prompts. arXiv preprint. ArXiv:2209.11055. — [arXiv:2209.11055](https://arxiv.org/abs/2209.11055) find
  48. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need.
  49. Francisco Viveros-Jiménez, Alexander Gelbukh, and Grigori Sidorov. 2013. Simple Window Selection Strategies for the Simplified Lesk Algorithm for Word Sense Disambiguation. In Advances in Artificial Intelligence and Its Applications, pages 217–227, Berlin, Heidelberg. Springer. find
  50. Yue Wang, Kai Zheng, Hua Xu, and Qiaozhu Mei. 2018. Interactive medical word sense disambiguation through informed learning. Journal of the American Medical Informatics Association, 25(7):800–808. find
  51. Elisa M. Wirsching, Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart. 2025. Multilanguage Word Embeddings for Social Scientists: Estimation, Inference, and Validation Resources for 157 Languages. Political Analysis, 33(2):156–163. find
  52. World Health Organization. 2019/2021. International Classification of Diseases, Eleventh Revision (ICD-11). WHO. find
  53. Shujian Zhang, Chengyue Gong, Xingchao Liu, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. 2022. ALLSH: Active Learning Guided by Local Sensitivity and Hardness. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1328–1342, Seattle, United States. Association for Computat… find
  54. Jingbo Zhu and Eduard Hovy. 2007. Active Learning for Word Sense Disambiguation with Methods for Addressing the Class Imbalance Problem. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 78… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/steglich-2026-active-learning-corpus-refinement-cost/