Elamur Research⌕
☰
home / papers / 2026 / Data Augmentation Based on Selective Masking of Language Models for One Health Context

Data Augmentation Based on Selective Masking of Language Models for One Health Context

Paper · 2026 · Youssef Mahdoubi, Najlae Idrissi, Mathieu Roche et al.

Summary

Text-based epidemiological monitoring in the One Health setting has little labeled data, which limits supervised classifiers. The authors test two applications: thematic classification (syndromic surveillance, biomedical, and plant health texts) and detection of epidemic misinformation. Their remedy is data augmentation through selective masking. A controlled share of words in each training text is masked, and a language model fills the gaps, producing one new labeled example per instance.

They build two families of strategies:

  • Lexical masking targets domain-lexicon terms.
  • Non-lexical masking hides words outside the lexicon so that domain information survives.

Each family comes in a plain variant and a TF-IDF-weighted variant. They then search for the best masking rate and compare the strategies with LLM paraphrasing (Mistral-7B, GPT-Neo-1.3B), using BERT Base fine-tuned on five English datasets.

At a 20% masking rate, selective masking beat both the LLM rewrites and the no-augmentation baseline on three of the five datasets. This suggests that a cheap masked-LM approach can rival costly LLM generation in low-resource health text classification.

Key points

  • The four strategies are Aug-SM-Lex, Aug-SM-NonLex, Aug-SM-Lex-TFIDF and Aug-SM-NonLex-TFIDF.
    • Lexical variants mask domain-lexicon 1- to 3-grams.
    • The TF-IDF versions restrict masking to low-scoring, less informative terms.
    • Aug-SM-Lex-TFIDF gives lexicon terms artificial low scores (0.0–0.2), which makes them candidates for masking.
  • The best masking rate was 20%. When the lexicon-driven selection falls short of the target rate, the remaining slots are filled with randomly chosen words.
  • In the main results table, one dataset's baseline accuracy rose from 70.79% to 74.57% with Aug-SM-Lex, but GPT-Neo-1.3B reached 76.28% on that dataset. On another dataset, no augmentation method beat the baseline (94.00% accuracy).
  • Augmentation is applied only to training data. Semantic fidelity and lexical diversity of the generated texts are checked with BERTScore and ROUGE.
  • Limitations are English-only data, reliance on domain lexicons, and evaluation on classification alone. Planned future work includes tests with RoBERTa and BioBERT, and using LLMs to generate lexicons and predict masked tokens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontpellierMontpellier · France3 works · 1,582 citations🎓Université Sultan Moulay SlimaneBeni Mellal · Morocco2 works · 35 citations

Also on the byline: AgroParisTech · Centre de Coopération Internationale en Recherche Agronomique pour le Développement · Institut National de Recherche pour l'Agriculture, l'Alimentation et l'Environnement · Territoires, Environnement, Télédétection et Information Spatiale.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksEpidemiology · 7 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Youssef Mahdoubi, Najlae Idrissi, Mathieu Roche and Sarah Valentin. 2026. Data Augmentation Based on Selective Masking of Language Models for One Health Context. Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.healing-1.23

@inproceedings{mahdoubi2026data,
  title = {{Data Augmentation Based on Selective Masking of Language Models for One Health Context}},
  author = {Youssef Mahdoubi, Najlae Idrissi, Mathieu Roche and Sarah Valentin},
  booktitle = {Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.healing-1.23},
  url = {https://doi.org/10.18653/v1/2026.healing-1.23}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akiko Aizawa. 2003. An information-theoretic perspective of tf-idf measures. Information Processing & Management, 39(1):45–65. find
  2. Laith Alzubaidi, Jinshuai Bai, Aiman Al-Sabaawi, Jose Santamaría, A. S. Albahri, Bashar Sami Nayyef Al-dabbagh, Mohammed A. Fadhel, Mohamed Manoufali, Jinglan Zhang, Ali H. Al-Timemy, and 1 others. 2023. A survey on deep learning tools dealing with data scarcity: Definitions, challenges, solutions,… find
  3. Anas Belfathi, Ygor Gallina, Nicolas Hernandez, Laura Monceaux, and Richard Dufour. 2024. Adaptation des modèles de langue à des domaines de spécialité par un masquage sélectif fondé sur le genre et les caractéristiques thématiques. In Actes de la 31ème Conf. sur le Traitement Automatique des Langue… find
  4. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. Zenodo. find
  5. Mariya Borovikova, Arnaud Ferré, Robert Bossy, Mathieu Roche, and Claire Nédellec. 2023. Could keyword masking strategy improve language model? In Natural Language Processing and Information Systems, pages 271–284. Springer, Cham. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  7. Limeng Cui and Dongwon Lee. 2020. Coaid: Covid-19 healthcare misinformation dataset. arXiv preprint. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT, pages 4171–4186.
  9. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proc. of the 2018 Conf. on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. ACL. find
  10. FalgunPatel19. 2022. Medical text dataset - cancer document classification. Kaggle. Retrieved from https://www.kaggle.com/datasets/falgunipatel19/ biomedical-text-publication-classification. Accessed September 26, 2025. ↗
  11. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proc. of the Tenth ACM SIGKDD International Conf. on Knowledge Discovery and Data Mining (KDD’04), Seattle, WA, USA. find
  12. Tatsuya Ishigaki, Yui Uehara, Goran Topić, and Hiroya Takamura. 2023. Pretraining language- and domain-specific BERT on automatically translated text. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 548–555. INCOMA Ltd., Shoumen, Bulgaria… find
  13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  14. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240.
  15. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. ACL.
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint.
  17. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conf. on Learning Representations, Workshop Track Proc., Scottsdale, Arizona, USA.
  18. Oxford Reference. 2010. A dictionary of biomedicine. Retrieved from https://www.oxfordreference.com/display/10.1093/acref/9780199549351. 001.0001/acref-9780199549351. Accessed September 24, 2025. — [doi:10.1093/acref/9780199549351](https://doi.org/10.1093/acref/9780199549351) ↗
  19. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 20… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  20. Parth Patwa, Shivam Sharma, Srinivas PYKL, Vineeth Guptha, Gitanjali Kumari, Md. Shad Akhtar, Asif Ekbal, Amitava Das, and Tanmoy Chakraborty. 2020. Fighting an infodemic: COVID-19 fake news dataset. CoRR, abs/2011.03327. — [arXiv:2011.03327](https://arxiv.org/abs/2011.03327) find
  21. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proc. of the 2014 Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. ACL.
  22. Gabriele Pergola, Elena Kochkina, Lin Gui, Maria Liakata, and Yulan He. 2021. Boosting low-resource biomedical qa via entity-aware masking strategies. In Proc. of the 16th Conf. of the European Chapter of the Association for Computational Linguistics (EACL), pages 1977–1985. find
  23. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proc. of the 29th International Conf. on Neural Information Processing Systems - Volume 1, NIPS’15, page 649–657, Cambridge, MA, USA. MIT Press.
  24. Huanhuan Zhao, Haihua Chen, Thomas A. Ruggles, Yunhe Feng, Debjani Singh, and Hong-Jun Yoon. 2024. Improving text classification with large language model-based data augmentation. Electronics, 13(13). find
  25. Joana C. Prata, Ana Isabel Ribeiro, and Teresa Rocha-Santos. 2022. Chapter 1—an introduction to the concept of one health. In One Health, pages 1–31. Academic Press. find
  26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  27. H. Rasamoelina, L. P. Veerapa-Mangroo, S. A. Bedja, and M. Roche. 2023. Mots-clés pour padi-web mis en place dans l’océan indien. CIRAD Dataverse. find
  28. Sarah Valentin, Elena Arsevska, Julien Rabatel, Sylvain Falala, Alizé Mercier, Renaud Lancelot, and Mathieu Roche. 2021. Padi-web 3.0: A new framework for extracting and disseminating fine-grained information from the news for animal disease surveillance. One Health, 13:100357. find
  29. William Yang Wang and Diyi Yang. 2015. That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using \#petpeeve tweets. In Proc. of the 2015 Conf. on Empirical Methods in Natural Language Processing, pages 2557–…
  30. Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing and the 9th International Joint Conf. on Natural Language Processing, EMNLP-IJCNLP, pages 6381–…
  31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In Proc. of EMNLP 2020: System Demonstrations, pages 38–45.
  32. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conf. on Learning Representations, Addis Ababa, Ethiopia. OpenReview.net. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mahdoubi-2026-data-augmentation-based-selective-masking/