Elamur Research⌕
☰
home / papers / 2022 / CompLx@SMM4H'22: In-domain pretrained language models for detection of adverse drug reaction mentions in English tweets

CompLx@SMM4H'22: In-domain pretrained language models for detection of adverse drug reaction mentions in English tweets

Paper · 2022 · Orest Xherija and Hojoon Choi

Summary

The CompLx team took part in sub-task 1a of the #SMM4H 2022 Shared Task: deciding whether an English tweet reports an Adverse Drug Reaction (ADR), meaning a harmful side effect of a medication. Tweets are informal, noisy, and full of misspelled drug and symptom names. The mere presence of a drug name says little about the label. The authors deliberately kept the system minimal. They fine-tuned an off-the-shelf RoBERTa model already pretrained on roughly 128 million tweets (cardiffnlp/twitter-roberta-base-mar2022). The input text was not normalized, and the hyperparameters were the Hugging Face Trainer defaults. A feed-forward head on the [CLS] vector chooses between "ADR" and "No ADR". On the hidden evaluation set, the model reached F1 0.652, against an average of 0.562 across all submissions, and it beat the earlier best published F1 of 0.63. The same recipe did poorly on the team's side entries (sub-tasks 3 and 8). The result shows that choosing an in-domain backbone can carry much of the work. The authors released the checkpoint, the code, and a web demo.

Key points

  • Data for 1a: 17,174 training tweets, 909 validation tweets and 10,969 unlabeled evaluation tweets. The team also entered sub-task 3a (medication-change tweets), 3b (WebMD reviews) and 8 (self-reported chronic stress tweets).
  • Training setup:
    • AdamW optimizer, learning rate 5e-5, batch size 16, and a linear schedule with no warm-up.
    • Gradient clipping, FP-16 mixed precision, and label smoothing of 0.1.
    • Maximum sequence length of 128, at most 25 epochs, and early stopping after 3 epochs without improvement.
    • All runs used a single RTX 2080 Ti with 11GB of memory.
  • Results on 1a:
    • Evaluation set: precision 0.737, recall 0.585, F1 0.652.
    • Validation set: F1 0.769. This set was also used for early stopping, so the score is optimistic.
  • Side tasks were weak:
    • 3a: F1 0.061 vs. an average of 0.456.
    • 3b: F1 0.723 vs. 0.818.
    • 8: F1 0.542 vs. 0.750.
    • Recall of 1.0 on 3b and 8 shows the model labeled nearly everything as positive.
  • Released artifacts: the checkpoint roberta-base-adr-smm4h2022 on Hugging Face, the GitHub repository CompLx-SMM4H2022, and an interactive Space called adr-mention-classifier.

Interesting ideas

  • I1. The paper's motivating example shows the same drug name ("vyvanse") in tweets with opposite labels, which demonstrates that keyword cues fail and the label depends on what the sentence says happened to whom.
  • I2. Picking a backbone pretrained on the target register (tweets) gave strong results with zero tuning, which suggests matching the domain matters more than fiddling with settings.
  • I3. Dynamic padding together with batching by sentence length, plus a modest length cap, avoids spending GPU memory on pad tokens and lets a small card handle larger batches.
  • I4. Label smoothing at 0.1 softens the target distribution, so the model is less punished for imperfect or ambiguous gold labels.
  • I5. Two evaluation lessons stand out: the gap between validation F1 (0.769) and held-out F1 (0.652) shows the cost of reusing the validation split for early stopping, and per-class precision/recall exposed degenerate always-positive behaviour that a single score could hide.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): An ADR test can be phrased as a frame check (a medication acts as the cause of an unwanted state in the speaker), so the decision comes with an explanation instead of a keyword hit.
  • I2 (tree): gparse could start from an encoder pretrained on text resembling its target genres (fables, school texts, social media) before parser training.
  • I3 (learn): The same batching trick can speed up gparse training runs on short treebank sentences.
  • I4 (learn): Silver trees agreed by witness parsers still contain errors, and smoothed targets could keep the student parser from overfitting to them.
  • I5 (learn): Keep a split never touched during model selection, and add a negative control such as a constant-label baseline so collapsed models are flagged automatically.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksMedicine · 241 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Orest Xherija and Hojoon Choi. 2022. CompLx@SMM4H'22: In-domain pretrained language models for detection of adverse drug reaction mentions in English tweets. Proceedings of the Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{xherija2022complx,
  title = {{CompLx@SMM4H'22: In-domain pretrained language models for detection of adverse drug reaction mentions in English tweets}},
  author = {Orest Xherija and Hojoon Choi},
  booktitle = {Proceedings of the Seventh Workshop on Social Media Mining for Health Applications, Workshop \& Shared Task},
  year = {2022},
  url = {https://aclanthology.org/2022.smm4h-1.47/}
}

References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  3. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. ↗
  4. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  5. Like Hui and Mikhail Belkin. 2021. Evaluation of neural architectures trained with square loss vs cross-entropy in classification. In International Conference on Learning Representations. find
  6. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8:64–77.
  7. Finally, one could aim to improve performance Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021. by modifying aspects of the objective function. AEDA: An Easier Data Augmentation Technique for 179 find
  8. Text Classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2748–2754, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  9. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. In International Conference on Learning Representations. ↗
  10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692. ↗
  11. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations. find
  12. Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collados. 2022. TimeLMs: Diachronic Language Models from Twitter. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 251–260, Dublin, Ireland. Association for Computational Linguistics. ↗
  13. Arjun Magge, Elena Tutubalina, Zulfat Miftahutdinov, Ilseyar Alimova, Anne Dirkson, Suzan Verberne, Davy Weissenbacher, and Graciela Gonzalez-Hernandez. 2021. DeepADEMiner: a deep learning pharmacovigilance pipeline for extraction and normalization of adverse drug event mentions on twitter. Journal of the American Medical Informatics Association, 28(10):2184–2192. ↗
  14. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed Precision Training. In International Conference on Learning Representations. ↗
  15. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the Difficulty of Training Recurrent Neural Networks. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1310–1318, Atlanta, GA, USA. PMLR. ↗
  16. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. ↗
  17. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  18. Alec Radford, Karthik Narasimha, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. Blog post, OpenAI. find
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  20. Michael Skinner. 2018. Product Categorization with LSTMs and Balanced Pooling Views. In SIGIR 2018 Workshop on eCommerce (ECOM 18), SIGIR ’18, Ann Arbor, MI, USA. ACM. find
  21. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826. ↗
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  23. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  24. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. ↗
  25. Jason Wei and Kai Zou. 2019. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382–6388, Hong Kong, China. Association for Computational Linguistics. ↗
  26. Davy Weissenbacher, Ari Z. Klein, Luis Gascó, Darryl Estrada-Zavala, Martin Krallinger, Yuting Guo, Yao Ge, Abeed Sarker, Ana Lucia Schmidt, Raul Rodriguez-Esteban, Mathias Leddin, Arjun Magge, Juan M. Banda, Vera Davydova, Elena Tutubalina, and Graciela Gonzalez-Hernandez. 2022. Overview of the Seventh Social Media Mining for Health Applications (#SMM4H) Shared Tasks at COLING 2022. In Proceedings of the Seventh Social Media Mining for Health (#SMM4H) Workshop and Shared Task, Gyeongju, Republic of Korea. Association for Computational Linguistics. find
  27. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xherija-2022-complx-smm4h-22-domain-pretrained/