Elamur Research⌕
☰
home / papers / 2024 / DILAB at #SMM4H 2024: RoBERTa Ensemble for Identifying Children's Medical Disorders in English Tweets

DILAB at #SMM4H 2024: RoBERTa Ensemble for Identifying Children's Medical Disorders in English Tweets

Paper · 2024 · Azmine Toushik Wasi and Sheikh Rahman

Summary

The paper describes the DILAB team's entry for SMM4H 2024 Task 5, a binary classification of English tweets. A positive tweet comes from a user who earlier reported a pregnancy and later says their child has ADHD, ASD, delayed speech or asthma. A negative tweet only mentions one of these disorders. The authors fine-tuned several pretrained encoders, each with a small MLP classifier on top: BERT-base, RoBERTa-base, RoBERTa-large and a Twitter-specific RoBERTa tuned for sentiment. They varied the classifier depth and the dropout rate. Validation scores swung noticeably from one epoch to the next, so the authors built an ensemble that votes over predictions from the three best training epochs of a single run, rather than trusting one checkpoint. The Twitter-adapted model was the strongest. The epoch ensemble reached about 0.898 F1 on the test set, slightly above the best single checkpoint, even though that single checkpoint looked better on validation. This suggests that domain-matched pretraining and checkpoint voting help in small, noisy social-media health classification tasks.

Key points

  • Data: 7398 training, 389 validation and 1947 test tweets; no extra data was used.
  • Backbone: cardiffnlp/twitter-roberta-base-sentiment-latest (R-TRBSL, from TimeLMs/TweetNLP, tuned on TweetEval). It beat BERT-base, RoBERTa-base and RoBERTa-large in validation F1.
  • Setup:
    • RobertaTokenizer with lowercased input and a 256-token limit.
    • MLP classifier of width 768 with dropout 0.3, which usually did better than 0.5.
    • BCE loss, Adam, learning rate 1e-5, batch size 8, seed 101, 10 epochs.
  • Test F1: 0.898 for the top-3-epoch ensemble and 0.892 for the best single checkpoint. The task mean was 0.822 and the median 0.901.
  • Precision and recall varied between configurations, while macro-F1 stayed fairly stable.

Interesting ideas

  • I1. Instead of picking one "best" epoch, let several strong checkpoints from the same run vote; this smooths out epoch-to-epoch noise at almost no extra training cost.
  • I2. A small validation set (here 389 items) can crown a checkpoint that later generalizes worse, so stability-oriented selection can beat chasing the top validation number.
  • I3. A base-size model pretrained on in-domain text (tweets) outperformed a larger general-domain model, so domain match can matter more than scale.
  • I4. The authors look at precision and recall separately even when the overall F1 barely moves, since configurations trade one for the other.
  • I5. The task separates a personal report about one's own child from a generic mention of a disorder, which is essentially a question about who experiences the condition.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could keep its top few checkpoints and vote on heads and labels per arc, giving steadier parses and a confidence signal where checkpoints disagree.
  • I2 (learn): evaluation should report spread across checkpoints or seeds and prefer configurations that stay stable, not single peaks on small dev sets.
  • I3 (tree): when parsing informal or dialogue-like text, adapting the encoder to that register may pay off more than moving to a bigger general encoder.
  • I4 (learn): witness parsers for silver data could be chosen to complement each other, high-precision ones for agreement filtering and high-recall ones for coverage.
  • I5 (frames): frames could explicitly mark whether the speaker is a participant (my child has X) or only mentions a topic, an explainable alternative to classifiers that rely on surface keywords.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shahjalal University of Science and TechnologySylhet · Bangladesh4 works · 1 citation

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksSocial Psychology · 330 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Azmine Toushik Wasi and Sheikh Rahman. 2024. DILAB at #SMM4H 2024: RoBERTa Ensemble for Identifying Children's Medical Disorders in English Tweets. Proceedings of the 9th Social Media Mining for Health Research and Applications (SMM4H 2024) Workshop and Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wasi2024dilab,
  title = {{DILAB at \#SMM4H 2024: RoBERTa Ensemble for Identifying Children's Medical Disorders in English Tweets}},
  author = {Azmine Toushik Wasi and Sheikh Rahman},
  booktitle = {Proceedings of the 9th Social Media Mining for Health Research and Applications (SMM4H 2024) Workshop and Shared Tasks},
  year = {2024},
  doi = {10.18653/v1/2024.smm4h-1.3},
  url = {https://doi.org/10.18653/v1/2024.smm4h-1.3}
}

References

10 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dongfang Xu, Guillermo Lopez Garcia, Lisa Raithel, Rolland Roller, Philippe Thomas, Eiji Aramaki, Shuntaro Yada, Pierre Zweigenbaum, Karen O’Connor, Yao Ge, Sudeshna Das, Abeed Sarker, Ari Klein, Lucia Schmidt, Vishakha Sharma, Raul Rodriguez-Esteban, Juan Banda, Ivan Flores Amaro, Sai Tharuni Samineni, and Graciela Gonzalez-Hernandez. 2024. Overview of the 9th social media mining for health applications (#SMM4H) shared tasks at ACL 2024. In Proceedings of the 9th Social Media Mining for Health Applications Workshop and Shared Task, Bangkok, Thailand. Association for Computational Linguistics. find
  2. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020. TweetEval:Unified Benchmark and Comparative Evaluation for Tweet Classification. In Proceedings of Findings of EMNLP.
  3. Jose Camacho-collados, Kiamehr Rezaee, Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthenis Antypas, Joanne Boisson, Luis Espinosa Anke, Fangyu Liu, and Eugenio Martínez Cámara. 2022. TweetNLP: Cutting-edge natural language processing for social media. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–49, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  6. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980. ↗
  7. Ari Z Klein, José Agustín Gutiérrez Gómez, Lisa D Levine, and Graciela Gonzalez-Hernandez. 2024. Using longitudinal twitter data for digital epidemiology of childhood health outcomes: An annotated data set and deep neural network classifiers. Journal of Medical Internet Research, 26:e50652. ↗
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  9. Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collados. 2022. TimeLMs: Diachronic language models from Twitter. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 251–260, Dublin, Ireland. Association for Computational Linguistics. ↗
  10. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. Preprint, arXiv:1910.03771. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wasi-2024-dilab-smm4h-2024-roberta-ensemble/