📄 BIT@UA at #SMM4H 2024 Tasks 1 and 5: finding adverse drug events and children's medical disorders in English tweets

Summary

The paper reports the BIT@UA team's systems for two #SMM4H 2024 shared tasks on English tweets. Task 1 required locating adverse drug event (ADE) mentions and mapping each one to a MedDRA identifier. Task 5 required separating tweets where the author says their own child has a medical disorder from tweets that only mention a disorder.

For Task 1, the team built a two-stage pipeline:

• Detection: a RoBERTa model, already trained on PubMed and on disease-annotated corpora, was fine-tuned to tag ADE tokens with a reduced IO scheme.
• Normalization: a random forest over count and TF-IDF features assigned the MedDRA code.

For Task 5, several classical scikit-learn classifiers and XGBoost were compared with a fine-tuned bert-base-uncased model.

Official test results:

• Task 1: F1 of 0.397 for recognition and 0.295 for normalization, slightly above the participants' mean and median but below the organizers' baseline.
• Task 5: BERT reached 0.840 F1.

The work shows that ADE normalization is hard on informal text. It also shows that fine-tuned transformers clearly beat sparse-feature models when classifying health-related tweets.

Key points

• Task 1 data: 18,185 training tweets, with only 1,239 containing ADEs (1,711 entities in total). The development set had 965 tweets with 87 entities, and the blind test set had 11,799 tweets.
• Task 1 model and filtering: the NER model was a Hugging Face checkpoint (raynardj, trained on NCBI and BC5CDR disease data). An added document-level filter that discarded ADE-free tweets did not help and was dropped.
• Task 1 dev–test gap: normalization F1 fell from 0.453 on development to 0.295 on test. The authors partly attribute this to the classifier only being able to output MedDRA codes seen in training. Test baseline scores were 0.481 for NER and 0.439 for normalization.
• Task 5 data: 7,398 training tweets (2,280 positive), 389 development tweets and 10,000 test tweets.
• Task 5 results: in 5-fold cross-validation, XGBoost with count features was the best classical model (F1 0.7431). BERT scored about 0.82–0.85 depending on the number of epochs. The 8-epoch model was submitted because it scored 0.8968 on development.

Interesting ideas

• I1. The tagging scheme was chosen after inspecting the data: IO tags replaced IOB because no two entities were ever adjacent, which reduces the label space at no cost.
• I2. Transfer was staged through a related intermediate task: general biomedical pretraining, then disease NER, then ADE fine-tuning, so each stage moves closer to the target.
• I3. A normalizer that treats identifiers as a closed set of class labels cannot output unseen codes, and this showed up as a large dev-to-test drop.
• I4. Task 5 is a contrast between merely mentioning a condition and reporting that someone close to the author has it.
• I5. Scores were reported against the mean, median and baseline of all participants, and dev versus test were compared to expose overfitting.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): before choosing chunk or multiword encodings in the en crate, check the corpus for adjacent spans and use the smallest label set that can still represent them.
• I2 (learn): train gparse or the taggers through intermediate silver tasks that lie near the target domain, instead of jumping straight from generic data to the final task.
• I3 (world): link mentions to WordNet/Wikidata through lexical or graph matching rather than a classifier over previously seen IDs, so rare concepts stay reachable and each link can be explained.
• I4 (frames): use such pairs as tests that the frame builder correctly assigns the holder ("my son has X") versus a generic mention, a distinction bag-of-words models handle poorly.
• I5 (learn): always report a trivial baseline and the held-out gap next to headline scores, as a cheap negative control on evaluation claims.