📄 NLP@VCU: Identifying Adverse Effects in English Tweets for Unbalanced Data

Summary

This short shared-task paper from Virginia Commonwealth University addresses SMM4H 2020 Track 2: deciding whether an English tweet mentions an adverse effect (AE) of a drug. The data are heavily skewed, with roughly one AE tweet per ten non-AE tweets, so the authors focus on how to handle the class imbalance. They clean the tweets by stripping the HTML ampersand entity and double quotes, replacing hashtags, links and usernames with fixed placeholder words, rewriting emojis as descriptive phrases, and lowercasing. Words are then embedded with GloVe vectors trained on Twitter, which beat word2vec and FastText in their trials. Classification is done by a standard CNN with convolution, max-pooling, a dense layer and softmax. They compare three ways to counter the skew: downsampling the negatives, repeating the positives, and Keras class weights. On the development set all three reach about the same F1 (around 0.50), but they balance precision and recall differently. On the hidden test set, downsampling scored best with F1 0.35. The paper gives a compact baseline and shows that development scores may not predict which imbalance strategy will win on new data.

Key points

• Data: 20,544 training tweets (1,903 AE), 5,134 development tweets (474 AE), and 4,759 test tweets. Both labelled sets are about 1:10 positive to negative.
• The best settings, found by experiment, were:
  • downsampling to 1 AE per 4 non-AE tweets;
  • oversampling each AE tweet 6 times;
  • Keras class weights of 10 for AE and 1 for non-AE.
• Development results (P / R / F1):
  • downsampling: 0.59 / 0.43 / 0.50;
  • oversampling: 0.56 / 0.44 / 0.49;
  • class weights: 0.43 / 0.58 / 0.50.
• On the test set, downsampling reached the top F1 of 0.35 and its recall exceeded its precision. The other two runs scored somewhat lower. Precision and recall were released only for the best run.
• Embeddings: GloVe (Twitter) gave the best results among GloVe, word2vec and FastText. Code is available at github.com/NLPatVCU/SMM4H. Planned future work includes character embeddings, RNNs and BERT.

Interesting ideas

• I1. Hashtags, links and user handles are replaced by fixed placeholder words, so noisy, open-ended tokens become a small closed set the model can learn from.
• I2. Emojis are rewritten as short descriptive phrases, which turns a pictogram into ordinary words that existing vocabularies already cover.
• I3. The choice of imbalance remedy mainly moves the precision/recall balance, not F1. Undersampling and oversampling favoured precision, while class weights favoured recall.
• I4. Three methods that tied on development data separated clearly on the test set, a reminder that a single dev split can hide differences.
• I5. Ratio and weight hyperparameters were tuned by a simple sweep, and the optimum (1:4, ×6, 10:1) did not fully restore balance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): the en tokenizer could map URLs, handles and hashtags to typed placeholder tokens with a fixed POS and lemma, so the tagger and parser never meet unknown junk strings in social-media input.
• I2 (tokens): a lookup table from emoji to phrase could feed the tagger normal words, and the world stage could then read the emoji's meaning, such as a feeling, as an explainable state.
• I3 (learn): when training rare-label classifiers, such as repair-rule triggers or rare dependency relations, the balancing method can be picked deliberately according to whether false positives or misses cost more.
• I4 (learn): evaluation should keep a second untouched held-out set, alongside negative controls, before declaring two training variants equivalent.
• I5 (learn): when mixing silver data with scarce gold examples, a sweep of partial reweighting ratios may beat naïvely forcing a 1:1 mix.