📄 HaleLab_NITK@SMM4H'24: Binary classification of English tweets reporting children's medical disorders

Summary

This paper is a short system description for Task 5 of the SMM4H 2024 shared task. The task is to label English tweets from users who have previously announced a pregnancy. A tweet is positive when the parent clearly states that their own child has ADHD, ASD, delayed speech or asthma. It is negative when the tweet only mentions a disorder or does not concern the parent's child at all. The authors used the corpus by Klein et al. (2023) and normalised the tweets heavily before modelling:

• unified disorder names, digits and kinship words
• stripped Twitter-specific noise
• expanded slang and contractions
• lemmatised the text with spaCy

They then fine-tuned RoBERTa-base with an extra dense ReLU layer and a sigmoid output. On the official 10,000-tweet test set the system reached F1 0.87, close to its validation score of 0.88. This is below the published RoBERTa-large baseline (0.927), which was measured on a smaller test set. The work matters because large-scale filtering of parents' first-hand health reports can support pediatric epidemiology and targeted family support.

Key points

• Data split: 7,398 training tweets, 389 validation tweets and 10,000 test tweets (Klein et al., 2023).
• Preprocessing steps:
  • mapped son, daughter and similar words to "child"
  • removed URLs, usernames, hashtags and emojis with the tweet-preprocessor library
  • expanded items like "lol", "thx" and "we're"
  • lowercased the text
  • limited repeated characters to two
  • lemmatised with spaCy
• Model and training: RoBERTa-base via Huggingface, max length 128, Adam, batch size 8, learning rate 1e-5. Training was capped at 10 epochs and halted early if there was no improvement for 4 epochs, with dropout 0.3, on a Colab T4 GPU.
• Validation scores: F1 0.88, precision 0.89, recall 0.88.
• Test scores: F1 0.87 (0.868), precision 0.86, recall 0.88.
• Comparison: the RoBERTa-large baseline from Klein et al. (2024) scored 0.927 on 1,947 tweets, so the two numbers are not directly comparable.

Interesting ideas

• I1. Kinship words with the same task role (son, daughter, child) are collapsed into one placeholder, so the model sees one consistent signal for "the poster's offspring".
• I2. Character elongation is capped at two repeats, which reduces noisy spellings to a small set of variants that can be looked up.
• I3. Chat abbreviations and contractions are expanded into full words before analysis, turning informal input into something closer to standard text.
• I4. The task asks whether the poster's own child has the condition, not whether a disorder is named, which is a question of who holds the condition.
• I5. The authors point out that their score and the baseline score come from test sets of different sizes and from different model sizes. This is a reminder that headline numbers can mislead.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Kin nouns could link to a shared ontology class such as child-of-speaker while keeping their surface lemmas, so entity tracking treats "my son" and "my daughter" alike.
• I2 (tokens): The en crate could apply the same cap before lemma and morphology lookup, so forms like "sooo" or "loooove" still map to dictionary entries.
• I3 (tokens): A small expansion table run before tagging would give the UD parser cleaner tokens for social-media input without retraining.
• I4 (frames): prag could answer this explicitly by checking that the holder of a has-condition frame is coreferent with the speaker's child. This gives a traceable decision instead of keyword matching.
• I5 (learn): Evaluation reports should always pair a system with its baselines on identical test sets and note any mismatch next to the numbers.