📄 HAMiSoN-Generative at ClimateActivism 2024: Stance Detection using generative large language models

Summary

This paper describes the UNED team's entry to the stance detection subtask of the CASE 2024 shared task on Climate Activism. The task is to label tweets about climate activism as Support, Oppose or Neutral. The authors compared four variants built on Llama 2 7B:

• a Chat model used generatively (seq2seq) with Meta's prompt format;
• two Chat models whose output layer was swapped for a 3-way linear classifier, one with formatted prompts and one with raw text;
• the base 7B model with the same classifier head.

All variants were fine-tuned with LoRA and 4-bit quantization. Tweets were preprocessed by removing URLs and user mentions and splitting hashtags into words. The authors also varied the training data: a 70/30 split of the CASE train set versus the full train set, each with or without SemEval-2016 stance data. Every fine-tuned model clearly beat the ClimateBERT baseline, and the classifier-head models outperformed generation. The best run placed 7th, close to the winner. Extra out-of-domain data rarely helped, which the authors attribute to topical and temporal mismatch between the datasets.

Key points

• Best result: the 7B Chat – clf no prompt model trained on the full CASE set reached macro-F1 0.7366, ranking 7th and only 0.0117 behind the top system (0.7483). The ClimateBERT baseline scored 0.545.
• Classifier head vs. generation: replacing the generative output layer with a classification layer generally beat the seq2seq setup (around 0.70). Zero-shot Llama 2 7B Chat scored just 0.5685.
• Training setup: LoRA used r = 64, α = 16 and dropout 0.1 with 4-bit quantization. This left about 350M trainable parameters, roughly 4.5% of the full model.
• Effect of extra data: adding SemEval-2016 improved only 2 of 8 runs, and using the full CASE set instead of the split improved only 3 of 8. Adding only the climate-related SemEval subset after the competition gave 0.7346, nearly identical to the best run.
• Error analysis: almost all errors involve the Neutral class, and Support/Oppose confusions are rare. The authors note lexical drift between the datasets: CASE contains terms like Greta Thunberg, #FridaysForFuture and the Ukrainian-Russian war, which do not appear in SemEval.