📄 SU-NLP at CASE 2021 Task 1: Protest News Detection for English

Summary

The paper presents the Sabancı University team's English-only systems for all four subtasks of the CASE 2021 shared task on protest news. The subtasks are: document-level event detection, sentence-level event detection, event sentence coreference, and token-level extraction of event triggers and arguments.

• Detection (subtasks 1–2): The authors compared simple CNN/RNN baselines with fine-tuned BERT, Albert and RoBERTa. They added a two-channel CNN (CNN-LexStem) that reads both surface words and lemmatized words, and a confidence-gated ensemble that only overrides RoBERTa when it is unsure.
• Coreference (subtask 3): They used a mention-pair approach, pairwise classification followed by clustering, and enlarged the training data with the AESPEN'20 corpus.
• Extraction (subtask 4): They stacked a BiLSTM and a CRF on top of transformer encoders.

The systems placed 3rd in subtasks 1 and 2 and 5th in subtask 3. A submission format error invalidated their subtask 4 entry, but their scores would have ranked it 2nd. The work shows that cheap architectural additions to a strong transformer give consistent gains on socio-political event processing.

Key points

• Detection data: Subtask 1 had 9324 news articles, of which 7407 contained no event. Subtask 2 had 22,825 sentences, only 4210 with an event. Rebalancing the classes did not help, so the natural ratio was kept.
• Baselines: RoBERTa-base was the strongest single model, with test F1 of 0.81 in subtask 1 and 0.82 in subtask 2. RNNs overfit heavily.
• Ensemble: RoBERTa outputs below 0.1 or above 0.9 were accepted as they were. Cases in between went to a grid-searched weighted ensemble, and RoBERTa+LexStem reached test F1 0.84 on subtask 2.
• Coreference: Adding the AESPEN'20 data (404 English articles) to the training set raised classification macro F1 from about 87 to 92 on validation. The best test CoNLL score was 78.66, from the Ensemble trained on the extended data.
• Extraction: The data was 808 sentences with 7 IOB-style labels. RoBERTa-BiLSTM-CRF reached 0.76 test F1, compared with 0.74 for plain RoBERTa. The most frequent confusions were organizer vs participant and place vs fname.

Interesting ideas

• I1. Accept the strong model's prediction at the confident extremes and send only the uncertain middle band to a weighted ensemble; this saves compute while repairing exactly the cases most likely to be wrong.
• I2. Feed surface forms and lemmas as two parallel channels, so a classifier notices an event regardless of tense or inflection while still seeing the original wording.
• I3. To isolate whether a gain comes from the new information or just from extra capacity, train controls that duplicate the same input (Lex+Lex, Stem+Stem) next to the mixed model.
• I4. When making train/validation splits for a clustering task, choose a random seed whose splits match in both the positive-pair ratio and the share of single-cluster documents, so evaluation is not skewed by structure.
• I5. An error analysis shows the hardest argument confusions are between roles with similar context (organizer vs participant, place vs facility), not between distant ones.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver-data agreement could use the primary parser alone when its scores are confident, and call extra witness parsers only on uncertain arcs.
• I2 (tokens): The en crate could expose lemma and surface features side by side to downstream event detectors, so that past and ongoing events are caught equally.
• I3 (learn): Every new feature channel in the parsers could be evaluated against such duplicate-input controls as a negative control.
• I4 (frames): Coreference evaluation sets could be split while matching chain-count statistics, so measured gains reflect the resolver rather than the split.
• I5 (frames): Role assignment from VerbNet could add explicit tests for agent-like roles that differ only in initiative, such as organizing versus taking part.