📄 CENTEMENT at SemEval-2018 Task 1: Classification of Tweets using Multiple Thresholds with Self-correction and Weighted Conditional Probabilities

Summary

The authors build a multi-label emotion classifier for SemEval-2018 Task 1, subtask E-c. Each Arabic or English tweet must be tagged with any subset of eleven emotions, or with none. They wanted something simple, fast and free of lexicons. The system trains only on the official training and development data and uses no stop-word lists or sentiment dictionaries.

Tweets are first cleaned. Emojis become identifiers, usernames are dropped, and words are stemmed (a tweet-specific stemmer for Arabic, Morphy for English). For each frequent token, the system then computes how strongly it leans toward each emotion compared with the others. Two refinements follow:

• an error-driven pass that demotes tokens which tend to cause wrong labels;
• a separate decision threshold for each emotion, chosen by Jaccard score.

The same machinery serves both languages. The system ranked 2nd of 14 for Arabic and 12th of 35 for English, and it trains in about a quarter of an hour on a laptop. This shows that a carefully tuned count-based method can compete with heavier neural systems.

Key points

• Token scores start from P(T|E) over the 2500 top tokens. Each token's values are normalised across emotions and centred on their mean, so evenly spread words count against every emotion. The result is then scaled by the standard deviation to favour skewed words.
• Self-correction is inspired by Brill (1995). It keeps a credit counter per token and emotion, and tokens with a negative balance are downweighted. A single pass is enough, and it touches about 1% of Arabic and 5% of English tokens.
• In the cumulative ablation (Table 1), Jaccard rises from 0.324 to 0.452 for Arabic and from 0.340 to 0.455 for English. The tuning and per-emotion threshold steps give the largest jumps.
• Stemming added 14.8% for English (Morphy) but only 2.67% for Arabic (the tweet-specific stemmer). Doubling emoticon counts added 0.44%.
• Adding EI-reg/EI-oc tweets or 10,000 keyword-filtered tweets peaked at roughly 0.43 Jaccard for English. Joy/love and anger/disgust were the most confusable pairs.

Interesting ideas

• I1. Rather than tuning weights, run the model over its own training data and give every cue credit or blame for each prediction; cues that end up net-harmful get muted. This cheap, transparent repair loop could work for any rule- or feature-based component.
• I2. Scores are made relative across labels and centred, so a cue that is equally common everywhere contributes nothing, while one that is below average actively votes against a label. This avoids a separate stop-list.
• I3. Each label gets its own decision threshold, picked by a coarse grid search against the task metric, because labels differ in base rate and score spread. A single global cutoff hid this.
• I4. A hashtag is kept as one token, and a split copy of its parts is added alongside it. Both the compound signal and the ordinary-word signal survive this way.
• I5. Extra training data harvested by keywords is first relabelled with the current classifier and then mixed in at reduced weight. This prevents the noisy data from overriding the gold set.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Self-induced repair rules could keep a right/wrong ledger per rule on silver data and demote rules whose balance goes negative, which keeps a readable audit trail.
• I2 (frames): Lexical cues for irony or speech acts could be scored against the other categories, so generic words cannot push any one reading.
• I3 (answer): Confidence cutoffs per question type in the readers could be tuned separately rather than sharing one global value.
• I4 (tokens): The tokenizer could emit hyphenated or underscored compounds along with their components as linked alternatives, so later stages can use whichever matches.
• I5 (learn): Silver trees from witness parsers could be weighted below gold treebank data during training, with the weight treated as a tunable knob checked against negative controls.