📄 Sarcasm and Sentiment Detection in Arabic: investigating the interest of character-level features

Summary

This paper describes one team's entry to the 2021 WANLP shared task, which targeted detecting sentiment and sarcasm in Arabic. The task has two parts: deciding whether a tweet is sarcastic (binary) and assigning it a positive, negative or neutral label. The data merges ArSarcasm and DAICT, and the authors held out part of the training material as a dev set. They used the same three systems for both subtasks:

• a Logistic Regression baseline on character n-grams;
• an LSTM network built on AraVec Twitter embeddings;
• a majority-voting ensemble over word, character and within-word n-gram features, reused from their NADI 2020 system.

They submitted the ensemble. It placed in the lower half of the rankings for both subtasks. After the deadline, they added TF-IDF weighting and class balancing to the plain character-level baseline, and it then beat both of the more elaborate systems on the test set. The main lesson is that character n-grams are a strong, cheap representation for noisy dialectal text, and that careful tuning of a simple model can matter more than added architecture, especially when classes are skewed.

Key points

• Data size: 9,549 training tweets, with 3,000 for dev and 3,000 for test.

• Class imbalance: Sarcastic tweets are only about 17% of the training data (1,666 vs 7,883). For sentiment, positive tweets make up roughly 18% and negative about 37%.

• N-gram sweep: Among MNB, decision trees, random forests, LR and SVM, Logistic Regression worked best. Its scores levelled off once the maximum character n-gram length reached 5.

• Ensemble design: scikit-learn FeatureUnion combined TF-IDF word 1–5-grams, character 1–4-grams and within-word character 1–5-grams. Four classifiers voted on the output: SGDClassifier, LinearSVC, MNB and RidgeClassifier.

• Ranking and baseline results: The submitted ensemble ranked 22nd of 37 for sarcasm and 15th of 22 for sentiment. The reweighted character baseline did better on test:

  • sarcastic-class F1 of 55.83 vs 41.09;
  • positive/negative F of 71.47 vs 65.06.

  On dev, the ensemble missed far more sarcastic tweets than it falsely flagged (363 false negatives vs 96 false positives).

Interesting ideas

• I1. Scan a grid of minimum and maximum character n-gram lengths and look for the point where scores stop improving before fixing the feature set, instead of guessing the range. — For Mova (tokens): A similar sweep could set the affix and n-gram window for out-of-vocabulary tagging and lemma guessing in crate en, using evidence rather than a fixed default.
• I2. Use a strong surface cue, such as an elongated laughter token, to pull out every example containing it, then review the cases whose label contradicts the cue as likely annotation errors. — For Mova (learn): The same cue-versus-label disagreement check can flag suspicious items in silver data or teacher-labelled sets before training, cheaply targeting human or LLM review.
• I3. Keep a simple but properly tuned baseline (TF-IDF plus class weights) next to complex systems, because here it outperformed them all. — For Mova (learn): Each new parser or reader stage could be reported against a tuned shallow control, so a gain is credited only when it beats that control and not an untuned strawman.
• I4. Treat character n-grams taken within word boundaries as a feature channel separate from n-grams that cross word boundaries, which keeps morphology apart from collocation signals. — For Mova (tokens): Separating word-internal from cross-word character evidence could help morphology and tokenization decisions on noisy social-media English without mixing the two kinds of signal.
• I5. For a rare class like sarcasm, check the false-negative/false-positive split and the per-class recall instead of accuracy, and fix a skew toward silence with class weighting. — For Mova (frames): When evaluating the irony detector in crate prag, report recall on ironic cases and weight that rare class, so the module does not quietly default to literal readings.