📄 Beyond Benchmark Accuracy: Robustness Evaluation of Hinglish Sentiment Models

Summary

Hinglish sentiment classifiers often post excellent test-set scores, but it is unclear whether these reflect real understanding of sentiment. The authors built a training corpus of 25,543 Hinglish tweets from a public emotion dataset, collapsing its emotion labels into positive, negative and neutral. They cleaned it with deduplication, noise removal and dictionary-based spelling normalisation. A leakage check using SHA-256 hashing and 3-gram Jaccard similarity found no overlap between splits. On this data, fine-tuned XLM-R scores 99.7% in-distribution, compared with 74.6% for mBERT.

The authors then stress-tested XLM-R in two ways. The first was a new 400-sentence adversarial set covering five linguistic phenomena. The second was English TweetEval with no further training. XLM-R drops to 42.5% on the adversarial set and 50.8% on TweetEval, and it handles negation and contrast especially badly. The study argues that headline accuracy can hide reliance on lexical cues. It recommends targeted adversarial and cross-domain testing before sentiment systems are deployed in sensitive, code-mixed settings.

Key points

• Data and splits: the corpus has 25,543 tweets divided into 18,786 train, 2,088 validation and 4,669 test. Classes are skewed: 48.4% negative, 33.3% positive and 18.3% neutral, so a majority-class baseline reaches 48.4%.
• Models and training: the models are bert-base-multilingual-cased (179M parameters) and xlm-roberta-base (278M parameters). Both were trained with AdamW at a learning rate of 2e-5 and batch size 32 for 3 epochs, with checkpoints chosen by validation macro F1.
• In-distribution results: XLM-R reaches 0.997 accuracy and 0.996 macro F1, while mBERT reaches 0.746 accuracy and 0.673 macro F1.
• Adversarial benchmark: the set has 80 items in each of five categories: negation, contrast, sarcasm, subtle sentiment and true neutral. Items were drafted from templates and then rewritten by two native speakers (Cohen's κ = 0.81). XLM-R's per-category accuracy is 25% for negation, 25% for contrast, 50% for sarcasm, 50% for subtle sentiment and 63% for true neutral. Its overall scores are 42.5% accuracy and 0.328 macro F1.
• Zero-shot transfer and limitations: on TweetEval, XLM-R gets 50.8% accuracy and 0.408 macro F1, against a random baseline of 0.333. The authors list three limitations: only two models were tested, no significance testing was done, and the benchmark is small. They plan to release the benchmark and code.