📄 LSIS at SemEval-2016 Task 7: Using Web Search Engines for English and Arabic Unsupervised Sentiment Intensity Prediction

Summary

This paper describes the LSIS entry to SemEval-2016 Task 7, which asks systems to rank English and Arabic words and short phrases by how strongly positive or negative they are. The authors use no supervised training. They first score each phrase from existing sentiment lexicons and polarity-labelled tweets, using the difference between its PMI with the positive class and its PMI with the negative class. When a phrase is absent from these resources (slang, glued hashtags, transliterated English in Arabic script), they query the Google Search API and compare how often the phrase appears within ten words of "excellent" versus "poor", following Turney's semantic orientation. A constant term is dropped because only the ranking is evaluated. Bing's proximity operator turned out to be unreliable. On the official data the system was second on the mixed-polarity English subtask and competitive on Arabic, although rivals were supervised. The work shows that cheap, transparent association statistics can come close to heavily trained systems on hard compositional cases.

Key points

• Three test sets: 2799 general English items, 1069 English mixed-polarity phrases (e.g. "lazy Weekend") and 1166 Arabic tweet phrases.
• English resources kept after testing:
  • Bing Liu lexicon, Sentiment140, NRC Hashtag Sentiment Lexicon and SentiWordNet;
  • SemEval-2013 Task 2 test tweets.
  • Arabic resources: ASTD, a UCI Arabic Twitter dataset, LABR lexicons and the multilingual NRC Hashtag lexicon.
• Development results on SemEval-2015 Task 10 SubtaskE (1315 phrases):
  • PMI lexicons alone: Kendall 0.443 / Spearman 0.620.
  • With Google added: 0.452 / 0.631, although Google was needed for only 5% of phrases.
  • Bing: 0.029 / 0.039, essentially no correlation.
• Arabic development set (200 phrases): adding Google lowered scores (Spearman 0.584 → 0.561). The authors attribute this to their choice of seed words. Single translations of "excellent" and "poor" often returned zero hits, so they used small groups of seed words instead.
• Official results (Kendall / Spearman):
  • Mixed polarity: 0.422 / 0.590, second behind ECNU.
  • General English: 0.345 / 0.508.
  • Arabic: 0.424 / 0.583.
  • Phrases with no search hits were set to a neutral 0.5.

Interesting ideas

• I1. Use a tiered backoff: trusted curated lexicons answer first, and the noisy open-web co-occurrence estimate is used only for the residue they cannot cover. This limits noise while still handling out-of-vocabulary items.
• I2. When the metric depends only on rank, any factor that is constant across items can be dropped, which saves queries and removes a source of noise.
• I3. The authors exposed a broken external tool with a monotonicity check: a wider proximity window returned fewer hits than a narrower one.
• I4. Single anchor words can be too sparse in a low-resource language, so a small group of expressive seed words gives a sturdier polarity signal, though the choice of seeds strongly affects the result.
• I5. When no evidence exists, the system assigns an explicit neutral value instead of guessing, so missing evidence is visible in the output.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Value and evaluative lookups in compiled knowledge could fall back to an anchor-word co-occurrence estimate for unseen words, with the lookup trace recording which tier supplied the answer.
• I2 (learn): Ranking-style evaluations could check whether expensive normalisation terms actually change the ordering before computing them.
• I3 (learn): Witness parsers and LLM teachers could be probed with similar invariance tests (more context should never yield fewer matches) as negative controls before their outputs are trusted as silver data.
• I4 (frames): Evaluative-stance or irony cues could be scored against a small curated set of polar anchor words rather than one prototype, with the anchor list kept explicit so it can be audited.
• I5 (answer): Readers could return a marked "no evidence / neutral" result when the text world lacks support, which keeps every answer explainable.