Elamur Research⌕
☰
home / papers / 2018 / LDR at SemEval-2018 Task 3: A Low Dimensional Text Representation for Irony Detection

LDR at SemEval-2018 Task 3: A Low Dimensional Text Representation for Irony Detection

Paper · 2018 · Bilal Ghanem, Francisco Rangel and Paolo Rosso

Summary

The paper reports the UPV team's entry to SemEval-2018 Task 3, a shared task on spotting ironic content in Twitter posts written in English. It tests whether LDR, a compact text representation the authors built earlier for language variety identification and author profiling, can also recognise irony. LDR starts from a Tf-Idf matrix and gives each term a class-dependent weight: its share of weight in one class's documents relative to all documents. Each tweet is then reduced to a few statistics of those weights per class, such as mean, spread, extremes and vocabulary proportion. The authors trained Weka classifiers with 10-fold cross-validation on the task data. In some runs they added two external Twitter corpora and combined classifiers by majority vote. On the official test set, macro F-score was 0.43 for binary Task A and 0.23 for four-class Task B. Their error analysis found that the features separate the classes weakly. The authors conclude that low-dimensional statistics like these miss irony, a phenomenon that depends on implicit meaning. The paper is a useful negative result for anyone tempted by purely distributional shortcuts.

Key points

  • Features: For every class, LDR computes six features per document from class-conditional term weights: AVG, STD, MIN, MAX, PROB and PROP. No external lexicons or hand-crafted cues are used.
  • Task data: The Task A training set has 3,834 tweets (1,911 ironic, 1,923 not). Task B splits the ironic tweets into three subtypes:
    • 1,390 with polarity contrast,
    • 316 situational,
    • 205 other irony.
  • External data: The unconstrained run added the English part of the Karoui et al. (2017) corpus (540 tweets) and about 67,800 #sarcasm-hashtag tweets from Ptáček et al. (2014).
  • Task A results: In cross-validation, DecisionStump reached about 64% accuracy and 65% F-score. Majority vote over the external-data runs improved accuracy by only 0.6%. On the test set, accuracy was 0.56 and F-score 0.43 for both runs.
  • Task B results and diagnostics: Task B test accuracy was 0.46 and macro F-score 0.23. Recall on the minority irony subtype fell as low as 6%. GainRatio rated even the best feature (AVG for the ironic class) at only about 10%.

Interesting ideas

  • I1. Check features before tuning classifiers. The authors ran GainRatio on their features before blaming the classifiers, and this showed that the representation itself carried little signal.
  • I2. Compare class centroids for right and wrong cases. They measured the distance between class centroids separately for correctly and wrongly classified items (0.143 vs 0.102), which shows how class overlap drives errors.
  • I3. Filtering rare terms erases content words. In the example "Yay jury duty," only "Yay" contributed to the representation, because rare terms had been removed. Frequency-based models can therefore drop exactly the content words whose situation carries the irony.
  • I4. Bigger distant-labelled data may not help. Adding a large #sarcasm-hashtag corpus barely helped, which warns that a large, loosely labelled, adjacent-phenomenon dataset may not transfer.
  • I5. Low-frequency subtypes collapse first. In Task B, the rare subtypes nearly vanished (6% and 29% correct), which shows that statistical summaries flatten the infrequent kinds of irony.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Before retraining a component that underperforms, measure how informative its input signals are, so that representation failures are kept apart from model failures.
  • I2 (learn): The same split-by-correctness centroid check could serve as a cheap diagnostic in evaluation reports, showing where silver-trained components confuse categories.
  • I3 (frames): The irony detector in prag should reason over the clash between an evaluative marker ("Yay") and the frame it attaches to (an unwanted obligation), rather than over word frequencies.
  • I4 (learn): Silver data from a neighbouring phenomenon (sarcasm vs irony) should be validated against negative controls before it enters training.
  • I5 (frames): Situational irony and irony without polarity contrast probably need explicit rules over world states and expectations rather than learned statistics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Politècnica de ValènciaValencia · Spain21 works · 3,275 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 works

Cite

Cite as Bilal Ghanem, Francisco Rangel and Paolo Rosso. 2018. LDR at SemEval-2018 Task 3: A Low Dimensional Text Representation for Irony Detection. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1086

@inproceedings{ghanem2018ldr,
  title = {{LDR at SemEval-2018 Task 3: A Low Dimensional Text Representation for Irony Detection}},
  author = {Bilal Ghanem, Francisco Rangel and Paolo Rosso},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1086},
  url = {https://doi.org/10.18653/v1/s18-1086}
}

References

16 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Francesco Barbieri and Horacio Saggion. 2014. Modelling irony in twitter. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 56–64.
  2. Paula Carvalho, Luı́s Sarmento, Mário J Silva, and Eugénio De Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s so easy;-. In Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, pages 53–56. ACM. find
  3. Raül Boluda Fabra, Francisco Rangel, and Paolo Rosso. 2015. Nlel upv autoritas participation at discrimination between similar languages (dsl) 2015 shared task. In Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, pages 52–58. find
  4. Marc Franco-Salvador, Francisco Rangel, Paolo Rosso, Mariona Taulé, and M Antònia Martı́t. 2015. Language variety identification using distributed representations of words and documents. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 28–40. Springer. find
  5. Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. Semeval-2015 task 11: Sentiment analysis of figurative language in twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 470–478.
  6. Po-Ya Angela Wang. 2013. # irony or# sarcasma quantitative and qualitative study based on twitter. In Proceedings of the 27th Pacific Asia Conference on Language, Information, and Computation (PACLIC 27), pages 349–356. find
  7. Jihen Karoui, Farah Benamara, Véronique Moriceau, Nathalie Aussenac-Gilles, and Lamia Hadrich Belguith. 2015. Towards a contextual pragmatic model to detect irony in tweets. In 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015), pages PP–644.
  8. Jihen Karoui, Benamara Farah, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pages 262–272.
  9. Tatiana Litvinova, Francisco Rangel, Paolo Rosso, Pavel Seredin, and Olga Litvinova. 2017. Overview of the rusprofiling pan at fire track on cross-genre gender identification in russian. Notebook Papers of FIRE, pages 8–10. find
  10. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223.
  11. Francisco Rangel, Marc Franco-Salvador, and Paolo Rosso. 2017a. A low dimensionality representation for language variety identification. arXiv preprint arXiv:1705.10754. ↗
  12. Francisco Rangel, Paolo Rosso, Martin Potthast, and Benno Stein. 2017b. Overview of the 5th author profiling task at pan 2017: Gender and language variety identification in twitter. In CLEF 2017 Labs and Workshops, Notebook Papers. CEUR Workshop Proceedings, volume 1866. find
  13. Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering, 74:1–12. ↗
  14. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Language resources and evaluation, 47(1):239–268. ↗
  15. Cynthia Van Hee, Els Lefever, and Vronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018).
  16. Byron C Wallace, Eugene Charniak, et al. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1035–1044.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghanem-2018-ldr-semeval-2018-task-3/