Elamur Research⌕
☰
home / papers / 2018 / Improving the results of string kernels in sentiment analysis and Arabic dialect identification by adapting them to your test set

Improving the results of string kernels in sentiment analysis and Arabic dialect identification by adapting them to your test set

Paper · 2018 · Radu Tudor Ionescu and Andrei M. Butnaru

Summary

The paper asks whether string kernels, which already lead several text classification benchmarks, can gain further accuracy by using the unlabeled test set during training. The authors combine two transductive techniques. First, they compute a kernel matrix over the union of training and test samples. They normalize it and convert it to an RBF form, then treat each row as a feature vector, so every sample is described by its similarity to all others, including the test items. A linear kernel over these vectors gives the adapted kernel. Second, a two-round self-training step (TKC) trains Kernel Ridge Regression, ranks test samples by confidence, and adds the top 1000 with their predicted labels before retraining. Experiments cover cross-domain English polarity classification on the Multi-Domain Sentiment Dataset and Arabic dialect identification on the ADI data from VarDial 2017. Both techniques beat the strongest string-kernel baselines, with statistically significant gains. This matters because cheap character-level models can be adapted to a new domain without any target labels.

Key points

  • The polarity data has four Amazon review domains (Books, DVDs, Electronics, Kitchen), each with 1000 positive and 1000 negative reviews. Models are evaluated in a multi-source setting and a single-source setting.
  • Both the presence bits kernel (K 0/1) and the intersection kernel (K ∩) use character 5–8-grams. Kernel Ridge Regression is run with regularization 10⁻⁵, and r = 1000 pseudo-labeled test samples are added.
  • In the multi-source setting, adding TKC beats the best baseline by over 1.5% on every target domain. For example, Books rises from 82.0% to 84.1%. In the single-source setting, gains exceed 2% everywhere and exceed 4% for E→B, E→D, B→K and D→K.
  • On ADI (Modern Standard Arabic plus Egyptian, Gulf, Levantine and Maghrebi), the authors apply their method to the shared-task winner's sum of four kernels (K 0/1, K ∩, K LRD, K ivec). This lifts accuracy from 64.17% to 66.73% on dev and from 76.27% to 78.35% on test.
  • Every claimed improvement is checked with a paired McNemar test at the 0.01 level. Fewer than 8% of the pseudo-labels added during self-training were wrong.

Interesting ideas

  • I1. A sample can be described by its vector of similarities to every training and test item, which quietly injects knowledge of the target domain into the features without using any target labels.
  • I2. Self-training can be kept to just two rounds with a fixed budget of the most confident predictions, and the share of wrong pseudo-labels can be measured afterwards to confirm that signal outweighs noise.
  • I3. Normalizing the kernel first puts all similarities on a fixed scale, which lets the RBF width be set to a constant instead of being tuned.
  • I4. Character n-grams of length 5–8 stay useful even on noisy ASR transcripts, which shows that subword overlap is robust to spelling and recognition noise.
  • I5. Judging every system comparison with a paired McNemar test, rather than reporting raw accuracy differences, separates real gains from chance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when building silver data for a new genre (fables, school texts), sentences could be ranked by their similarity profile to the target collection, so training emphasizes the closest in-domain examples.
  • I2 (tree): the perceptron parser or gparse could retrain once on its most confident parses of unannotated target text, with the error rate of the added trees audited against a small gold sample.
  • I3 (tokens): a character n-gram similarity between unknown and known words could guide lemma and tag guesses for OOV tokens without adding a tuned hyperparameter.
  • I4 (tokens): this could serve as a fallback cue for normalizing misspelled or garbled tokens before tagging, kept separate from the explainable main pipeline.
  • I5 (learn): candidate repair rules or retrained models could be accepted only when a paired significance test on the same evaluation items favors them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of BucharestBucharest · Romania21 works · 516 citations🔬Inception Institute of Artificial IntelligenceAbu Dhabi · United Arab Emirates6 works · 292 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Radu Tudor Ionescu and Andrei M. Butnaru. 2018. Improving the results of string kernels in sentiment analysis and Arabic dialect identification by adapting them to your test set. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1135

@inproceedings{ionescu2018improving,
  title = {{Improving the results of string kernels in sentiment analysis and Arabic dialect identification by adapting them to your test set}},
  author = {Radu Tudor Ionescu and Andrei M. Butnaru},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1135},
  url = {https://doi.org/10.18653/v1/d18-1135}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic dialect detection in arabic broadcast speech. In Proceedings of INTERSPEECH, pages 2934–2938.
  2. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boomboxes and blenders: Domain adaptation for sentiment classification. In Proceedings of ACL, pages 187–205. find
  3. D. Bollegala, D. Weir, and J. Carroll. 2013. Cross-Domain Sentiment Classification Using a Sentiment Sensitive Thesaurus. IEEE Transactions on Knowledge and Data Engineering, 25(8):1719–1731. ↗
  4. Andrei M. Butnaru and Radu Tudor Ionescu. 2018. UnibucKernel Reloaded: First Place in Arabic Dialect Identification for the Second Year in a Row. In Proceedings of VarDial Workshop of COLING, pages 77–87. ↗
  5. Minmin Chen, Kilian Weinberger, and John Blitzer. 2011. Co-Training for Domain Adaptation. In Proceedings of NIPS, pages 2456–2464. find
  6. Mădălina Cozma, Andrei Butnaru, and Radu Tudor Ionescu. 2018. Automated essay scoring with string kernels and word embeddings. In Proceedings of ACL, pages 503–509. find
  7. Thomas G. Dietterich. 1998. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation, 10(7):1895–1923. ↗
  8. Marc Franco-Salvador, Fermin L. Cruz, Jose A. Troyano, and Paolo Rosso. 2015. Cross-domain polarity classification using a knowledge-enhanced meta-classifier. Knowledge-Based Systems, 86:46–56. ↗
  9. Rosa M. Giménez-Pérez, Marc Franco-Salvador, and Paolo Rosso. 2017. Single and Cross-domain Polarity Classification using String Kernels. In Proceedings of EACL, pages 558–563. find
  10. Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. 2007. Correcting sample selection bias by unlabeled data. In Proceedings of NIPS, pages 601–608. ↗
  11. Xingchang Huang, Yanghui Rao, Haoran Xie, Tak-Lam Wong, and Fu Lee Wang. 2017. Cross-Domain Sentiment Classification via Topic-Related TrAdaBoost. In Proceedings of AAAI, pages 4939–4940. find
  12. Radu Tudor Ionescu. 2013. Local Rank Distance. In Proceedings of SYNASC, pages 221–228. find
  13. Radu Tudor Ionescu. 2015. A Fast Algorithm for Local Rank Distance: Application to Arabic Native Language Identification. In Proceedings of ICONIP, volume 9490, pages 390–400.
  14. Radu Tudor Ionescu and Andrei Butnaru. 2017. Learning to Identify Arabic and German Dialects using Multiple Kernels. In Proceedings of VarDial Workshop of EACL, pages 200–209.
  15. Radu Tudor Ionescu and Marius Popescu. 2016a. Native Language Identification with String Kernels. In Knowledge Transfer between Computer Vision and Text Mining, Advances in Computer Vision and Pattern Recognition, chapter 8, pages 193–227. Springer International Publishing. ↗
  16. Radu Tudor Ionescu and Marius Popescu. 2016b. UnibucKernel: An Approach for Arabic Dialect Identification based on Multiple String Kernels. In Proceedings of VarDial Workshop of COLING, pages 135–144.
  17. Radu Tudor Ionescu and Marius Popescu. 2017. Can string kernels pass the test of time in native language identification? In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 224–234. find
  18. Radu Tudor Ionescu, Marius Popescu, and Aoife Cahill. 2014. Can characters reveal your native language? a language-independent approach to native language identification. In Proceedings of EMNLP, pages 1363–1373. find
  19. Radu Tudor Ionescu, Marius Popescu, and Aoife Cahill. 2016. String kernels for native language identification: Insights from behind the curtains. Computational Linguistics, 42(3):491–525. ↗
  20. Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Christopher J. C. H. Watkins. 2002. Text classification using string kernels. Journal of Machine Learning Research, 2:419–444. find
  21. Shervin Malmasi and Marcos Zampieri. 2017. Arabic Dialect Identification Using iVectors and ASR Transcripts. In Proceedings of the VarDial Workshop of EACL, pages 178–183.
  22. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective Self-training for Parsing. In Proceedings of NAACL, pages 152–159.
  23. Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen. 2010. Cross-domain Sentiment Classification via Spectral Feature Alignment. In Proceedings of WWW, pages 751–760. find
  24. Marius Popescu and Cristian Grozea. 2012. Kernel methods and string kernels for authorship analysis. In Proceedings of CLEF (Online Working Notes/Labs/Workshop). find
  25. Marius Popescu, Cristian Grozea, and Radu Tudor Ionescu. 2017. HASKER: An efficient algorithm for string kernels. Application to polarity classification in various languages. In Proceedings of KES, pages 1755–1763. find
  26. Marius Popescu and Radu Tudor Ionescu. 2013. The Story of the Characters, the DNA and the Native Language. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 270–278. find
  27. John Shawe-Taylor and Nello Cristianini. 2004. Kernel Methods for Pattern Analysis. Cambridge University Press. ↗
  28. Baochen Sun, Jiashi Feng, and Kate Saenko. 2016. Return of Frustratingly Easy Domain Adaptation. In Proceedings of AAAI, pages 2058–2065. find
  29. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the VarDial Evaluation Campaign 2017. In Proceedings of VarDial Workshop of EACL, pages 1–15.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ionescu-2018-improving-results-string-kernels-sentiment/