Elamur ResearchβŒ•
☰
home / papers / 2013 / UMBC_EBIQUITY-CORE: Semantic Textual Similarity Systems

UMBC_EBIQUITY-CORE: Semantic Textual Similarity Systems

Summary

The paper describes three systems submitted by the UMBC/JHU team to the *SEM 2013 Semantic Textual Similarity (STS) shared task. The task asks for a 0–5 similarity score for each sentence pair. All three systems rely on one word-similarity component. It applies LSA to word co-occurrence counts from a cleaned Stanford WebBase corpus, then adjusts the scores with WordNet relations. The first run, PairingWords, aligns terms across the two sentences and subtracts a penalty for weak alignments. It needs no training. The other two runs, Galactus and Saiyan, feed many lexical and semantic features into support vector regression. Galactus trains on all STS 2012 data. Saiyan uses separate training data and features for each dataset. PairingWords achieved the highest mean correlation of all 89 submitted runs. The two regression runs finished second and fourth. The results suggest that a strong word-similarity measure plus a simple alignment can beat heavier feature-based learners. The simple system was also more stable across unfamiliar test domains.

Key points

  • The LSA corpus comes from the February 2007 Stanford WebBase crawl, about 100 million pages from more than 50,000 sites. After filtering for paragraph length, English text, punctuation ratio and duplicates, about 3 billion words remained. The authors released this corpus.

  • They built two co-occurrence models with window sizes Β±1 and Β±4. The vocabulary of about 29,000 POS-tagged entries includes more than 22,000 common words and noun phrases plus more than 2,000 WordNet verb phrases. Counts were log-transformed and reduced by SVD to 300 dimensions.

  • WordNet relations are turned into a path distance D. The final score is the LSA similarity plus 0.5Β·e^(βˆ’0.25Β·D), capped at 1.0. Words with ten or more senses are compared through less ambiguous synonyms.

  • Official weighted means:

    • PairingWords: 0.6181 (rank 1)
    • Galactus: 0.5927 (rank 2)
    • Saiyan: 0.5683 (rank 4)

    PairingWords also ranked first on FNWN with 0.5818. Galactus used 52 features with LIBSVM and a grid search for parameters.

  • On the 2012 data, an antonym penalty raised the average correlation only from 0.6826 to 0.6871, so the authors left it out of the submitted run. Saiyan's OnWN model overfit: applying its headlines model to OnWN raised correlation from 0.55 to 0.71.

Interesting ideas

  • I1. Two context windows for two kinds of similarity. A narrow window captures close "concept" similarity within one part of speech. A wide window links related words across parts of speech, such as a verb and a noun. Keeping both lets downstream tasks choose which sense of similarity they need.
  • I2. Patching distributional scores with a lexicon. WordNet relations add a bonus that decays with path distance, and the result is capped. This corrects known LSA weaknesses, such as low scores for highly polysemous synonyms. The same recipe could apply to any language that has a wordnet-style resource, including Ukrainian.
  • I3. Filtering out rare word senses. The authors count only "significant" senses, judged by sense rank and frequency. This stops obscure meanings from creating false synonym links, which helps anyone building lexical resources from sense inventories.
  • I4. Dropping components that add little. The antonym penalty brought only a marginal gain, so the authors removed it to keep the system robust. Weighing small gains against complexity is a sensible habit for shared-task systems.
  • I5. Training-free systems for new domains. The untrained alignment system behaved more predictably on unseen datasets than per-domain regression models. When test domains are unknown, a strong unsupervised baseline may beat tuned learners.

Our summary of the paper, in our own words β€” not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

86 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 81 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{han2013umbc,
  title = {{UMBC\_EBIQUITY-CORE: Semantic Textual Similarity Systems}},
  author = {Han, Lushan and Kashyap, Abhay L. and Finin, Tim and Mayfield, James and Weese, Jonathan},
  booktitle = {Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity},
  year = {2013},
  doi = {10.13016/m28w38626},
  url = {https://doi.org/10.13016/m28w38626}
}

References

37 references, 6 of them with a card of ours. β†— leads to the work, Β«findΒ» searches for one we have no id for yet.

  1. Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: a pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics, pages 385–393. Association for Computational Linguistics. find our card
  2. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *sem 2013 shared task: Semantic textual similarity, including a pilot on typed-similarity. In *SEM 2013: The Second Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics. find our card
  3. M. Baroni, S. Bernardini, A. Ferraresi, and E. Zanchetta. 2009. The wacky wide web: A collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209–226. find
  4. Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL on Interactive presentation sessions, COLING-ACL ’06, pages 69–72, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  5. C. Burgess, K. Livesay, and K. Lund. 1998. Explorations in context space: Words, sentences, discourse. Discourse Processes, 25:211–257. find
  6. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27. find
  7. T.A.S. Coelho, Pável Pereira Calado, Lamarque Vieira Souza, Berthier Ribeiro-Neto, and Richard Muntz. 2004. Image retrieval using multiple evidence ranking. IEEE Trans. on Knowl. and Data Eng., 16(4):408–417. find
  8. Michael John Collins. 1999. Head-driven statistical models for natural language parsing. Ph.D. thesis, University of Pennsylvania. find
  9. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407. find
  10. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: exploiting massively parallel news sources. In Proceedings of the 20th international conference on Computational Linguistics, COL-ING ’04. Association for Computational Linguistics. find our card
  11. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In HLT-NAACL 2013. find our card
  12. Lushan Han and Tim Finin. 2013. UMBC webbase corpus. http://ebiq.org/r/351. β†—
  13. Lushan Han, Tim Finin, and Anupam Joshi. 2012. Schema-free structured querying of dbpedia data. In Proceedings of the 21st ACM international conference on Information and knowledge management, pages 2090–2093. ACM. find
  14. Lushan Han, Tim Finin, Paul McNamee, Anupam Joshi, and Yelena Yesha. 2013. Improving Word Similarity by Augmenting PMI with Estimates of Word Polysemy. IEEE Transactions on Knowledge and Data Engineering, 25(6):1307–1322. find
  15. Zellig Harris. 1968. Mathematical Structures of Language. Wiley, New York, USA. find
  16. M. Hart. 1997. Project gutenberg electronic books. http://www.gutenberg.org/wiki/Main Page. β†—
  17. David Kauchak and Regina Barzilay. 2006. Paraphrasing for automatic evaluation. In HLT-NAACL ’06, pages 455–462. find our card
  18. T. Landauer and S. Dumais. 1997. A solution to plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. In Psychological Review, 104, pages 211–240. find
  19. Y. Li, Z.A. Bandar, and D. McLean. 2003. An approach for measuring semantic similarity between words using multiple information sources. IEEE Transactions on Knowledge and Data Engineering, 15(4):871–882. find
  20. Dekang Lin. 1998a. Automatic retrieval and clustering of similar words. In Proc. 17th Int. Conf. on Computational Linguistics, pages 768–774, Montreal, CN. find
  21. Dekang Lin. 1998b. An information-theoretic definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning, ICML ’98, pages 296–304, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  22. Charles T. Meadow. 1992. Text Information Retrieval Systems. Academic Press, Inc. find
  23. 2008. Google word frequency counts. http://bit.ly/10JdTRz. β†—
  24. Donald Metzler, Susan Dumais, and Christopher Meek. 2007. Similarity measures for short segments of text. In Proceedings of the 29th European conference on IR research, pages 16–27. Springer-Verlag. find
  25. Jean-Baptiste Michel, Yuan K. Shen, Aviva P. Aiden, Adrian Veres, Matthew K. Gray, The Google Books Team, Joseph P. Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, Steven Pinker, Martin A. Nowak, and Erez L. Aiden. 2011. Quantitative analysis of culture using millions of digitized books… find
  26. Rada Mihalcea, Courtney Corley, and Carlo Strapparava. 2006. Corpus-based and knowledge-based measures of text semantic similarity. In Proceedings of the 21st national conference on Artificial intelligence, pages 775–780. AAAI Press. find
  27. G.A. Miller. 1995. WordNet: a lexical database for English. Communications of the ACM, 38(11):41. find our card
  28. Saif Mohammad, Bonnie Dorr, and Graeme Hirst. 2008. Computing word-pair antonymy. In Proc. Conf. on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-2008), October. find
  29. Tony Rose, Mark Stevenson, and Miles Whitehead. 2002. The reuters corpus volume 1 - from yesterdays news to tomorrows language resources. In In Proceedings of the Third International Conference on Language Resources and Evaluation, pages 29–31. find
  30. Mehran Sahami and Timothy D. Heilman. 2006. A web-based kernel function for measuring the similarity of short text snippets. In Proceedings of the 15th international conference on World Wide Web, WWW ’06, pages 377–386. ACM. find
  31. Frane Saric, Goran Glavas, Mladen Karan, Jan Snajder, and Bojana Dalbelo Basic. 2012. Takelab: systems for measuring semantic text similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics, pages 441– 448. Association for Computational Linguistics. find
  32. Bharath Sriram, Dave Fuhry, Engin Demir, Hakan Ferhatosmanoglu, and Murat Demirbas. 2010. Short text classification in twitter to improve information filtering. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 841–842. ACM. find
  33. Stanford. 2001. Stanford WebBase project. http://bit.ly/WebBase. β†—
  34. Kristina Toutanova, Dan Klein, Christopher Manning, William Morgan, Anna Rafferty, and Michel Galley. 2000. Stanford log-linear part-of-speech tagger. http://nlp.stanford.edu/software/tagger.shtml. β†—
  35. UMBC. 2013a. Graph of relations project. http://ebiq.org/j/95. β†—
  36. UMBC. 2013b. Semantic similarity demonstration. http://swoogle.umbc.edu/SimService/. β†—
  37. Z. Wu and M. Palmer. 1994. Verb semantic and lexical selection. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics (ACL-1994), pages 133–138, Las Cruces (Mexico). find

↑ back to top

This page in Markdown β†’ https://md.research.elamur.ai/free/papers/han-2013-umbc-ebiquity-core-semantic-textual/