Elamur Research⌕
☰
home / papers / 2012 / UKP: Computing Semantic Textual Similarity by Combining Multiple Content Similarity Measures

UKP: Computing Semantic Textual Similarity by Combining Multiple Content Similarity Measures

Summary

The paper describes the system UKP submitted to SemEval-2012's pilot task on Semantic Textual Similarity (STS). In that task, systems assign a graded, symmetric similarity score to pairs of sentences. The authors argue that a single similarity measure rarely captures everything relevant, so they compute a large pool of measures and let a supervised model combine them. The pool has more than 300 feature variants:

  • Surface measures: string overlap and character/word n-grams.
  • Semantic measures: WordNet-based word similarity and ESA vectors.
  • Expansion measures: lexical substitution and back-translation via Moses, used to bridge vocabulary mismatches.
  • Structure and style measures.

The log-transformed scores feed a linear regression model in WEKA, trained separately for each known dataset. Roughly 20 features survive in the final models. The best configuration, Run 2, placed first among 89 runs on two of the three official metrics and second on the third. The system did not win any single dataset, so its strength lies in consistency across very different text types. This makes feature combination an attractive, robust baseline for graded similarity.

Key points

  • Official test results for Run 2: ALL r = .823 (rank #1, 99% CI .807–.837), Mean r = .677 (#1) and ALLnrm r = .857 (#2).
  • Cross-validation on training data (10-fold): r = .724 on MSRpar, .868 on MSRvid and .742 on SMTeuroparl.
  • Surprise test sets: OnWN and SMTnews were scored with a model trained on all training sets pooled together.
  • Word similarity: Resnik's measure on WordNet beat Jiang-Conrath and Lin. It was lifted to sentence level using the idf-weighted, bidirectional best-match aggregation of Mihalcea et al.
  • ESA resources: WordNet, Wikipedia and Wiktionary.
  • Features the classifier dropped:
    • BIUTEE entailment scores.
    • All structure and style features (stopword and POS n-grams, word pair order, function word frequencies, TTR).
    • Distributional Thesaurus features, except the one for cardinal numbers (CD).
  • Back-translation setup: Moses trained on Europarl, pivoting through Dutch, German and Spanish. It helped most when word similarity was computed on the original sentence joined with all three back-translations.

Interesting ideas

  • I1. Build many features, let the model prune them. Generate hundreds of cheap similarity variants and let a simple regression choose among them. This is easy to port to languages with fewer resources, such as Ukrainian, where no single strong measure may exist.
  • I2. Expand text with round-trip translation. Translating a sentence out through pivot languages and back adds synonyms that can close lexical gaps between paraphrases. It needs only off-the-shelf MT, not a curated thesaurus.
  • I3. Log-transform scores before linear fitting. Applying a log to raw similarity scores lets a plain linear model act log-linearly. It is a lightweight way to cope with skewed feature distributions.
  • I4. Override with a normalized exact match. Strip everything except alphanumerics, and if the two strings then match, output the maximum score. This rule-based fix catches cases where the learned model is needlessly uncertain.
  • I5. Probe an unknown test set with a dummy run. Run 3 returned random values in [4.5, 5] on the surprise sets. A deliberately naive submission like this helps reveal how much a metric or dataset rewards real modelling versus score-range effects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany58 works · 15,788 citations🔬DIPF | Leibniz Institute for Research and Information in EducationFrankfurt am Main · Germany6 works · 1,833 citations

Cite

Cite as Daniel Bär, Chris Biemann, Iryna Gurevych and Torsten Zesch. 2012. UKP: Computing Semantic Textual Similarity by Combining Multiple Content Similarity Measures. *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bar2012ukp,
  title = {{UKP: Computing Semantic Textual Similarity by Combining Multiple Content Similarity Measures}},
  author = {Daniel Bär, Chris Biemann, Iryna Gurevych and Torsten Zesch},
  booktitle = {*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)},
  year = {2012},
  doi = {10.17185/duepublico/72186},
  url = {https://doi.org/10.17185/duepublico/72186}
}

References

36 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation, in conjunction with the 1st Joint Conference on Lexical and Computational Semantics. our card
  2. Lloyd Allison and Trevor I. Dix. 1986. A bit-string longest-common-subsequence algorithm. Information Processing Letters, 23:305–310. find
  3. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v.2.0. Journal of Technology, Learning, and Assessment, 4(3). find
  4. Alberto Barrón-Cedeño, Paolo Rosso, Eneko Agirre, and Gorka Labaka. 2010. Plagiarism Detection across Distant Language Pairs. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 37–45. find
  5. Chris Biemann. 2012. Creating a System for Lexical Substitutions from Scratch using Crowdsourcing. Language Resources and Evaluation: Special Issue on Collaboratively Constructed Language Resources, 46(2). find
  6. Andrei Z. Broder. 1997. On the resemblance and containment of documents. Proceedings of the Compression and Complexity of Sequences, pages 21–29. find
  7. Paul Clough, Robert Gaizauskas, Scott S.L. Piao, and Yorick Wilks. 2002. METER: MEasuring TExt Reuse. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 152–159. find
  8. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges, Lecture Notes in Computer Science, pages 177–190. Springer. find
  9. Liviu P. Dinu and Marius Popescu. 2009. Ordinal measures in authorship identification. In Proceedings of the 3rd PAN Workshop. Uncovering Plagiarism, Authorship and Social Software Misuse, pages 62–66. find
  10. William B. Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources. In Proceedings of the 20th International Conference on Computational Linguistics, pages 350–356. our card
  11. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press. find
  12. Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, pages 1606–1611. find
  13. Dan Gusfield. 1997. Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. Cambridge University Press. find
  14. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11(1):10–18. find
  15. Vasileios Hatzivassiloglou, Judith L. Klavans, and Eleazar Eskin. 1999. Detecting text similarity over short passages: Exploring linguistic feature combinations via machine learning. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corp… find
  16. Matthew A. Jaro. 1989. Advances in record linkage methodology as applied to the 1985 census of Tampa Florida. Journal of the American Statistical Association, 84(406):414–420. find
  17. Jay J. Jiang and David W. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th International Conference on Research in Computational Linguistics. find
  18. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Pro… find
  19. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of the 10th Machine Translation Summit, pages 79–86. find
  20. Thomas K. Landauer, Peter W. Foltz, and Darrell Laham. 1998. An introduction to latent semantic analysis. Discourse Processes, 25(2):259–284. find
  21. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710. find
  22. Dekang Lin and Patrick Pantel. 2001. Discovery of Inference Rules for Question Answering. Natural Language Engineering, 7(4):343–360. find
  23. Dekang Lin. 1998a. An information-theoretic definition of similarity. In Proceedings of International Conference on Machine Learning, pages 296–304. find
  24. Dekang Lin. 1998b. Automatic Retrieval and Clustering of Similar Words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics, pages 768–774. our card
  25. Caroline Lyon, James Malcolm, and Bob Dickerson. 2001. Detecting short passages of similar text in large document collections. In Proceedings of Conference on Empirical Methods in Natural Language Processing, pages 118–125. find
  26. Philip M. McCarthy and Scott Jarvis. 2010. MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381–92. find
  27. Rada Mihalcea, Courtney Corley, and Carlo Strapparava. 2006. Corpus-based and Knowledge-based Measures of Text Semantic Similarity. In Proceedings of the 21st National Conference on Artificial Intelligence, pages 775–780. find
  28. Alvaro Monge and Charles Elkan. 1997. An efficient domain-independent algorithm for detecting approximately duplicate database records. In Proceedings of the SIGMOD Workshop on Data Mining and Knowledge Discovery, pages 23–29. find
  29. Frederick Mosteller and David L. Wallace. 1964. Inference and disputed authorship: The Federalist. Addison-Wesley. find
  30. Philip Resnik. 1995. Using Information Content to Evaluate Semantic Similarity in a Taxonomy. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453. our card
  31. Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of the International Conference on New Methods in Language Processing, pages 44–49. find
  32. Efstathios Stamatatos. 2011. Plagiarism detection using stopword n-grams. Journal of the American Society for Information Science and Technology, 62(12):2512–2527. find
  33. Asher Stern and Ido Dagan. 2011. A Confidence Model for Syntactically-Motivated Entailment Proofs. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, pages 455–462. find
  34. Mildred C. Templin. 1957. Certain language skills in children. University of Minnesota Press. find
  35. William E. Winkler. 1990. String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage. In Proceedings of the Section on Survey Research Methods, pages 354–359. find
  36. Michael J. Wise. 1996. YAP3: Improved detection of similarities in computer program and other texts. In Proceedings of the 27th SIGCSE technical symposium on Computer science education, pages 130–134. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bar-2012-ukp-computing-semantic-textual-similarity/