Elamur Research⌕
☰
home / papers / 2011 / Learning Discriminative Projections for Text Similarity Measures

Learning Discriminative Projections for Text Similarity Measures

Summary

Classic vector-space similarity, such as cosine over TFIDF vectors, lets each term match only itself. Related words like synonyms therefore add nothing to the score, and across languages, where vocabularies barely overlap, the approach fails completely. Yih and colleagues propose S2Net (Similarity Learning via Siamese Neural Network), which learns a linear projection from sparse term vectors into a compact concept space. Two identical copies of the network encode the two texts of a labeled pair. The shared projection matrix is trained by gradient descent so that the cosine of the projected vectors agrees with the labels. A pairwise logistic loss on score differences rewards correct ordering rather than exact score values. The model has the same form as LSA/PCA, OPCA and CCA, but its objective can be shaped to the target metric, and it scales to many training pairs. In cross-lingual comparable-document retrieval it beats OPCA, the strongest earlier method, using 500 dimensions instead of 2,000. In query-to-ad-landing-page relevance it is the only tested method that tops raw TFIDF cosine in both AUC and NDCG. The work shows that discriminative training can produce compact, task-tuned similarity measures.

Key points

  • S2Net is a two-layer linear network whose input is a TFIDF-weighted term vector and whose output is a k-dimensional concept vector. Its weights are exactly the d×k projection matrix, so it is directly comparable to PCA, OPCA and CCA.
  • Baselines included TFIDF cosine, generative topic models (PLSA implemented as LDA with MAP inference, JPLSA, CPLSA), linear projections (LSA/PCA, CL-LSI, OPCA, CCA) and the metric learner HDLR.
  • On ad relevance (Table 3), S2Net with 1000 dimensions reached an AUC of 0.892, compared with 0.861 for TFIDF and 0.853 for CPLSA. The best NDCG@1, 0.860, came at 750 dimensions.
  • Eigen-decomposition methods trained in roughly 2–3 hours on an eight-core server. S2Net's cost scales about linearly with dimensions and examples, at O(mnk) per iteration. For cross-lingual retrieval at k=1000, each iteration took about 48 minutes, and convergence needed around 80 iterations.
  • HDLR could use only a few thousand labeled constraints, failed to improve on its initial model for cross-lingual retrieval, and was very sensitive to hyperparameters.

Interesting ideas

  • I1. Match the loss to the evaluation goal. When similarity is used to rank candidates, train on whether a better pair scores above a worse one rather than regressing onto target scores. This transfers to any retrieval or deduplication system where only the ordering matters.
  • I2. Keep the model form of the baselines. S2Net deliberately has the same shape as PCA/CCA-style projections, so any gain clearly comes from the training objective rather than extra capacity. This is a clean experimental design for testing new objectives.
  • I3. Use negative examples generously. The authors credit part of the improvement to learning from many dissimilar pairs, which OPCA/CCA and paired topic models cannot exploit well. Mining abundant negatives is a cheap lever in many similarity tasks.
  • I4. Report quality across dimensions. Evaluating at several concept-space sizes shows that strong accuracy is possible at low dimension, which matters for storage and fast search indexes.
  • I5. Consider asymmetric projections. The authors suggest learning separate matrices when the compared texts come from different distributions, such as queries versus pages or two languages. This is a natural extension for cross-lingual work, for example Ukrainian–English matching.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

61 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 57 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yih2011learning,
  title = {{Learning Discriminative Projections for Text Similarity Measures}},
  author = {Yih, Wen-tau and Toutanova, Kristina and Platt, John C. and Meek, Christopher},
  booktitle = {Proceedings of the Fifteenth Conference on Computational Natural Language Learning},
  year = {2011},
  url = {https://aclanthology.org/W11-0329/}
}

References

36 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of HLT-NAACL, pages 19–27, June. find
  2. Arthur Asuncion, Max Welling, Padhraic Smyth, and Yee Whye Teh. 2009. On smoothing and inference for topic models. In UAI. find
  3. Roberto Basili, Marco Cammisa, and Alessandro Moschitti. 2005. Effective use of WordNet semantics via kernel-based learning. In CoNLL. find
  4. David M. Blei, Andrew Y. Ng, Michael I. Jordan, and John Lafferty. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. find our card
  5. Stephan Bloehdorn and Alessandro Moschitti. 2007. Combined syntactic and semantic kernels for text classification. In ECIR, pages 307–318. find
  6. Andrei Z. Broder, Peter Ciccolo, Marcus Fontoura, Evgeniy Gabrilovich, Vanja Josifovski, and Lance Riedel. 2008. Search advertising using web relevance feedback. In CIKM, pages 1013–1022. find
  7. Jane Bromley, James W. Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. 1993. Signature verification using a “Siamese” time delay neural network. International Journal Pattern Recognition and Artificial Intelligence, 7(4):669–688. find
  8. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In ICML. find our card
  9. Y. Choi, M. Fontoura, E. Gabrilovich, V. Josifovski, M. Mediano, and B. Pang. 2010. Using landing pages for sponsored search ad selection. In WWW. find
  10. Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In Proceedings of CVPR-2005, pages 539–546. find
  11. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML. find
  12. Nello Cristianini, John Shawe-Taylor, and Huma Lodhi. 2002. Latent semantic kernels. Journal of Intelligent Information Systems, 18(2–3):127–152. find
  13. Jason V. Davis and Inderjit S. Dhillon. 2008. Structured metric learning for high dimensional problems. In KDD, pages 195–203. find
  14. Jason V. Davis, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit S. Dhillon. 2007. Information-theoretic metric learning. In ICML. find
  15. Scott Deerwester, Susan Dumais, George Furnas, Thomas Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391– 407. find
  16. Konstantinos I. Diamantaras and S.Y. Kung. 1996. Principal Component Neural Networks: Theory and Applications. Wiley-Interscience. find
  17. Susan T. Dumais, Todd A. Letsche, Michael L. Littman, and Thomas K. Landauer. 1997. Automatic cross-linguistic information retrieval using latent semantic indexing. In AAAI-97 Spring Symposium Series: Cross-Language Text and Speech Retrieval. find
  18. Ofer Egozi, Evgeniy Gabrilovich, and Shaul Markovitch. 2008. Concept-based feature generation and selection for information retrieval. In AAAI. find
  19. Pascale Fung and Lo Yuen Yee. 1998. An IR approach for translating new words from nonparallel, comparable texts. In Proceedings of COLING-ACL. find
  20. Kuzman Ganchev, Joao Graca, Jennifer Gillenwater, and Ben Taskar. 2009. Posterior regularization for structured latent variable models. Technical Report MS-CIS-09-16, University of Pennsylvania. find
  21. Jianfeng Gao, Kristina Toutanova, and Wen-tau Yih. 2011. Clickthrough-based latent semantic models for web search. In SIGIR. find
  22. G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, July. find
  23. Thomas Hofmann. 1999. Probabilistic latent semantic indexing. In SIGIR ’99, pages 50–57. find
  24. K. Jarvelin and J. Kekalainen. 2000. Ir evaluation methods for retrieving highly relevant documents. In SI-GIR, pages 41–48. find
  25. Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In Proc. of COLING-ACL 98. find
  26. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Pres. find
  27. David Mimno, Hanna W. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009. Polylingual topic models. In EMNLP. find
  28. Dragos Stefan Munteanu and Daniel Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics, 31:477–504. find our card
  29. Jorge Nocedal and Stephen Wright. 2006. Numerical Optimization. Springer, 2nd edition. find
  30. John Platt, Kristina Toutanova, and Wen-tau Yih. 2010. Translingual document representations from discriminative projections. In EMNLP. find
  31. Reinhard Rapp. 1999. Automatic identification of word translations from unrelated English and German corpora. In Proceedings of the ACL, pages 519–526. find
  32. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In ACL. find our card
  33. Alexei Vinokourov, John Shawe-taylor, and Nello Cristianini. 2003. Inferring a semantic representation of text via cross-language correlation analysis. In NIPS-15. find
  34. Vishnu Vyas and Patrick Pantel. 2009. Semi-automatic entity set refinement. In NAACL ’09, pages 290–298. find
  35. Wen-tau Yih and Ning Jiang. 2010. Similarity models for ad relevance measures. In MLOAD - NIPS 2010 Workshop on online advertising. find
  36. Appendix A. Gradient Derivation The gradient of the loss function in Eq. (2) can be derived as follows. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yih-2011-learning-discriminative-projections-text-similarity/