Elamur Research⌕
☰
home / papers / 2015 / Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval

Summary

The paper tackles a gap in text representation learning. Unsupervised objectives such as word prediction do not target the end task, while single-task supervised training is often starved of labeled data. The authors propose a multi-task deep neural network that learns one shared semantic vector for queries and documents. It is trained jointly on two very different kinds of objectives: binary query classification over four domains (Restaurant, Hotel, Flight, Nightlife) and web-search ranking based on click data. Inputs are bag-of-words hashed into letter trigrams, then projected into a shared 300-dimensional layer. On top sit task-specific 128-dimensional layers with either a logistic output for classification or a cosine-similarity ranker in the style of DSSM. Training uses SGD, picking a random task for each update. The model beats strong baselines on both query classification and web search, and multi-task models often help some tasks while hurting others, so the gains across all tasks stand out. The shared representation also makes adaptation to new domains cheaper, needing far fewer in-domain labels in most data regimes.

Key points

  • Architecture:
    • Input is a 500k-word bag-of-words, hashed into roughly 50k letter trigrams (l1).
    • A tanh projection produces the shared semantic layer (l2, 300 dimensions).
    • Per-task tanh layers produce 128-dimensional representations (l3).
  • Task outputs:
    • Query classification is framed as four separate binary classifiers with a sigmoid output and cross-entropy loss.
    • Web search uses a pairwise, click-based objective: a softmax over cosine scores for one clicked document against J sampled non-clicked documents, with a smoothing factor γ tuned on held-out data.
  • Training setup:
    • Mini-batches of 1024 samples with a fixed learning rate of 0.1/1024.
    • Uniform fan-in/fan-out initialization; layer-wise pre-training and AdaGrad/momentum gave no extra accuracy.
    • 800K iterations, about 13 hours on one NVidia K20 GPU.
  • Domain adaptation (measured by AUC):
    • An SVM on the fixed shared representation outperformed SVMs on Word3gram and Letter3gram features in the Hotel, Flight and Restaurant domains.
    • The exception was the regime with more than about 1 million in-domain samples.
  • DNN comparison:
    • Training all layers from scratch (DNN1) only won with millions of samples.
    • Freezing the multi-task W1 (DNN2) was best for medium-sized data.
    • The SVM on the shared representation was best at a few thousand samples.
    • Nightlife results deviated, which the authors attribute to a tiny test set of 298 queries.

Interesting ideas

  • I1. Joining unlike objectives through one shared layer. Classification and ranking objectives can share a single representation layer while keeping separate heads. Any project with heterogeneous supervision, such as labeled categories plus similarity or click pairs, could pool it into one encoder this way.
  • I2. Random task sampling per mini-batch. Picking a task at random for each update approximately optimizes the sum of all objectives. It avoids complicated loss weighting and makes adding a new task trivial.
  • I3. Letter-trigram hashing. Hashing words into letter trigrams keeps the input layer small and handles out-of-vocabulary words and spelling variants. This seems attractive for morphologically rich or noisy text, for example Ukrainian user queries with typos.
  • I4. Independent binary classifiers per domain. Using one binary classifier per domain instead of a single multi-class model lets you add a new domain without retraining the existing ones. This is a practical design choice for growing label sets.
  • I5. Evaluating representations by label budget. Plotting performance against the amount of in-domain training data shows where frozen shared features help and where training from scratch wins. That kind of curve is more informative than a single score when deciding how to deploy into a new domain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 65 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/n15-1092

@inproceedings{liu2015representation,
  title = {{Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval}},
  author = {Liu, Xiaodong and Gao, Jianfeng and He, Xiaodong and Deng, Li and Duh, Kevin and Wang, Ye-yi},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1092},
  url = {https://doi.org/10.3115/v1/n15-1092}
}

References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 238–247, Ba… find our card
  2. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. the Journal of machine Learning research, 3:993–1022. find our card
  3. Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. 2012. Joint learning of words and meaning representations for open-text semantic parsing. In AISTATS. find
  4. Andrew P Bradley. 1997. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1145–1159. find
  5. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89–96. ACM. find our card
  6. Rich Caruana. 1997. Multitask learning. Machine Learning, 28. find
  7. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar, October. Association for Computational Linguistics. find our card
  8. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  9. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6). find
  10. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  11. E. Gabrilovich and S. Markovitch. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. In IJCAI. find
  12. Jianfeng Gao, Xiaodong He, and Jian-Yun Nie. 2010. Clickthrough-based translation models for web search: from word models to phrase models. In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 1139–1148. ACM. find
  13. Jianfeng Gao, Kristina Toutanova, and Wen-tau Yih. 2011. Clickthrough-based latent semantic models for web search. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 675–684. ACM. find
  14. Jianfeng Gao, Xiaodong He, Wen-tau Yih, and Li Deng. 2014a. Learning continuous phrase representations for translation modeling. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 699–709, Baltimore, Maryland, June. Association f… find
  15. Jianfeng Gao, Patrick Pantel, Michael Gamon, Xiaodong He, and Li Deng. 2014b. Modeling interestingness with deep neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2–13, Doha, Qatar, October. Association for Computational Linguis… find
  16. Thomas Hofmann. 1999. Probabilistic latent semantic indexing. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 50–57. ACM. find
  17. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 2333–2338. ACM. find
  18. Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In NIPS. find
  19. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. 2014. A neural network for factoid question answering over paragraphs. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 633–644, Doha, Qatar, October. Assoc… find
  20. Kalervo Järvelin and Jaana Kekäläinen. 2000. Ir evaluation methods for retrieving highly relevant documents. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 41–48. ACM. find
  21. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 655–665, Baltimore, Maryland, June. Association for Computa… find our card
  22. Juha Karhunen. 1998. Principal component neural net-workstheory and applications. Pattern Analysis & Applications, 1(1):74–75. find
  23. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the International Conference on Machine Learning (ICML). find our card
  24. R. Bernardi M. Baroni and R. Zamparelli. 2013. Frege in space: A program for compositional distributional semantics. Linguistic Issues in Language Technologies. find
  25. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  26. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia,… find our card
  27. Tomás̆ Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013c. Distributed representations of words and phrases and their compositionality. In NIPS. find
  28. Dmitrijs Milajevs, Dimitri Kartsaklis, Mehrnoosh Sadrzadeh, and Matthew Purver. 2014. Evaluating neural word representations in tensor-based compositional settings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 708–719, Doha, Qatar, October.… find
  29. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems 26 (NIPS 2013). find
  30. Gregoire Montavon, Genevieve Orr, and Klaus-Robert Muller. 2012. Neural Networks: Tricks of the Trade 2nd ed. springer. find
  31. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics. find
  32. B. Schölkopf, A. Smola, and K.-R. Müller. 1998. Nonlinear component analysis as kernel eigenvalue problem. Neural Computation, 10. find
  33. Dou Shen, Rong Pan, Jian-Tao Sun, Jeffrey Junfeng Pan, Kangheng Wu, Jie Yin, and Qiang Yang. 2006. Query enrichment for web-query classification. ACM Trans. Inf. Syst., 24(3):320–352, July. find
  34. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 101–110. ACM. find
  35. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  36. Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394, Uppsala, Sweden, July. find our card
  37. Chengxiang Zhai and John Lafferty. 2001. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 334–342. ACM. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2015-representation-learning-multi-task-deep/