Elamur Research⌕
☰
home / papers / 2017 / Learning to select data for transfer learning with Bayesian Optimization

Learning to select data for transfer learning with Bayesian Optimization

Paper · 2017 · Sebastian Ruder and Barbara Plank

Summary

NLP models lose accuracy when test data come from a domain unlike their training data. When several source domains are available, it matters which training examples are used. Earlier data selection work relied on one hand-picked similarity metric, usually studied for a single task. Ruder and Plank instead learn the selection measure. Each source example receives a score from a weighted linear combination of features:

  • Similarity features: Jensen-Shannon, Rényi, cosine and others, computed over term, topic and embedding representations.
  • Diversity features: entropy, type-token ratio and others.

The weights are tuned with Bayesian Optimization. The black-box function trains a model on the top-ranked examples and returns its score on a small target validation set, so the approach does not depend on the model. Experiments cover sentiment analysis on Amazon reviews, plus POS tagging and dependency parsing on SANCL 2012. On all three tasks the learned measures outperform existing similarity metrics, and they perform on par with a leading domain adaptation approach. Diversity complements similarity, and the learned weights partly carry over across models, domains and related tasks.

Key points

  • Optimization setup: a Gaussian Process prior with Expected Improvement acquisition. The authors also tried FABOLAS but found its results inconclusive for their small training sets.
  • Similarity features: Jensen-Shannon, Rényi (α = 0.99), Bhattacharyya, cosine, Euclidean and variational distance. They are computed over term distributions, LDA topic distributions and weighted word embeddings.
  • Diversity features: number of word types, type-token ratio, entropy, Simpson's index, Rényi entropy and quadratic entropy.
  • Data and models:
    • Sentiment: Amazon reviews, 4 domains with 2,000 labeled reviews each, a linear SVM on tf-idf unigrams and bigrams, and n = 1600 selected examples.
    • POS tagging and parsing: SANCL 2012, 6 domains, a Structured Perceptron tagger and a Bi-LSTM parser (Kiperwasser and Goldberg, 2016) evaluated with LAS.
  • Cross-task transfer: weights transfer well between POS tagging and parsing, but performance drops sharply between unrelated tasks such as sentiment analysis and POS tagging. Combining similarity and diversity features gives the most robust transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands49 works · 4,843 citations🎓Ollscoil na Gaillimhe – University of GalwayGalway · Ireland14 works · 2,181 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Sebastian Ruder and Barbara Plank. 2017. Learning to select data for transfer learning with Bayesian Optimization. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1038

@inproceedings{ruder2017learning,
  title = {{Learning to select data for transfer learning with Bayesian Optimization}},
  author = {Sebastian Ruder and Barbara Plank},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1038},
  url = {https://doi.org/10.18653/v1/d17-1038}
}

References

53 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. A El-Gamal. 1991. The role of priors in active Bayesian learning in the sequential statistical decision framework. In Maximum Entropy and Bayesian Methods, pages 33–38. Springer Netherlands. find
  2. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A Simple But Tough-to-Beat Baseline for Sentence Embeddings. In ICLR 2017. find
  3. Yang Fan, Fei Tian, Tao Qin, Jiang Bian, and Tie-Yan Liu. 2017. Learning What Data to Learn. In Workshop track - ICLR 2017. find
  4. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. ACM.
  5. Jing Jiang and ChengXiang Zhai. 2007. Instance Weighting for Domain Adaptation in NLP. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, (October):264– 271.
  6. Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2007. Analysis of representations for domain adaptation. Advances in Neural Information Processing Systems, 19:137–144. find
  7. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations. Transactions of the Association for Computational Linguistics, 4:313–327.
  8. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. 2009. Curriculum learning. In International Conference on Machine Learning, ICML.
  9. Anil Bhattacharya. 1943. On a measure of divergence between two statistical population defined by their population distributions. Bulletin Calcutta Mathematical Society 35.99-109 (1943): 28., 35(99-109):28. find
  10. Aaron Klein, Stefan Falkner, and Frank Hutter. 2017. Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017). find
  11. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(4-5):993–1022.
  12. Lillian Lee. 2001. On the Effectiveness of the Skew Divergence for Statistical Language Analysis. AIS-TATS (Artificial Intelligence and Statistics), pages 65–72. find
  13. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. Annual Meeting-Association for Computational Linguistics, 45(1):440.
  14. J Lin. 1991. Divergence Measures Based on the Shannon Entropy. IEEE Transactions on Information theory, 37(1):145–151. find
  15. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain Adaptation with Structural Correspondence Learning. Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP ’06), pages 120–128.
  16. Ji Ma, Yue Zhang, and Jingbo Zhu. 2014. Tagging The Web: Building A Robust Web Tagger with Neural Network. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 144–154. find
  17. Danushka Bollegala, David Weir, and John Carroll. 2011. Using Multiple Sources to Construct a Sentiment Sensitive Thesaurus for Cross-Domain Sentiment Classification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pag… find
  18. David J. C. MacKay. 1992. Information-Based Objective Functions for Active Data Selection. Neural Computation, 4(4):590–604. find
  19. Yishay Mansour. 2009. Domain Adaptation with Multiple Sources. find
  20. Neural Information Processing Systems Conference (NIPS 2009). find
  21. Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016. Domain Separation Networks. NIPS. find
  22. E Brochu, V M Cora, and N De Freitas. 2010. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. In CoRR. find
  23. David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 28–36. Association for Computational Linguistics.
  24. Hal Daumé III. 2007. Frustratingly Easy Domain Adaptation. Association for Computational Linguistic (ACL), (June):256–263.
  25. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. NIPS.
  26. Kevin Duh, Graham Neubig, Katsuhito Sudoh, and Hajime Tsukada. 2013. Adaptation Data Selection using Neural Language Models: Experiments in Machine Translation. ACL-2013, (1):678–683. find
  27. Shachar Mirkin and Laurent Besacier. 2014. Data selection for compact adapted smt models. In Eleventh Conference of the Association for Machine Translation in the Americas (AMTA). find
  28. Michael T. Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G. Dietterich. 2005. To Transfer or Not To Transfer. NIPS Workshop on Inductive Transfer. find
  29. Jonas Močkus. 1974. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference Novosibirsk. find
  30. Robert C Moore and William D Lewis. 2010. Intelligent Selection of Language Model Training Data. ACL-2010: 48th Annual Meeting of the Association for Computational Linguistics, (July):220–224.
  31. Sebastian Ruder, Parsa Ghaffari, and John G. Breslin. 2017. Data Selection Strategies for Multi-Domain Sentiment Analysis. In arXiv preprint arXiv:1702.02426. — [arXiv:1702.02426](https://arxiv.org/abs/1702.02426) find
  32. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359. find
  33. Tobias Schnabel and Hinrich Schütze. 2014. FLORS: Fast and Simple Domain Adaptation for Part-of-Speech Tagging. TACL, 2:15–26. find
  34. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543.
  35. Burr Settles. 2012. Active learning literature survey. Morgan and Claypool. find
  36. Claude E Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(July 1928):379–423. find
  37. Slav Petrov, Pi-Chuan Chang, Michael Ringgaard, and Hiyan Alshawi. 2010. Uptraining for accurate deterministic question parsing. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 705–713. Association for Computational Linguistics. find
  38. Edward H. Simpson. 1949. Measurement of diversity. Nature. find
  39. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. find
  40. Neural Information Processing Systems Conference (NIPS 2012). find
  41. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL), 59. find
  42. Andres Søgaard. 2011. Data Point Selection for Cross-Language Adaptation of Dependency Parsers. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (HLT ’11): Short Papers, pages 682–686.
  43. Barbara Plank and Gertjan van Noord. 2011. Effective Measures of Domain Similarity for Parsing. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 1:1566–1576. find
  44. Yulia Tsvetkov, Manaal Faruqui, Wang Ling, and Chris Dyer. 2016. Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). find
  45. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
  46. Vincent Van Asch and Walter Daelemans. 2010. Using Domain Similarity for Performance Estimation. Computational Linguistics, (July):31–36. find
  47. C. Radhakrishna Rao. 1982. Diversity and dissimilarity coefficients: a unified approach. Theoretical population biology, 21(1):24–43. find
  48. Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. Dynamic data selection for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
  49. Carl Edward Rasmussen. 2006. Gaussian processes for machine learning. MIT Press. find
  50. Sujith Ravi, Kevin Knight, and Radu Soricut. 2008. Automatic prediction of parser accuracy. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 887–896. Association for Computational Linguistics. find
  51. Fangzhao Wu and Yongfeng Huang. 2016. Sentiment Domain Adaptation with Multiple Sources. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), pages 301–310. find
  52. Robert Remus. 2012. Domain adaptation using Domain Similarity- and Domain Complexity-based Instance Selection for Cross-Domain Sentiment Analysis. In IEEE ICDM SENTIRE-2012. find
  53. Alfréd Rényi. 1961. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ruder-2017-learning-select-data-transfer-learning/