Elamur Research⌕
☰
home / papers / 2020 / Exploring and Predicting Transferability across NLP Tasks

Exploring and Predicting Transferability across NLP Tasks

Summary

Fine-tuning BERT on an intermediate task before the target task can help, but it was unclear when this works and which source task to pick. The authors run a large transfer study over 33 English datasets, 11 in each of three classes: text classification/regression, question answering, and sequence labeling. They follow the STILTs recipe with BERT-Base Uncased and test more than 3,000 source–target combinations in three data regimes, where training sets are either complete or cut to 1K examples. Transfer turns out to help more often than earlier work suggested, especially for small targets. Even small or very different source tasks can help, and out-of-class transfer often succeeds. Task and domain similarity matter more than raw source size when data is scarce. To make source selection practical, they build gradient-based task embeddings (TaskEmb), adapted from Task2Vec, and a text-based baseline (TextEmb). Ranking candidates by cosine similarity usually finds the best or a near-best source. The released task library supports cheaper transfer choices for low-resource tasks.

Key points

  • The benchmark has 33 datasets. The CR tasks come mostly from GLUE, plus SNLI and SciTail. The QA tasks come from MultiQA, such as SQuAD, NewsQA, DROP, and WikiHop. The SL tasks follow Liu et al. (2019a), such as CCG, POS-PTB, POS-EWT, NER, and Chunk.
  • There are three regimes: FULL→FULL, FULL→LIMITED, and LIMITED→LIMITED. LIMITED means a random sample of 1K training examples, and results are averaged over 20 restarts. Success is measured as relative gain over direct fine-tuning on the target.
  • Some results are counterintuitive. Part-of-speech tagging helps the DROP QA dataset, and the small STS-B dataset is the top source for MRPC and QQP.
  • TaskEmb derives task vectors from BERT's layer-wise gradients on the task loss and mainly reflects task similarity. TextEmb mainly reflects domain, for example by grouping the Penn Treebank SL tasks together.
  • The authors release pretrained models, task embeddings, and code at github.com/tuvuumass/task-transferability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

94 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 87 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.635 · arXiv 2005.00770

@inproceedings{vu2020exploring,
  title = {{Exploring and Predicting Transferability across NLP Tasks}},
  author = {Vu, Tu and Wang, Tong and Munkhdalai, Tsendsuren and Sordoni, Alessandro and Trischler, Adam and Mattarella-Micke, Andrew and Maji, Subhransu and Iyyer, Mohit},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.635},
  eprint = {2005.00770},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.635}
}

References

59 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdalghani Abujabal, Rishiraj Saha Roy, Mohamed Yahya, and Gerhard Weikum. 2019. ComQA: A community-sourced dataset for complex factoid question answering with paraphrase clusters. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  2. Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Ludmann, Duc-Duy Nguyen, and Johan Bos. 2017. The parallel meaning bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the Conference of the… find our card
  3. Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. 2019. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV 2019), pages 6430– 6439. find
  4. Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-based question answering with knowledge graph. In Proceedings of the International Conference on Computational Linguistics (COLING 2016), pages 2503–2514. find
  5. Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), pages 164–169. find our card
  6. Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic tagging with deep residual networks. In Proceedings of the International Conference on Computational Linguistics (COLING 2016), pages 3531–3541. find
  7. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 632–642. find
  8. Kris Cao and Dani Yogatama. 2020. Modelling latent skills for multitask language generation. arXiv preprint arXiv:2002.09543. — [arXiv:2002.09543](https://arxiv.org/abs/2002.09543) find
  9. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval 2017), pages 1–14. find
  10. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Huma… find our card
  11. Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009), page 758–759. find our card
  12. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Proceedings of the International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment (MLCW 2005), pa… find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAAC… find
  14. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP 2005). find our card
  15. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Hu… find our card
  16. Jessica Ficler and Yoav Goldberg. 2016. Coordination annotation extension in the Penn tree bank. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2016), pages 834–842. find
  17. Thomas M. J. Fruchterman and Edward M. Reingold. 1991. Graph drawing by force-directed placement. find
  18. Software: Practice and Experience (SPE 1991), 21(11):1129–1164. find
  19. Julia Hockenmaier and Mark Steedman. 2007. CCGbank: A corpus of CCG derivations and dependency structures extracted from the Penn treebank. find our card
  20. Computational Linguistics (CL 2007), 33(3):355–396. find
  21. Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. 2017. First Quora Dataset Release: Question pairs. find
  22. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS 2002), 20(4):422–446. find
  23. Emma Kerinec, Chloé Braud, and Anders Søgaard. 2018. When does deep multi-task learning work for loosely related document classification tasks? In Proceedings of the EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP (EMNLP Workshop BlackboxNLP 2018), pages 1–8. find
  24. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of the Conference on Artificial Intelligence (AAAI 2018). find
  25. Hector Levesque. 2011. The winograd schema challenge. In Proceedings of the AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning (AAAI Spring Symposium 2011), volume 46, page 47. find
  26. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find our card
  27. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. Multi-task deep neural networks for natural language understanding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2019), pages 4487–4496. find our card
  28. Computational Linguistics: Human Language Technologies (NAACL 2018), pages 2227–2237. find
  29. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. find our card
  30. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Ass… find our card
  31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR 2020), 21(140):1–67. find
  32. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2018), pages 784–789. find our card
  33. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2016), pages 2383– 2392. find our card
  34. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. Proceedings of the International Conference on Learning Representations (ICLR 2016). find our card
  35. Marek Rei and Helen Yannakoudakis. 2016. Compositional sequence labeling models for error detection in learner writing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2016), pages 1181–1191. find
  36. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. find our card
  37. Computational Linguistics (CL 1993), 19(2):313–330. find
  38. Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. find our card
  39. Héctor Martı́nez Alonso and Barbara Plank. 2017. When is multitask learning effective? semantic sequence prediction under varying data conditions. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), pages 44–53. find
  40. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. find our card
  41. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Kar… find
  42. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2014), pages 2897– 2904. find
  43. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models reading comprehension across documents. Transactions of the Association for Computational Linguistics (TACL 2018), 6:287–302. for semantic compositionality ove… find
  44. Alon Talmor and Jonathan Berant. 2019a. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2019), pages 4911–4921. find our card
  45. Alon Talmor and Jonathan Berant. 2019b. MultiQA repository. find
  46. Alon Talmor, Mor Geva, and Jonathan Berant. 2017. Evaluating semantic parsing against a simple web-based question answering model. In Proceedings of the Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 161–167. find
  47. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In Proceedings of the Conference on Computational Natural Language Learning and the Learning Language in Logic Workshop (CoNLL-LLL 2000), pages 127–132. find our card
  48. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Conference on Natural Language Learning (CoNLL 2003), pages 142–147. find
  49. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the Workshop on Representation Learning for NLP (RepL4NLP 2017), pages 191–200. find our card
  50. Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019a. Can you tell me how to get past sesame street? sentence-… find
  51. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018), pages 111… find
  52. Thomas Wolf, L Debut, V Sanh, J Chaumond, C Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. find our card
  53. Xi Yan, David Acuna, and Sanja Fidler. 2020. Neural data server: A large-scale search engine for transfer learning data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020). find
  54. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pa… find our card
  55. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2011), pages 180–189. find our card
  56. Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. 2018. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), pages 3712–3722. find
  57. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019b. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the International Conference on Learning Representations (ICLR 2019). find our card
  58. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics (TACL 2019), 7:625–641. find
  59. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/