Elamur Research
☰
home / papers / 2018 / Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1811.01088

Summary

Large pretrained sentence encoders are usually fine-tuned directly on each target task. That step can be fragile, especially when target data are scarce. The authors propose STILTs (Supplementary Training on Intermediate Labeled-data Tasks), which adds an extra fine-tuning stage on a large labeled task between unsupervised pretraining and target-task training. They apply it to BERT, GPT and an ELMo variant. The intermediate tasks are MNLI, SNLI, QQP and a synthetic fake-sentence-detection task built from BooksCorpus. Evaluation covers the nine GLUE tasks, including runs where target training sets are cut to 1k or 5k examples. Picking the best intermediate task for each target, BERT on STILTs reached a GLUE score of 81.8, the top leaderboard result at the time and 1.4 points above plain BERT. Sentence-pair tasks and small datasets gained most, while single-sentence tasks like CoLA and SST did not improve. The method also made 24-layer BERT fine-tuning noticeably more stable. It matters because it is cheap: only one more fine-tuning round using existing infrastructure.

Key points

  • Pipeline: (1) unsupervised pretraining, (2) supervised training on a data-rich intermediate task (MNLI, SNLI, QQP or BooksCorpus real/fake detection), (3) fine-tuning on the GLUE target task. GPT and ELMo were run with the jiant toolkit (AllenNLP/PyTorch), and BERT with the HuggingFace PyTorch port.
  • The "Best of Each" BERT setup scored 81.8 on the GLUE test set (+1.4 over BERT). For BERT on RTE, MNLI as the intermediate task added about 8 points on the test set.
  • Stability with 5k training examples (24-layer BERT): degenerate runs fell from 5 to 1 on SST and from 2 to 0 on STS. Variance across random restarts also dropped.
  • Benefits were larger with downsampled training sets (1k/5k), reaching up to about 10 points for some task pairs. They were also larger for GPT and ELMo than for BERT.
  • Compared with multitask training for GPT with RTE as the target, sequential STILTs did best:
    • plain fine-tuning: 54.2
    • MNLI→RTE: 70.4
    • joint {MNLI, RTE}: 68.6
    • joint {MNLI, RTE}→RTE: 67.5
  • Evaluation details: WNLI was handled by predicting the majority label, and reported numbers use the older QNLIv1.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

260 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 239 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.01088 · arXiv 1811.01088

@misc{phang2018sentence,
  title = {{Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks}},
  author = {Phang, Jason and Févry, Thibault and Bowman, Samuel R.},
  year = {2018},
  doi = {10.48550/arxiv.1811.01088},
  eprint = {1811.01088},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.01088}
}

References

31 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In EACL. find
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. find
  3. Samuel R. Bowman, Ellie Pavlick, Edouard Grave, Benjamin Van Durme, Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, and Berlin Chen. 2019. Looking for ELMo’s friends: Sentence-level pretraining be… — [arXiv:1812.10860](https://arxiv.org/abs/1812.10860) find
  4. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval-2017. find
  5. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. find our card
  6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP. find our card
  7. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In ACL. find our card
  8. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810.04805. find
  10. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proc. International Workshop on Paraphrasing (IWP). find our card
  11. Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In NAACL. find
  12. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. AllenNLP: A deep semantic natural language processing platform. find our card
  13. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. find
  14. Emma Kerinec, Chloé Braud, and Anders Søgaard. 2018. When does deep multi-task learning work for loosely related document classification tasks? In Proc. EMNLP Workshop BlackboxNLP. find
  15. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In Aaai spring symposium: Logical formalizations of commonsense reasoning, volume 46, page 47. find
  16. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS. find our card
  17. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. find
  18. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. find
  19. Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In EMNLP. find our card
  20. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Unpublished manuscript accessible via the OpenAI Blog. find
  21. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP. find our card
  22. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. find
  23. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J. Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In ICLR. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  25. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804.07461. find our card
  26. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint 1805.12471. find
  27. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proc. Eighth International Joint Conference on Natural Language Processing. find our card
  28. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL. find
  29. Kelly Zhang and Samuel R. Bowman. 2018. Language modeling teaches you more syntax than translation does: Lessons learned through auxiliary task analysis. arXiv preprint 1809.10040. find
  30. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015a. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19… find our card
  31. Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015b. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv:1506.06724. find our card