Elamur Research⌕
☰
home / papers / 2018 / Strong Baselines for Neural Semi-Supervised Learning under Domain Shift

Strong Baselines for Neural Semi-Supervised Learning under Domain Shift

Paper · 2018 · Sebastian Ruder and Barbara Plank

Summary

The paper asks whether recent neural methods for unsupervised domain adaptation really beat older, task-agnostic semi-supervised techniques. The authors note that new methods are often tested on one task, on private data, or against weak baselines. They port three classic bootstrapping algorithms to neural models: self-training, tri-training, and tri-training with disagreement. They test these on two tasks with different structure, POS tagging (including the SANCL web domains) and sentiment classification. The comparison includes temporal ensembling, DANN, and asymmetric tri-training. They also introduce MT-Tri, a multi-task variant of tri-training. In MT-Tri, three output heads share one Bi-LSTM encoder, and an orthogonality loss keeps the heads diverse, which cuts the cost of training three separate models. MT-Tri gives the best sentiment results. However, plain tri-training with a few practical adjustments is the most reliable method overall and wins on POS tagging. The broader lesson is that classic bootstrapping deserves a place as a standard baseline. Results should also be reported over several runs.

Key points

  • Three bootstrapping algorithms are re-examined for neural networks: self-training (Yarowsky, 1995), the original tri-training method proposed by Zhou and Li (2005), and Søgaard's (2010) disagreement-based version of tri-training. They are compared with temporal ensembling (Laine and Aila, 2017), DANN (Ganin et al., 2016), and asymmetric tri-training (Saito et al., 2017).
  • In MT-Tri, three softmax heads sit on a shared Bi-LSTM. Orthogonality is enforced only between heads m1 and m2, with weight γ=0.01, which was not tuned. The third head is trained on pseudo-labeled data alone, so it gradually becomes target-specific.
  • To keep costs down, the classic methods draw 10k unlabeled candidates per epoch. The neural methods use a candidate pool that grows linearly.
  • The SANCL target domains are hard:
    • OOV rates are 6.8–11%, compared with 2.3% on WSJ.
    • Unseen word-tag rates are 2.91% for answers and 3.47% for emails, compared with 0.61% on WSJ.
    • Unknown tags (ADD, GW, NFP, XX) appear in every target domain except weblogs.
  • On POS tagging, classic tri-training significantly beats the baseline in 4/5 domains and does best on OOV and rare tokens. No bootstrapping method handles unseen word-tag combinations well; the less lexicalized FLORS tagger is better there in 4/5 domains.

Interesting ideas

  • I1. Rank instead of threshold: Neural confidence scores are poorly calibrated, so a fixed probability cutoff for pseudo-labels is unreliable. Taking the top-n most confident predictions each epoch ("throttling") relies only on the relative order of scores. This makes it a safer choice for self-labeling with any neural model.
  • I2. Cheap ensemble diversity via orthogonal heads: Training separate models on bootstrap samples is expensive. Instead, one encoder can be shared, with an orthogonality penalty that pushes the classifier heads to use different features. This gives co-training-style disagreement at roughly the cost of a single model.
  • I3. Subsample the unlabeled pool: Labeling only a random batch of unlabeled candidates per epoch makes agreement-based methods practical with millions of unlabeled examples.
  • I4. Analyze errors by frequency bins and unseen word-tag pairs: Breaking accuracy down by binned log frequency, OOVs, and unknown word-tag combinations shows where each adaptation method helps or fails. This is a reusable diagnostic for tagging or parsing on low-resource or noisy domains, such as Ukrainian web text.
  • I5. Always include classic baselines and multiple runs: The paper's main finding is that a carefully tuned old method can beat new ones. This argues for adding general-purpose SSL baselines and reporting variance across runs when evaluating any new adaptation method.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands49 works · 4,843 citations🎓Ollscoil na Gaillimhe – University of GalwayGalway · Ireland14 works · 2,181 citations

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 works

Cite

Cite as Sebastian Ruder and Barbara Plank. 2018. Strong Baselines for Neural Semi-Supervised Learning under Domain Shift. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1096

@inproceedings{ruder2018strong,
  title = {{Strong Baselines for Neural Semi-Supervised Learning under Domain Shift}},
  author = {Sebastian Ruder and Barbara Plank},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1096},
  url = {https://doi.org/10.18653/v1/p18-1096}
}

References

56 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Abney. 2007. Semisupervised learning for computational linguistics. CRC Press. find
  2. Isabelle Augenstein, Sebastian Ruder, and Anders Søgaard. 2018. Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate Label Spaces. In Proceedings of NAACL-HLT 2018. find
  3. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. Annual Meeting-Association for Computational Linguistics, 45(1):440.
  4. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain Adaptation with Structural Correspondence Learning. Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP ’06), pages 120–128.
  5. Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016. Domain Separation Networks. NIPS. find
  6. Rich Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning. find
  7. Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. 2006. Semi-Supervised Learning, volume 1. MIT press. find
  8. Hao Cheng, Hao Fang, and Mari Ostendorf. 2015. Open-domain name error detection using a multi-task rnn. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 737–746. Association for Computational Linguistics. find
  9. Trevor Cohn and Lucia Specia. 2013. Modelling annotator bias with multi-task gaussian processes: An application to machine translation quality estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32–42, Sofia, Bulgaria.… find
  10. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. arXiv preprint arXiv:1706.09733.
  11. Timothy Dozat and Christopher D. Manning. 2017. Deep Biaffine Attention for Neural Dependency Parsing. In Proceedings of ICLR 2017.
  12. Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC press. find
  13. Meng Fang and Trevor Cohn. 2016. Learning when to trust distant supervision: An application to low-resource pos tagging using cross-lingual projection. In Proceedings of CoNLL-16. find
  14. Meng Fang and Trevor Cohn. 2017. Model transfer for tagging low-resource languages using a bilingual dictionary. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 587–593. Association for Computational Linguistics. find
  15. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-Adversarial Training of Neural Networks. Journal of Machine Learning Research, 17:1–35. find
  16. Rob van der Goot, Barbara Plank, and Malvina Nissim. 2017. To normalize, or not to normalize: The impact of normalization on part-of-speech tagging. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 31–39, Copenhagen, Denmark. Association for Computational Linguistics. find
  17. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. Proceedings of ICML 2017. find
  18. Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what’s next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 473–483, Vancouver, Canada. Association for Computational…
  19. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018. find
  20. Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. 2017. Snapshot Ensembles: Train 1, get M for free. In Proceedings of ICLR 2017. find
  21. Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in nlp. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 264–271. Association for Computational Linguistics.
  22. Young-Bum Kim, Karl Stratos, and Dongchan Kim. 2017. Adversarial adaptation of synthetic or stale data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1297–1307, Vancouver, Canada. Association for Computational Linguistics. find
  23. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional lstm feature representations. Transactions of the Association for Computational Linguistics, 4:313–327.
  24. Samuli Laine and Timo Aila. 2017. Temporal Ensembling for Semi-Supervised Learning. In Proceedings of ICLR 2017. find
  25. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition. In NAACL-HLT 2016.
  26. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1–10, Vancouver, Canada. Association for Computational Linguistics.
  27. Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. 2015. The variational fair autoencoder. arXiv preprint arXiv:1511.00830. — [arXiv:1511.00830](https://arxiv.org/abs/1511.00830) find
  28. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114.
  29. Christopher D Manning. 2015. Computational linguistics and deep learning. Computational Linguistics, 41(4):701–707. find
  30. David McClosky, Eugene Charniak, and Mark Johnson. 2006a. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, New York City, USA. Association for Computational Linguistics.
  31. David McClosky, Eugene Charniak, and Mark Johnson. 2006b. Reranking and Self-Training for Parser Adaptation. International Conference on Computational Linguistics (COLING) and Annual Meeting of the Association for Computational Linguistics (ACL), (July):337–344.
  32. Gábor Melis, Chris Dyer, and Phil Blunsom. 2017. On the State of the Art of Evaluation in Neural Language Models. In arXiv preprint arXiv:1707.05589. find
  33. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980.
  34. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
  35. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL), 59. find
  36. Barbara Plank. 2011. Domain adaptation for parsing. University Library Groningen. find
  37. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
  38. Roi Reichart and Ari Rappoport. 2007. Self-training for enhancement and domain adaptation of statistical parsers trained on small datasets. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 616–623. find
  39. Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338– 348, Copenhagen, Denmark. Association for Computatio…
  40. Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2017. Learning what to share between loosely related tasks. arXiv preprint arXiv:1705.08142. find
  41. Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. 2017. Asymmetric Tri-training for Unsupervised Domain Adaptation. In ICML 2017. find
  42. Tobias Schnabel and Hinrich Schütze. 2014. FLORS: Fast and Simple Domain Adaptation for Part-of-Speech Tagging. TACL, 2:15–26. find
  43. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational L… find
  44. Anders Søgaard. 2010. Simple semi-supervised training of part-of-speech taggers. In Proceedings of the ACL 2010 Conference Short Papers, pages 205–208. find
  45. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 231–235, Berlin, Germany. Association for Computational Linguistics.
  46. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15:1929–1958. find
  47. Mark Steedman, Rebecca Hwa, Stephen Clark, Miles Osborne, Anoop Sarkar, Julia Hockenmaier, Paul Ruhlen, Steven Baker, and Jeremiah Crim. 2003. Example selection for bootstrapping statistical parsers. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the… find
  48. Jun Suzuki and Hideki Isozaki. 2008. Semi-supervised sequential labeling and segmentation using gigaword scale unlabeled data. pages 665–673. find
  49. Vincent Van Asch and Walter Daelemans. 2016. Predicting the effectiveness of self-training: Application to sentiment classification. arXiv preprint arXiv:1601.03288. — [arXiv:1601.03288](https://arxiv.org/abs/1601.03288) find
  50. Fangzhao Wu and Yongfeng Huang. 2016. Sentiment Domain Adaptation with Multiple Sources. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), pages 301–310. find
  51. David Yarowsky. 1995. Unsupervised Word Sense Disambiguation Rivaling Supervised Methods. In Proceedings of the 33rd annual meeting on Association for Computational Linguistics.
  52. Wenpeng Yin, Tobias Schnabel, and Hinrich Schütze. 2015. Online Updating of Word Representations for Part-of-Speech Tagging. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, September, pages 1329–1334. find
  53. Guangyou Zhou, Zhiwen Xie, Jimmy Xiangji Huang, and Tingting He. 2016. Bi-transferring deep neural networks for domain adaptation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 322–332, Berlin, Germany. Association for Compu… find
  54. Zhi-Hua Zhou and Ming Li. 2005. Tri-Training: Exploiting Unlabeled Data Using Three Classifiers. IEEE Trans.Data Eng., 17(11):1529–1541. find
  55. Xiaojin Zhu. 2005. Semi-Supervised Learning Literature Survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison. find
  56. Xiaojin Zhu and Andrew B Goldberg. 2009. Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning, 3(1):1–130. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ruder-2018-strong-baselines-neural-semi-supervised/