GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-5446.pdf
Summary
Most NLU systems that work above the word level are built for one task and tend to fail when the data comes from a different domain. To push the field toward general-purpose language understanding, the authors introduce GLUE. It brings together nine existing English sentence-level tasks in three groups: single-sentence classification, similarity/paraphrase, and inference. The tasks differ widely in training-set size and genre. Results are reported for each task and as an average. Evaluation runs through a public online platform with a leaderboard, and test labels for four tasks are kept private to keep comparisons fair. The authors also release a small diagnostic set of NLI pairs, built by experts and tagged by linguistic phenomenon. The baselines are a multi-task BiLSTM with a shared encoder and a separate head for each task, the same model trained on one task at a time, ELMo-based variants, and pretrained sentence encoders. Models trained directly on GLUE data did best, but every system got fairly low absolute scores, both on the benchmark and on the diagnostic set. GLUE rewards models that learn from few examples and transfer knowledge between tasks.
Key points
- Tasks: CoLA and SST-2 (single sentence); MRPC, QQP and STS-B (similarity/paraphrase); MNLI, QNLI, RTE and WNLI (inference). QNLI is SQuAD recast as NLI, and WNLI is the Winograd Schema Challenge recast as NLI.
- Training data ranges from 393k examples (MNLI) and 364k (QQP) down to 2.5k (RTE) and 634 (WNLI). All tasks are binary classification except STS-B, which is regression, and MNLI, which has three classes.
- The diagnostic set has 550 hand-built NLI sentence pairs covering phenomena such as logical operators, lexical entailment and world knowledge. It was checked with a hypothesis-only baseline and by six NLP researchers who reviewed a sample.
- Average scores: the multi-task BiLSTM got 69.0, its single-task version 64.8, GenSen 66.6 (best of the pretrained encoders) and CBoW 58.9.
- Results were weakest on the small tasks. The best CoLA result was 35.0 Matthews correlation, from the single-task model, and no system beat 65.1 on WNLI.
Cited by
- ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic
- Quantifying Attention Flow in Transformers
- AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
- Muppet: Massive Multi-task Representations with Pre-Finetuning
- Unsupervised Domain Clusters in Pretrained Language Models
- MEGA: Multilingual Evaluation of Generative AI
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- The Falcon Series of Open Language Models
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
- MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
- Analysis Methods in Neural Language Processing: A Survey
- Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
- Abductive Commonsense Reasoning
- Experience Grounds Language
- Systematic Inequalities in Language Technology Performance across the World's Languages
- On the Opportunities and Risks of Foundation Models
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
- A Survey on Evaluation of Large Language Models
- TabFact: A Large-scale Dataset for Table-based Fact Verification
- Benchmarking Large Language Models in Retrieval-Augmented Generation
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- 8-bit Optimizers via Block-wise Quantization
- Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- GLM: General Language Model Pretraining with Autoregressive Blank Infilling
- Reducing Transformer Depth on Demand with Structured Dropout
- LawBench: Benchmarking Legal Knowledge of Large Language Models
- IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
- Making Pre-trained Language Models Better Few-shot Learners
- DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models
- Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
- PPT: Pre-trained Prompt Tuning for Few-shot Learning
- LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
- Parameter-Efficient Transfer Learning with Diff Pruning
- Evaluating Large Language Models: A Comprehensive Survey
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
- Measuring Massive Multitask Language Understanding
- TRUE: Re-evaluating Factual Consistency Evaluation
- An Analysis of Natural Language Inference Benchmarks through the Lens of Negation
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Unsupervised Syntactically Controlled Paraphrase Generation with Abstract Meaning Representations
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
- Large Language Models Can Self-Improve
- ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation
- TrustLLM: Trustworthiness in Large Language Models
- Editing Models with Task Arithmetic
- What Does BERT Learn about the Structure of Language?
- Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
- SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
- Investigating Reasons for Disagreement in Natural Language Inference
- SpanBERT: Improving Pre-training by Representing and Predicting Spans
- Challenges and Applications of Large Language Models
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
- Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering
- Revealing the Dark Secrets of BERT
- Hurdles to Progress in Long-form Question Answering
- FlauBERT: Unsupervised Language Model Pre-training for French
- MLQA: Evaluating Cross-lingual Extractive Question Answering
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Datasets: A Community Library for Natural Language Processing
- On the Sentence Embeddings from Pre-trained Language Models
- CMMLU: Measuring massive multitask language understanding in Chinese
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- Towards Debiasing Sentence Representations
- XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
- How Can We Accelerate Progress Towards Human-like Linguistic Generalization?
- K-BERT: Enabling Language Representation with Knowledge Graph
- Multi-Task Deep Neural Networks for Natural Language Understanding
- GPT Understands, Too
- P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
- WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
- What Makes Good In-Context Examples for GPT-3?
- AgentBench: Evaluating LLMs as Agents
- DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules
- Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4
- On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey
- "I'd rather just go to bed": Understanding Indirect Answers
- LLM-Pruner: On the Structural Pruning of Large Language Models
- Merging Models with Fisher-Weighted Averaging
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
- GAIA: a benchmark for General AI Assistants
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
- SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
- LILA: A Unified Benchmark for Mathematical Reasoning
- MTEB: Massive Text Embedding Benchmark
- CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
- A Comprehensive Overview of Large Language Models
- Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
- Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark
Cite
@inproceedings{wang2018glue,
title = {{GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding}},
author = {Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},
booktitle = {Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP},
year = {2018},
doi = {10.18653/v1/w18-5446},
eprint = {1804.07461},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/w18-5446}
}References
- lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451. find
- Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. find
- Allen Nie, Erin D Bennett, and Noah D Goodman. 2017. Dissent: Sentence representation learning from explicit discourse relations. arXiv preprint 1710.04334. — [arXiv:1710.04334](https://arxiv.org/abs/1710.04334) find
- Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. find
- Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL 2018. find
- Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. In 11th International Workshop on Semantic Evaluations. find
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. Association for Computational Linguistics. find our card
- Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, De… find our card
- Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… find
- Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. find
- Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J. Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In Proceedings of ICLR. find
- William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of IWP. find our card
- Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint 1805.12471. find
- Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9. Association for Computational Linguistics. find
- Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL 2018. find
- Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
- Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-Thought vectors. In Advances in neural information processing systems, pages 3294–3302. find
- Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In Aaai spring symposium: Logical formalizations of commonsense reasoning, volume 46, page 47. find
- Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage find