SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

1221 papers of our library cite it (of the 140673 whose bibliographies we read); 106 of them have a page here, the first 100 below, 1115 not yet.

• ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic
• Muppet: Massive Multi-task Representations with Pre-Finetuning
• Unsupervised Domain Clusters in Pretrained Language Models
• The Falcon Series of Open Language Models
• Detecting Language Model Attacks with Perplexity
• PaLM 2 Technical Report
• TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
• Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
• GPT-NeoX-20B: An Open-Source Autoregressive Language Model
• On the Opportunities and Risks of Foundation Models
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
• LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
• A Survey on Evaluation of Large Language Models
• TabFact: A Large-scale Dataset for Table-based Fact Verification
• PaLI: A Jointly-Scaled Multilingual Language-Image Model
• Benchmarking Large Language Models in Retrieval-Augmented Generation
• Adapting Language Models to Compress Contexts
• Training Verifiers to Solve Math Word Problems
• Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages
• A Survey on In-context Learning
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• GLM: General Language Model Pretraining with Autoregressive Blank Infilling
• LawBench: Benchmarking Legal Knowledge of Large Language Models
• IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
• Time Travel in LLMs: Tracing Data Contamination in Large Language Models
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• PPT: Pre-trained Prompt Tuning for Few-shot Learning
• LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
• Evaluating Large Language Models: A Comprehensive Survey
• FOLIO: Natural Language Reasoning with First-Order Logic
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
• Measuring Massive Multitask Language Understanding
• CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
• Measuring Coding Challenge Competence With APPS
• Measuring Mathematical Problem Solving With the MATH Dataset
• An Analysis of Natural Language Inference Benchmarks through the Lens of Negation
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Language Is Not All You Need: Aligning Perception with Language Models
• Large Language Models Can Self-Improve
• TrustLLM: Trustworthiness in Large Language Models
• COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs
• Revealing the Dark Secrets of BERT
• FlauBERT: Unsupervised Language Model Pre-training for French
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
• CMMLU: Measuring massive multitask language understanding in Chinese
• Few-shot Learning with Multilingual Language Models
• How Can We Accelerate Progress Towards Human-like Linguistic Generalization?
• GPT Understands, Too
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
• Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
• GAIA: a benchmark for General AI Assistants
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• LILA: A Unified Benchmark for Mathematical Reasoning
• MTEB: Massive Text Embedding Benchmark
• SGPT: GPT Sentence Embeddings for Semantic Search
• CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
• A Comprehensive Overview of Large Language Models
• Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
• Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark
• TALM: Tool Augmented Language Models
• NOPE: A Corpus of Naturally-Occurring Presuppositions in English
• Kosmos-2: Grounding Multimodal Large Language Models to the World
• True Few-Shot Learning with Language Models
• How Context Affects Language Models' Factual Predictions
• KILT: a Benchmark for Knowledge Intensive Language Tasks
• Humanity's Last Exam
• Tool Learning with Foundation Models
• InFoBench: Evaluating Instruction Following Ability in Large Language Models
• Pre-trained Models for Natural Language Processing: A Survey
• DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
• XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• LaMP: When Large Language Models Meet Personalization
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• How many data points is a prompt worth?
• It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
• Green AI
• SCROLLS: Standardized CompaRison Over Long Language Sequences
• mGPT: Few-Shot Learners Go Multilingual
• When Your Language Model Cannot Even Do Determiners Right: Probing for Anti-Presuppositions and the Maximize Presupposition! Principle
• PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• IMPLI: Investigating NLI Models' Performance on Figurative Language
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• UL2: Unifying Language Learning Paradigms
• Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
• SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• BLiMP: The Benchmark of Linguistic Minimal Pairs for English
• Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
• Larger language models do in-context learning differently
• Ethical and social risks of harm from Language Models
• Transformers: State-of-the-Art Natural Language Processing
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1905.00537 · arXiv 1905.00537

@misc{wang2019superglue,
  title = {{SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems}},
  author = {Wang, Alex and Pruksachatkun, Yada and Nangia, Nikita and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},
  year = {2019},
  doi = {10.48550/arxiv.1905.00537},
  eprint = {1905.00537},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1905.00537}
}

References

74 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stephen H. Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alexander Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher Ré, and Rob Malkin. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In SIGMOD. ACM, 2018. find
2. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 2006. URL http://u.cs.biu.ac.il/~nlp/RTE2/P… ↗
3. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. In Textual Analysis Conference (TAC), 2009. URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.232.1231. ↗
4. Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc. Modeling empathy and distress in reaction to news stories. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. find
5. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. Re-evaluation the role of bleu in machine translation research. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL). Association for Computational Linguistics, 2006. URL https://www.a… ↗
6. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguis… find
7. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2018a. find our card
8. Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. Ultra-fine entity typing. In Proceedings of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 2018b. URL https://www.aclweb.org/anthology/P18-1009. ↗
9. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find our card
10. Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. BAM! Born-again multi-task networks for natural language understanding. In Proceedings of the Association of Computational Linguistics (ACL). Association for Computational Linguistics, 2019b. URL https://arxiv… find
11. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning (ICML). Association for Computing Machinery, 2008. URL https://dl.acm. org/citation.cfm?id=1… ↗
12. Alexis Conneau and Douwe Kiela. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the 11th Language Resources and Evaluation Conference. European Language Resource Association, 2018. URL https://www.aclweb.org/anthology/L18-1269. ↗ our card
13. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computati… find our card
14. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment. Springer, 2006. URL https: //link.springer.com/chapter/10.1007/117… find
15. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2015. URL http://papers. nips.cc/paper/5949-semi-supervised-sequence-learning.pdf. find
16. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. 2019. To appear in Proceedings of Sinn und Bedeutung 23. Data can be found at https://github.com/mcdm/CommitmentBank/. ↗ our card
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)… find
18. William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of IWP, 2005. find our card
19. Manaal Faruqui and Dipanjan Das. Identifying well-formed natural language questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2018. URL https://www.aclweb.org/anthology/D18-1091. ↗
20. Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. International Conference on Machine Learning (ICML), 2018. URL http://proceedings.mlr.press/v80/furlanello18a.html. ↗
21. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software, 2017. URL https://www.aclweb.org/an… ↗ our card
22. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. Association for Computational Linguistics, 2007. find
23. Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… find our card
24. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Association for Computationa… find our card
25. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1503.02531, 2015. URL https://arxiv.org/abs/1503.02531. ↗ our card
26. Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2017. doi: 10.18653/v1/D17-1215. URL https: //www.aclweb.org/antholog… find our card
27. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find our card
28. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6980, 2014. URL https://arxiv.org/abs/1412.6980. ↗
29. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, 2015. find
30. Nikita Kitaev, Steven Cao, and Dan Klein. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 1… find our card
31. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for the Winograd schema challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837–4842, Florence, Italy, July 2019. Associati… find
32. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, September 2017. doi: 10.18653/v1/D17-1018. URL https://www.aclweb.o… ↗ our card
33. Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012. URL http://dl.acm.org/citation.cfm?id=3031843.3031909. ↗
34. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Languag… find
35. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (N… find our card
36. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Association for Computation… find
37. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint 1904.09482, 2019c. URL http://arxiv.org/abs/1904.09482. — [arXiv:1904.09482](https://arxiv.org/abs/1904.09482) ↗
38. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint 1901.11504, 2019d. find our card
39. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. arXiv preprint 1807.11714, 2018. URL http://arxiv.org/abs/1807.11714. ↗ our card
40. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/7209-learned-in-translation-contextualized-word-vectors… ↗ our card
41. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint 1806.08730, 2018. URL https://arxiv.org/abs/1806.08730. ↗ our card
42. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 2019. URL https://arxiv.org/abs/1902.01007. ↗ our card
43. Richard T. McCoy and Tal Linzen. Non-entailed subsequences as a challenge for natural language inference. In Proceedings of the Society for Computational in Linguistics (SCiL) 2019, 2019. URL https://scholarworks.umass.edu/scil/vol2/iss1/46/. ↗
44. George A Miller. WordNet: a lexical database for english. Communications of the ACM, 1995. URL https://www.aclweb.org/anthology/H94-1111. ↗ our card
45. Aakanksha Naik, Abhilasha Ravichander, Norman M. Sadeh, Carolyn Penstein Rosé, and Graham Neubig. Stress test evaluation for natural language inference. In International Conference on Computational Linguistics (COLING), 2018. find
46. Nikita Nangia and Samuel R. Bowman. Human vs. Muppet: A conservative estimate of human performance on the GLUE benchmark. In Proceedings of the Association of Computational Linguistics (ACL). Association for Computational Linguistics, 2019. URL https: //woollysocks.github.io/assets/GLUE\_Human\_Baseli… find
47. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In Advances in Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 2017. URL https://openreview… find
48. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2014. doi: 10.3115/v1/D14-1162. URL https://www.aclweb.… ↗
49. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-… find
50. Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint 1811.01088, 2018. URL https: //arxiv.org/abs/1811.01088. find our card
51. Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Asso… find our card
52. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proce… find our card
53. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. Unpublished ms. available through a link at https://blog.openai.com/language-unsupervised/. ↗
54. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2016. doi: 10.18653/v1/D16-1264. UR… find our card
55. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011. find
56. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistic… find our card
57. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. SocialIQa: Commonsense reasoning about social interactions. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. URL https://arxiv.org/abs/1904.09728. ↗ our card
58. Nathan Schneider and Noah A Smith. A corpus and model integrating multiword expressions and supersenses. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Association for Computational Linguistics… find our card
59. Karin Kipper Schuler. Verbnet: A Broad-coverage, Comprehensive Verb Lexicon. PhD thesis, 2005. URL http://verbs.colorado.edu/~kipper/Papers/dissertation.pdf. ↗
60. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association fo… find
61. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. International Conference on Learning Repr… find our card
62. Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, and Niranjan Balasubramanian. Repurposing entailment for multi-hop question answering tasks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
63. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019a. URL https://openreview. net/forum?id=rJ4km2R5t7. find our card
64. Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Haokun Liu, Najoung Kim, Phu Mon Htut, Thibault F’evry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann,… find
65. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association of Computational Linguists, 2019. URL https://arxiv.org/abs/1805.12471. ↗
66. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics (TACL), 2018. URL https://www.aclweb.org/anthology/Q18-1042. ↗ our card
67. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Association for… find
68. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems (NeurIPS), 2019. find our card
69. Fabio Massimo Zanzotto and Lorenzo Ferrone. Have you lost the thread? discovering ongoing conversations in scattered dialog blocks. ACM Transactions on Interactive Intelligent Systems (TiiS), 2017. find
70. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. 2018. URL https://www.aclweb.org/anthology/D18-1009. ↗
71. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885, 2018. find our card
72. Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019. URL https://arxiv.org/abs/1904.01130. ↗ our card
73. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Assoc… find our card
74. Table 4: Baseline performance on the SuperGLUE development. Model Metrics Avg BoolQ Acc. CB Acc./F1 Most Frequent Class 47.7 CBOW 47.7 BERT 72.2 BERT++ 74.6 62.2 62.4 77.7 80.1 50.0/22.2 71.4/49.6 94.6/93.7 96.4/95.0 find