Transformers: State-of-the-Art Natural Language Processing

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

7291 papers of our library cite it (of the 140673 whose bibliographies we read); 220 of them have a page here, the first 100 below, 7071 not yet.

• Quantifying Attention Flow in Transformers
• AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level
• Resolving Prepositional Phrase Attachment Ambiguities with Contextualized Word Embeddings
• Style Classification of Rabbinic Literature for Detection of Lost Midrash Tanhuma Material
• Detecting Narrative Patterns in Biblical Hebrew and Greek
• Unsupervised Domain Clusters in Pretrained Language Models
• Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning
• Refusal in Language Models Is Mediated by a Single Direction
• MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• SliceGPT: Compress Large Language Models by Deleting Rows and Columns
• Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
• Qwen Technical Report
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions
• On the Opportunities and Risks of Foundation Models
• Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer
• AMR Parsing via Graph-Sequence Iterative Inference
• CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization
• LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
• German's Next Language Model
• Open-source Frame Semantic Parsing
• Rethinking embedding coupling in pre-trained language models
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• RobBERT: a Dutch RoBERTa-based Language Model
• Mind2Web: Towards a Generalist Agent for the Web
• CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims
• Few-NERD: A Few-shot Named Entity Recognition Dataset
• Word-Level Coreference Resolution
• LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
• QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
• SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
• The Truth, The Whole Truth, and Nothing but the Truth: A New Benchmark Dataset for Hebrew Text Credibility Assessment
• LCHAIM - Investigating Long Context Reasoning in Hebrew
• MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages
• SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
• Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
• Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation
• Injecting Numerical Reasoning Skills into Language Models
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Time Travel in LLMs: Tracing Data Contamination in Large Language Models
• Evaluating Factuality in Generation with Dependency-level Entailment
• Annotating and Modeling Fine-grained Factuality in Summarization
• Applying Occam's Razor to Transformer-Based Dependency Parsing: What Works, What Doesn't, and What is Really Necessary
• Coordinate Constructions in English Enhanced Universal Dependencies: Analysis and Computational Modeling
• Parameter-Efficient Transfer Learning with Diff Pruning
• How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
• HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
• Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All
• Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language
• Towards a Unified View of Parameter-Efficient Transfer Learning
• Aligning AI With Shared Human Values
• Pretrained Transformers Improve Out-of-Distribution Robustness
• CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
• Measuring Mathematical Problem Solving With the MATH Dataset
• Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling
• Surface Form Competition: Why the Highest Probability Answer Isn't Always Right
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
• An Analysis of Natural Language Inference Benchmarks through the Lens of Negation
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Expectations over Unspoken Alternatives Predict Pragmatic Inferences
• A fine-grained comparison of pragmatic language understanding in humans and language models
• OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
• ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation
• COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs
• Editing Models with Task Arithmetic
• The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
• Neural CRF Model for Sentence Alignment in Text Simplification
• SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
• Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction
• SpanBERT: Improving Pre-training by Representing and Predicting Spans
• RealTime QA: What's the Answer Right Now?
• Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• OCR-free Document Understanding Transformer
• Large language models are zero-shot reasoners
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
• GeDi: Generative Discriminator Guided Sequence Generation
• Reformulating Unsupervised Style Transfer as Paraphrase Generation
• Data Augmentation using Pre-trained Transformer Models
• TellMeWhy: A Dataset for Answering Why-Questions in Narratives
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition
• AlephBERT: A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With
• Explicit Morphological Knowledge Improves Pre-training of Language Models for Hebrew
• HeRo: RoBERTa and Longformer Hebrew Language Models
• Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Language
• From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
• FlauBERT: Unsupervised Language Model Pre-training for French
• Can language models capture syntactic associations without surface cues? A case study of reflexive anaphor licensing in English control constructions
• PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-demos.6 · arXiv 1910.03771

@inproceedings{wolf2020transformers,
  title = {{Transformers: State-of-the-Art Natural Language Processing}},
  author = {Wolf, Thomas and Debut, Lysandre and Sanh, Victor and Chaumond, Julien and Delangue, Clement and Moi, Anthony and Cistac, Pierric and Rault, Tim and Louf, Remi and Funtowicz, Morgan and Davison, Joe and Shleifer, Sam and Platen, Patrick von and Ma, Clara and Jernite, Yacine and Plu, Julien and Xu, Canwen and Scao, Teven Le and Gugger, Sylvain and Drame, Mariama and Lhoest, Quentin and Rush, Alexander},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-demos.6},
  eprint = {1910.03771},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-demos.6}
}

References

46 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear, 7(1). a. PyTorch Hub. https://pytorch.org/hub/ . Accessed: 2020-6-29. b. ↗
2. TensorFlow hub. https://www.tensorflow. org/hub . Accessed: 2020-6-29. ↗
3. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. Flair: An easy-to-use framework for state-of-the-art nlp. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages… find our card
4. Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. 2019. exBERT: A visual analysis tool to explore learned representations in transformers models. find
5. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. find
6. Zhiting Hu, Haoran Shi, Bowen Tan, Wentao Wang, Zichao Yang, Tiancheng Zhao, Junxian He, Lianhui Qin, Di Wang, Xuezhe Ma, Zhengzhong Liu, Xiaodan Liang, Wangrong Zhu, Devendra Singh Sachan, and Eric P Xing. 2018. Texar: A modularized, versatile, and extensible toolkit for text generation. find
7. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620. find
8. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529. find our card
9. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The Long-Document transformer. find
10. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F T Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in c++. find our card
11. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for automatic knowledge graph construction. In ACL. find our card
12. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne. 2018. JAX: composable transformations of Python+NumPy programs. find
13. Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. 2019. Supervised multimodal bitransformers for classifying images and text. find
14. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. find our card
15. Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. 2018. {TVM}: An automated end-to-end optimizing compiler for deep learning. In 13th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 18),… find our card
16. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72, Vancouver, Canada. Association for Computational Linguistics. find our card
17. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. find
18. Guillaume Lample and Alexis Conneau. 2019a. Cross-lingual language model pretraining. In NeurIPS. find
19. Guillaume Lample and Alexis Conneau. 2019b. Cross-lingual language model pretraining. find
20. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a Fixed-Length context. find
21. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. find
22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. find
23. Hang Le, Loı̈c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoı̂t Crabbé, Laurent Besacier, and Didier Schwab. 2020. Flaubert: Unsupervised language model pre-training for french. In Proceedings of The 12th Language Resources and Evaluation Conferen… find our card
24. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. find our card
25. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
26. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-to-Sequence pre-training for natural language generation, translation, and comprehension. find our card
27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified Text-to-Text transformer. find
28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. RoBERTa: A robustly optimized BERT pretraining approach. find
29. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. find
30. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar S. Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. find
31. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053. find
32. Edward Loper and Steven Bird. 2002. NLTK: The natural language toolkit. find
33. Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M Rush. 2017. Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE transactions on visualization and computer graphics, 24(1):667–676. find
34. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55–60. aclwe… find our card
35. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. In ACL. find our card
36. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416. find our card
37. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In I Guyon, U V Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6294–6305. Curran As… find our card
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł Ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I Guyon, U V Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett, editors, Advances in Neural Information Processing Sys… find
39. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2018. Model cards for model reporting. find
40. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. find our card
41. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. ArXiv, abs/1905.00537. find our card
42. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. find
43. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR. find our card
44. Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, and Samuel R Bowman. 2020. jiant: A software toolkit for research on general-purpose text understanding models. arXiv preprint arXiv:2003.02249. find
45. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. ArXiv, abs/1906.08237. find our card
46. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. find our card